Connectors Are Where the Months Go

Ingestion is where the calendar disappears

Storage is solved. Compute is solved. Query engines are solved. If a data platform project runs long, look at ingestion, because that is where the months are.

The reason is that connector effort is not distributed evenly. Four of your six sources take an hour each. The other two take three weeks each. Average that out and you get a misleading "two weeks per source" number that hides both the cheap wins and the real blocker.

Sort your sources into three classes before you plan anything, because the plan for each class is different.

Class one: modern API, maintained connector

Shopify, BigCommerce, NetSuite, Square, Lightspeed, Toast, most cloud POS, most cloud ERP, Snowflake and BigQuery if you already have them.

Time: 20 to 60 minutes. Authenticate, pick objects, set a sync interval, start the backfill. The backfill runs unattended for a few hours.

The only thing that turns an hour into a week here is credentials. Somebody has to create an API user with read scope, and that somebody is often a vendor's support desk with a four-day turnaround. Request every credential in week one, in parallel, before any engineering starts. This one habit saves more calendar than any technical decision on the project.

Class two: legacy database, direct read access

An on-premise SQL Server behind the POS. An Oracle instance under a 2014 ERP. A Postgres box the WMS vendor manages.

Time: 1 to 4 days, assuming you can get a read replica and a network path.

The work is real but bounded: a replica or a read-only user, a VPN or a tunnel, an incremental extraction strategy, and the discovery that the updated_at column is null on 30% of rows so incremental sync has to key off something else.

The failure mode is not technical. It is a database the vendor treats as theirs, where read access is a contract negotiation rather than a permission grant. Start that conversation in week one too. It has a lead time and no engineering can shorten it.

Class three: no API, no database access

Labor scheduling tools that export CSV on request. Franchise POS reporting portals. A supply chain system whose entire integration story is a nightly file to an SFTP box.

Time: 2 to 3 weeks, and the ongoing maintenance cost is higher than everything else combined.

You are building a scraper or a file pipeline: fetch, parse, validate, handle the day the header row changes, handle the day the file is late, handle the day the file is there but truncated. All of that is a week. The other two weeks are finding out which of those failure modes actually happens.

The nightly CSV to SFTP trap

Flat-file feeds look like the fast option because the first one works in an afternoon. They are the slowest thing on the project measured over a year.

They break silently. A file that does not arrive produces no error, just a table that stops updating, and somebody notices nine days later when a number looks stale. A file that arrives truncated is worse, because the number does not look stale. It looks wrong in a way nobody catches.

If a source can only give you files, three rules. Alert on absence, not just on failure: expected by 6am, page at 7am. Row-count check every load against a trailing average and reject outside a band. Never overwrite the previous load until the new one validates.

That is a day of extra work per feed and it is the difference between a feed that runs for two years and a feed that quietly poisons a forecast in month four.

Connect in this order

Not alphabetically, and not by whoever is loudest. Order by answers-per-hour-of-work.

First: POS. One source, cheap to connect, answers more questions than everything else combined. Sales, units, transactions, tender, discounts, store performance, and a rough shape of demand.

Second: inventory or on-hand. Paired with POS it unlocks stockouts, sell-through, days of supply, and shrink signals. Two sources answer maybe 60% of the twenty questions an operator asks.

Third: item and cost master from the ERP. Turns revenue into margin. Usually a small extract, often the easiest connection on the list.

Fourth: e-commerce. Nearly always a class one connector, half an hour of work, and it makes omnichannel questions answerable.

Fifth and later: labor, supply chain, finance detail, marketing. Real value, lower answers-per-hour, and frequently the class three sources.

The point of the ordering is that by the end of week one you are answering questions with two sources, while the hard connector work runs in the background instead of blocking everything in front of it.

Build or buy the connector layer

Buy for class one. Nobody should be writing a Shopify connector in 2026. Managed ingestion runs a few hundred to a few thousand dollars a month at mid-market volumes, and it is cheaper than one engineer-week a quarter of maintenance.

Buy or build for class two, depending on whether your vendor is supported. A direct database replication job is well-trodden engineering and a reasonable thing to own.

Build for class three, because nobody sells a connector for your one weird system, and accept that you now own it. Budget maintenance, not just build. A flat-file pipeline costs a few hours a quarter forever.

The mistake that costs the most is the reverse: building class one connectors because they are easy, then discovering you own eleven API integrations that break whenever a vendor ships a version bump.

What a fast ingestion plan looks like

Week 1, day 1. Every credential requested. Every vendor access conversation opened. Two class one sources connected and backfilling.

Week 1, day 3. POS and inventory queryable. First joined question answered, badly.

Week 2. ERP item and cost landed. Class two sources connected as replicas come online.

Weeks 3 to 5. Class three work, in the background, with monitoring built in from the first load rather than added after the first silent failure.

The whole ingestion layer for a mid-market retailer is three to five weeks of elapsed time and well under two weeks of actual engineering, provided the credential requests went out on day one. When ingestion is quoted at four months, the constraint is almost always access, not code.