You Do Not Have to Move the Data First

The assumption that adds six months

Almost every data platform plan opens with the same premise: consolidate the data first, answer questions second. Move POS, ERP, inventory, and e-commerce into one store, model it, then start work.

For a lot of the questions retailers actually ask, that premise is wrong. You can query the systems where the data already lives, get answers this month, and consolidate later for the cases that genuinely need it.

This is not a trick and it is not new. It is federation, read replicas, and external tables, all of which are ordinary technology. What is unusual is treating them as a legitimate phase one rather than a compromise.

Three ways to query without moving

Read replica. A copy of the operational database maintained by the source system, queried directly. No pipeline, no transformation, no lag beyond replication. Every major database supports it and most vendors will provision one.

External tables. Files in cloud storage registered as tables in your query engine. The e-commerce platform is already dropping a nightly export somewhere. Point at it instead of ingesting it.

Federated query. The query engine reaches into the remote system at query time and joins the result to local data. Slower per query, zero pipeline to maintain.

All three give you a SQL surface over data you have not moved. In calendar terms, the difference between this and a pipeline is days versus weeks per source.

When in-place is the right call

Current-state questions. What is on hand now, what is open on order, which stores are behind plan this week. The source system already holds current state, and moving it adds latency and a failure mode without adding information.

Low query volume. A monitoring job that runs twelve queries an hour is invisible to any production database with a replica. A BI tool with 80 analysts is not.

Single-system questions. Anything answerable inside the POS alone needs no consolidation, and a surprising share of the twenty questions are single-system.

Early project phases. Weeks one through four, when the goal is to find out whether the questions are answerable at all and what the definitions have to be. Pipelines built before that discovery get rebuilt after it.

When it fails, specifically

Be precise about the limits, because the pattern gets oversold and then abandoned for the wrong reasons.

Load on the operational system. Never query the primary. A replica is fine, the primary is a production incident waiting for a Saturday. If the vendor will not provide a replica, that source is a pipeline source, and that is the end of the discussion.

History. Operational systems purge. A POS that keeps 90 days of line detail cannot answer a year-over-year question, and no amount of clever querying fixes it. History is the strongest argument for ingestion, and it is the reason to start landing raw data early even while you query in place.

Cross-system joins at volume. Joining a million POS rows to an ERP item master across a federated link is slow and gets slower. Small dimension tables federate fine. Fact-to-fact joins across systems need the data in one place.

Schema volatility. Querying the operational schema directly means a vendor upgrade can break your queries with no warning. A pipeline gives you a place to absorb that. In-place gives you a Monday morning surprise.

The hybrid that most retailers should run

The useful pattern is not lake or in-place. It is both, split by what the data is for.

Land and keep: transactional history. POS lines, orders, receipts, adjustments. Ingest these from week one because history accumulates and cannot be recovered later. Cheap to store, impossible to backfill once purged.

Query in place: current state and reference. On-hand, open orders, item master, store master, vendor master. These are small, they change constantly, and a nightly copy is worse than reading the live replica.

Snapshot on a schedule: state that matters historically. On-hand by SKU by store, captured daily and kept. This is the one people miss. On-hand is a current-state question until somebody asks what inventory looked like on the Friday before the promo, at which point a year of daily snapshots you never took becomes the answer you cannot produce.

That split takes about a day to design and it removes most of the ingestion work from the critical path while protecting the data that is genuinely irrecoverable.

What it buys in calendar time

A four-source project, planned as full ingestion: three to five weeks of connector work before the first joined answer, with the hardest source setting the pace.

The same project, hybrid: transactional history ingesting in the background from day one, reference data queried in place from day two, first joined answer in week one against live reference tables and whatever history has landed.

The saving is not in total engineering. It is in ordering. Every week the platform spends producing no answers is a week of political decay, and the hybrid pattern removes the dependency between "the WMS connector is done" and "somebody sees a number."

The two objections you will hear

"We need one source of truth." You need one definition of truth, which is a semantic problem, not a storage problem. Two systems queried with one shared definition of net sales are more consistent than one warehouse where three teams wrote three different margin queries.

"Governance requires consolidation." Sometimes true, usually for access logging and retention. Read-only access with query logging on a replica satisfies most of what an auditor asks about, and it is a materially easier review than write access to a new store. Confirm what your specific obligation is before accepting a six-month consolidation as its price.

Neither objection is silly. Both are worth an hour with the people who raised them, and both usually resolve to a narrower requirement than the one that was stated.