AI Infrastructure for Retail: Seven Layers, and the Four You Do Not Have

What "AI infrastructure" actually means when you are not OpenAI

Most writing about AI infrastructure is about GPUs. If you run 40 to 800 stores, GPUs are irrelevant to you. You are renting inference by the token from three vendors and you will never buy an H100.

The AI infrastructure that matters at mid-market is the plumbing between a model and a decision: where the data lives, how the model reaches it, who approved the reach, what the model is allowed to do with what it found, and how anyone finds out later whether the answer was right.

Seven layers. Most retailers have three of them and are surprised when the pilot stalls.

The seven layers

1. Source systems. POS, ERP, WMS, labor, e-commerce, finance. You already have these. The AI-relevant question is whether each one has a documented read path that does not involve a nightly CSV to an SFTP box.

2. Storage and compute. Snowflake, BigQuery, Databricks, or a managed Postgres if your volumes are small. This is settled technology. Retailers who spend nine months choosing here are avoiding a harder decision further up the stack.

3. The semantic layer. The definitions of your metrics, your fiscal calendar, your store hierarchy, your SKU tiering. This is the layer that decides whether the model returns the right number, and it is the layer nearly everyone skips.

4. Model access. API keys, provider routing, rate limits, fallback when Anthropic or OpenAI has an incident. Ten lines of config until you have four teams using it, at which point it is a real service.

5. Orchestration. What runs when. Scheduled monitors, event triggers, multi-step chains where step three depends on step two. This is where "we have a chatbot" turns into "we have a system."

6. Governance and identity. Which agent can see which tables, which actions need a human, what gets logged. Covered below and by every auditor you will meet in the next three years.

7. Delivery. How the output reaches a district manager who has never opened a BI tool. Email, Slack, Teams, a card in an existing app. If this layer is bad, the other six do not matter.

What mid-market retailers actually have

The common inventory, from readiness assessments across mid-market operators: layers 1 and 2 exist. Layer 4 exists as an API key in someone's environment file. Layers 3, 5, 6, and 7 do not exist at all.

That configuration produces exactly one thing: a demo that works and a deployment that does not. The demo works because the person running it knows which table to point at and can read SQL. The deployment fails because nobody else does.

The fix is not more model. It is layers 3 and 7, in that order.

The semantic layer is the bottleneck, not the model

A retailer with 480 tables and no metric definitions gets roughly 40 to 60% accuracy from a general-purpose model pointed at the raw schema. The queries execute. The numbers are wrong often enough that no VP will use them.

The same retailer, after two weeks of describing the 40 tables that answer 90% of questions, gets to the mid-80s. No model change. No new vendor.

Three artifacts do most of that work. A metric dictionary with one canonical definition per KPI. A fiscal calendar encoded as a date dimension table rather than living in a planner's head. A store and SKU hierarchy with the closed stores and the discontinued items flagged rather than deleted.

That is a data engineering project with an AI payoff. Budget it as data engineering. It will get approved faster.

Buy versus build, layer by layer

The default for mid-market should be buy at layers 2, 4, 5, and 6, build at layers 1 and 3, and negotiate hard at layer 7.

Layer 3 is the one to build, because your metric definitions are your business logic and no vendor can write them for you. Everything else is generic engineering that a vendor's team will do better because they do it for 40 customers.

The failure pattern runs the other way. Retailers build orchestration and governance from scratch, because those feel like engineering, and skip the semantic layer, because it feels like documentation. Eighteen months later they own a scheduler and still cannot answer what margin was last week.

The cost shape nobody models

AI infrastructure cost at mid-market is not dominated by inference. On a typical deployment answering a few thousand questions a month, model spend runs a few hundred to a few thousand dollars. The warehouse costs more. The people cost far more than both.

What does blow up is unbounded agent loops. An agent that decides its own next query, with no step cap, can run 200 queries chasing one anomaly. The model bill is annoying. The warehouse bill is the real one.

Three controls prevent it. A hard step cap per task. A daily token budget per agent, enforced at the gateway rather than by policy. And result caching, because the same 30 questions get asked every Monday morning.

Why the CIO owns this and not the CDO

AI infrastructure spans identity, network, vendor risk, data, and application delivery. Four of those five sit with the CIO in every mid-market org chart. The fifth usually reports to the CIO too.

More practically: the AI systems that fail in mid-market do not fail on model quality. They fail on access provisioning, on nobody owning the integration when the ERP upgrades, and on an output nobody can reach. Those are operational failures, and operations is the CIO's job.

The CDO or analytics lead should own layer 3. The CIO owns the other six and owns the fact that they connect.

The sequence that works

Weeks 1 to 2: pick one decision that a person makes weekly with bad information. Not a use case. A decision, with a name attached.

Weeks 3 to 6: build the semantic layer for the 10 to 20 tables that decision needs. Nothing else.

Weeks 7 to 8: wire read-only access, a model provider, and one delivery channel that the decision-maker already opens every day.

Weeks 9 to 12: run it in parallel with the existing process. Compare answers weekly. Fix the semantic layer where they diverge.

After 12 weeks you have a working thin slice through all seven layers and a real accuracy number. Then widen. Retailers who build layer by layer instead of slice by slice are usually 18 months in with nothing a store manager has ever seen.

Where Ward sits

Ward is layers 5, 6, and 7 as a managed service, sitting on top of whatever you already run at layers 1 through 4. We connect read-only, run the monitoring and orchestration ourselves, keep the audit trail, and deliver insight cards to the people who make the decisions.

We do not replace your warehouse and we do not want your GPU budget. We want the four layers you do not have, without an 18-month build to get them.