A 48-hour acceptance test any vendor can run.
Everyone in this category claims fast time to value and nobody means the same thing by it. Here is an acceptance test with a stopwatch and a rubric: connect one source, define ten metrics, run twenty agent queries, then spend four hours trying to break it. Score out of 100. Run it against us, against them, or against all three.
The number that decides the project is the one nobody quotes.
Time to value in this category is measured to whichever milestone flatters the vendor. Rows landing in a table. A first chart. A demo environment that was built the week before. All of those are real and none of them is the thing you need, which is the first number you would put in front of your CFO without checking it twice.
That number is gated by semantics, not by ingestion. Getting data into a warehouse has been a solved problem for a decade. Agreeing what the numbers mean, and reproducing what your existing reports already say, is what takes a quarter, and it is the part a two-day test can genuinely measure if it is pointed at the right thing.
So: this protocol is ours and it is not about us. The rubric does not mention any product. The twenty questions are specified as a composition rather than a list, so that no vendor, us included, can prepare against them. If a step in here looks like it was written so that only one product can pass it, that is a bug and we would like to hear about it.
Six things, all yours to bring.
A 48-hour test that spends its first day waiting on a firewall rule is measuring your change process. Settle these a week earlier, and be honest with yourself about how long they took, because that time is real and it belongs in the project plan even though it is not in the score.
| One source, reachable | Pick the system that carries the transaction. In grocery that is POS or the warehouse table it lands in. It has to be reachable from wherever the vendor runs, and confirming that is a networking question you should settle a week earlier. |
| A read-only credential, already issued | Scoped to named schemas. Issuing it is usually a ticket rather than a decision, and it is the single most common reason a two-day test becomes a two-week one. |
| Ten metrics, named in advance by you | Written down before any vendor sees them. If the vendor helps choose the list you are testing their demo, not your estate. |
| Twenty questions, written and sealed | Composed to the mix below, in your own words, before the test starts. Do not show them to the vendor in advance and do not add to them once you have seen an answer. |
| One person who can settle what a metric means | Available for the middle day. This is the binding constraint on every project of this kind and the protocol is built to expose it rather than hide it. |
| The numbers your existing reports produce | For the same ten metrics, over the same period. This is the answer key. Without it you are scoring plausibility, which is the thing that cannot be scored. |
Connect one source
The clock starts when the vendor receives the credential. Not when the call is booked, not when the contract is signed.
What you measure. Time to first answer: the elapsed minutes from credential handover to a correct answer to one question you can verify independently.
Procedure
- Hand over the credential and start the clock. Record the timestamp.
- The vendor connects the source and profiles it. You watch. Do not help.
- Ask one question with a known answer, for example units sold for one store last Tuesday.
- Record the elapsed time and whether the answer matched. A near-miss is a miss.
What to watch. Watch what happens when the schema is not what the vendor expected. Every estate has a column that is a string where it should be a date. A product that stops and names the problem is behaving correctly. One that silently coerces it is telling you what the next six months look like.
Define ten metrics
The step that actually decides the schedule. Loading data is easy and has been for a decade. Agreeing what the numbers mean is the work.
What you measure. Metric fidelity: how many of the ten reproduce the number your existing report produces, to the tolerance you set in advance.
Procedure
- Set the tolerance before you start. One percent is a reasonable default and fresh food metrics may need more.
- Give the vendor the list of ten metric names and nothing else. Not the formulas.
- Let them draft the definitions. Time the drafting separately from the correcting.
- Compare each to the answer key. Record match, near miss within tolerance, or wrong.
- For every wrong one, record whether the error was in the formula, the grain, the filter or the join.
What to watch. The interesting number is not how many were right first time. It is how long it took to correct a wrong one, and whether the correction stuck: a definition you fixed in one place and that then resolves the same way everywhere is a semantic layer, and a definition you fixed in one report is a report.
Run twenty agent queries
Your sealed list, unchanged. The mix matters more than the count, and it is specified below so a vendor cannot ask for an easier one.
What you measure. Answer quality, scored per question against the rubric. Score the two refusable questions on whether they were refused.
Procedure
- Run all twenty without intervening. Resist explaining a question that was misread. That misreading is data.
- Score each answer: correct, correct but unverifiable, wrong, or refused.
- For every answer, record whether you could see the SQL and the definition it resolved through.
- For the three ambiguous questions, record whether the system asked for clarification or guessed.
- For the two refusable questions, record whether it refused and whether it said why.
What to watch. A wrong answer that shows its working is recoverable. A right answer you cannot verify is worse than it looks, because it is indistinguishable from the wrong one that will arrive next week.
Composing the twenty
Written in your words, about your estate, sealed before the test. The mix is fixed so the test is comparable between vendors. The questions are not published, because a published list is a list to prepare against.
| Count | Kind | What it tests |
|---|---|---|
| 8 | Straightforward | One metric, one grain, one period, answerable from the source you connected. These establish the floor. A product that misses here is not in the evaluation. |
| 5 | Cross-source | Need a join between the connected source and a second one, or between two tables that are only related through a dimension. This is where a system that is really a chat wrapper over one table separates from one that is not. |
| 3 | Deliberately ambiguous | Use a term your own teams disagree about. "Shrink last month" is perfect if your commercial and supply chain teams mean different things by it. Scoring is on whether it asked, not on which answer it picked. |
| 2 | Must be refused | Ask for something outside the granted scope, for example a table you deliberately did not grant, or a column you masked. A correct answer here is a refusal that names why. |
| 2 | Contested definition | A metric where you know two of your existing reports disagree. Score on whether the system surfaced the conflict rather than silently picking one side. |
Break it on purpose
The four hours most evaluations skip, and the ones that predict what the deployment feels like in month six.
What you measure. Governance and reversibility, scored on what you can prove rather than what you are told.
Procedure
- Revoke access to one table at the source. Ask a question that needs it. The answer should be a refusal naming the table, not a wrong number and not a stack trace.
- Ask for something the scope never included. Record whether it was refused and whether the refusal named the scope.
- Pull the log for the whole test. Check you can answer: which identity asked, what definition resolved, what SQL ran, what tables it touched.
- Find the two refused questions from phase three in the log. If refusals are not logged, the log cannot evidence an access control.
- Revoke everything. Record what remains: copies of your data, cached results, and how long they persist.
What to watch. Ask for the log as a file rather than a screen. A log you can only read in the vendor's console is a log you cannot put in front of your auditor or join to anything else.
One hundred points, weighted by what fails deployments.
Governance carries twenty because it is the category that fails a rollout in month six rather than in week one. Metric fidelity carries twenty-five because it is the work. Speed carries twenty, which is less than the marketing in this category would imply and more than it deserves.
Time to first answer / 20
| Under 1 hour from credential to a verified correct answer | 20 |
| Under 4 hours | 15 |
| Under 24 hours | 8 |
| Under 48 hours | 3 |
| Not reached inside the window | 0 |
Metric fidelity / 25
| 2.5 points per metric that reproduces your existing number within tolerance, first draft | 25 |
| 1.5 points per metric correct after one round of correction | 15 |
| 0.5 points per metric correct only after you supplied the formula | 5 |
| Zero for any metric still wrong at hour 24 | 0 |
Answer quality across the twenty / 25
| 1 point per correct answer whose SQL and definition you could inspect | 1 |
| 0.5 points per correct answer you could not verify | 1 |
| 1 point per ambiguous question that triggered a clarification rather than a guess | 1 |
| Minus 1 point per confidently wrong answer with no working shown | 1 |
Governance / 20
| Both refusable questions refused, with the reason named | 8 |
| Revoking a table at the source produced a refusal rather than a wrong answer | 4 |
| The log answers all four questions: who, which definition, what SQL, which tables | 6 |
| Refusals appear in the log | 2 |
Reversibility / 10
| Revoking the credential ends all access immediately | 4 |
| You can enumerate exactly what data remains outside your systems | 4 |
| Retention of anything that remains is stated and configurable | 2 |
Answer quality is the one category scored per item rather than per band: add the per-answer points across the twenty and cap the category at twenty-five. The minus one for a confidently wrong answer with no working shown is deliberate and it is the only negative in the rubric. An unverifiable wrong answer is not a neutral outcome, it is the failure mode the whole exercise exists to detect.
What each band means.
Decide the bar before you run the test, not after you see the number. Writing it down in advance is the only defence against the score becoming an argument for whichever product someone had already chosen.
| Score | Verdict | What to do with it |
|---|---|---|
| 85 and above | Proceed to a paid pilot | It did the thing. Scope the pilot on the metrics that failed rather than repeating the ones that passed. |
| 70 to 84 | Proceed, with the gap written into the contract | Usually a governance or fidelity gap. Both are fixable and both cost time, so price that time now rather than discovering it in month three. |
| 50 to 69 | Not yet | Re-run in two quarters or narrow the scope to the part that scored. A score in this band nearly always means the semantic work is not done, and no amount of the vendor's effort substitutes for your side of it. |
| Below 50 | No | Whatever the demo showed, it did not survive contact with your estate. That is what the protocol is for. |
Bring your own twenty questions. We will not see them first.
Deploying Ward is free, so the protocol costs you the two days and the credential and nothing else. Phase four is the part we would rather you did not skip.
What the protocol points at.
The audit log, with the schema
Phase four asks whether the log answers four questions. This is the record that does, field by field, with the DDL.
Read the schema →Ten metrics, five of them hard
Fill rate, shrink, on-shelf availability, promo lift and waste, with formula, grain and edge cases. Half your list, if you are a grocer.
The definitions →dbt Semantic Layer
What runs on dbt Core, what needs a paid plan, where the compute lands, and who dbt SL is still right for.
Read the comparison →Snowflake Cortex Analyst
Semantic views and RBAC inheritance are strong. The costs are the model, the compute and a default worth checking today.
Read the comparison →Find out what your data has been hiding.
Tell us about your operation. We’ll show you the problems Ward catches, and the ones your current tools miss.