Most AI consultancy engagements that disappoint share a single root cause: the buyer didn't ask the right questions before signing. Every consultancy can show a polished deck and name-drop a few clients. The hard part is separating the firms that ship working systems from the firms that ship strategy memos and walk away.
This is a working buyer's framework. A scoring rubric, the red flags to watch for, and a small set of questions that reveal what kind of partner you're really hiring.
A scoring rubric for AI consultancies
Score each candidate from 1 (worst case) to 5 (best case) across nine dimensions. Anything under 30/45 total is a real concern. Anything over 38 is a strong fit.
| Criterion | 1 / 5 looks like | 5 / 5 looks like |
|---|---|---|
| Discovery vs. delivery balance | 6–8 weeks of discovery before any code | 1–2 weeks of focused discovery, then a working prototype |
| What gets delivered | A strategy deck and recommendations | Working software, deployed in your environment |
| Team seniority | A PM fronting offshore juniors you never meet | Senior engineers doing the work directly, named on the contract |
| AI evaluation & quality | "We'll test it before launch" | Custom eval suites and structured logging built in from day one |
| Model choice & vendor neutrality | Locked into one provider regardless of fit | Claude, ChatGPT, Gemini evaluated per use case with clear rationale |
| Integration approach | One-off custom integrations for every tool | MCP servers, reusable patterns, agents that reach into your systems cleanly |
| Code & IP ownership | Licensed platform you must keep paying to access | You own every line of code, on your accounts, from day one |
| Ongoing maintenance | Hand-off then unavailable | Optional retainer OR a clean handoff with real documentation |
| Risk transparency | "Nothing should go wrong if you follow our process" | Names specific risks upfront with mitigation plans |
Red flags to watch for
Any one of these alone isn't disqualifying. Three or more is.
- "We're excited about AI." Excitement isn't capability. Ask for specifics.
- Massive teams with unclear roles. Usually means you're paying for layered management.
- AI as a buzzword. No specific tools, frameworks, or model names mentioned.
- No mention of evaluation. If they can't tell you how they know the AI is working in production, they don't know either.
- Vague code ownership. "We'll work that out in the SOW" is a no.
- Hourly billing with no upper bound. Outcomes-based pricing aligns incentives. Hourly does the opposite.
- No honest disqualifiers. A consultancy that says it's right for every problem is right for none.
Six questions that reveal posture
The scoring rubric covers what to look for. These six questions tell you who you're actually dealing with. Ask all of them in a single conversation and pay attention to whether the answers are specific, honest, and grounded in real work.
FAQ
How long should I spend evaluating AI consultancies?
For a 4–16 week engagement, two to four weeks of evaluation is reasonable. Talk to three partners minimum, score them against the rubric above, and ask each one a question you already know the answer to (to check whether they bluff or admit they don't know).
What's a fair price range to expect?
It depends on scope. As a rough frame: a focused 4–6 week AI adoption assessment usually runs in the low five figures. A 4–12 week agents and automations build usually runs in the mid five to low six figures. An 8–16 week custom software replatform usually runs in the mid-to-high six figures. Outcomes-based pricing should be the norm. Firms that quote hourly with no upper bound are a red flag.
Should I run an RFP?
RFPs are useful when you need to compare apples-to-apples on a well-scoped problem. They're counterproductive when you're still figuring out what to build — they reward partners who write good documents, not partners who build good software. For AI work, a paid two-week discovery engagement with one or two finalists usually tells you more than an RFP ever will.
Should I ask for references?
Yes. Ask each reference three specific questions: "What surprised you about working with this firm?", "What would you do differently?", and "Would you hire them again for a different project?" The third question is the most honest signal you'll get.
What if my team isn't technical enough to evaluate AI-specific answers?
Bring in an independent advisor for the evaluation conversations. A one-to-two hour consult with someone senior who has actually shipped AI systems will cost far less than picking the wrong consultancy.
If you're looking for an AI consultancy that delivers working systems, where senior people do the work, and you own everything that gets shipped — FM might be the right fit. Most engagements start with a 30-minute scoping call. No decks, no hard sell.




