Every founder or CTO I talk to has the same opening line: 'We want to use AI.' What follows that sentence is almost never a well-defined problem. It's usually a vague direction — speed up support, make sense of our data, automate the thing our ops team does manually every morning. That gap between 'we want AI' and 'here is the actual problem we need to solve' is where most AI projects fail before a single line of code gets written. I've spent the last two years doing AI consulting work alongside full-stack development, and the pattern is consistent enough that I want to write it down honestly.
TL;DR — AI consulting engagements succeed when they start with a measurable problem baseline, not a technology choice. Define the metric first, choose the tool second. Every successful AI deployment I have seen inverted the typical order.
What 'Messy' Actually Means in Practice
When a client says their data is 'a bit messy,' they usually mean one of a few things. Sometimes it means five years of customer records exported to a spreadsheet that three different people have edited manually, with no consistent date format and a 'notes' column that reads like a group chat. Sometimes it means the core business logic lives entirely in one senior employee's head and has never been written down. Sometimes it means the system they need to integrate with has no API — just a legacy desktop app from 2009 that only runs on Windows XP.
I've hit all three. The honest truth is that 'messy' is not a blocker — it's just the starting condition. But you have to go in with eyes open, because the mess is what determines scope, timeline, and whether AI is even the right tool.
- Bad data: inconsistent schemas, missing fields, duplicates, free-text where structured data should be
- No APIs: vendor lock-in, legacy systems, or simply a tool that predates the concept of integration
- Tribal knowledge: processes that work because one person knows the exceptions, undocumented edge cases
- Unclear success criteria: 'faster' and 'smarter' are not KPIs — clients often don't know what good looks like yet
- Infrastructure gaps: no logging, no monitoring, no staging environment, prod-only deploys
The single most common reason AI projects fail is not the model. It's that nobody cleaned the data before the model touched it, so the model confidently automated the wrong thing at scale.
The Consulting Process: Discovery Before Code
The first thing I do with any new client is refuse to write code for at least two weeks. That sounds counterproductive. It saves projects. The discovery phase is where you find out whether the problem is actually solvable with AI, what data exists and in what shape, and what the client will actually measure success against.
Week 1: Discovery
Discovery means sitting in on real workflows, not just the demo version. I ask to see the actual Notion doc, the actual spreadsheet, the actual Slack thread where decisions get made. I interview the person who does the manual work — not the manager who describes the manual work. Those are often completely different accounts of the same process.
- 01Shadow the workflow end-to-end — watch, don't design yet
- 02Identify every data source touched: files, databases, SaaS tools, email, spreadsheets
- 03List every manual decision point and what information drives that decision
- 04Ask: what breaks when the key person is on holiday? That's where the fragility is
- 05Document what 'done well' looks like — get specific, get examples
Week 2: Scoping and Spike
After discovery I write a scope document that is deliberately opinionated. I tell the client what I think the actual problem is (which is often different from what they said), what a realistic AI solution covers, and — importantly — what it does not cover. I include a risk register. I include a data quality assessment. If the data is too bad to build on, I say so before anyone writes a contract.
The spike is a throwaway prototype. The goal is not to impress — it's to surface the hard problems before they hit a production timeline. A one-week spike on a retrieval pipeline will tell you more about what's feasible than a month of planning.
A Real Example: NestJS RAG Pipeline for Support Triage
One client ran a SaaS product with a support team that spent 60% of their time answering questions already covered in their documentation. The ask was: 'Can AI answer these?' The real problem, discovered in week one, was that their documentation was scattered across a Confluence wiki, a Notion workspace, a set of Loom videos, and a folder of Word docs nobody had touched since 2022. Three sources had contradictions. One source had been deprecated but not marked as such.
The solution was a RAG (Retrieval-Augmented Generation) pipeline: ingest the documentation into a vector store, retrieve relevant chunks at query time, generate an answer grounded in those chunks. Here's a simplified version of the NestJS ingestion service I built:
// src/ingestion/document-ingestion.service.ts
import { Injectable, Logger } from '@nestjs/common';
import { InjectRepository } from '@nestjs/typeorm';
import { Repository } from 'typeorm';
import { OpenAIEmbeddings } from '@langchain/openai';
import { RecursiveCharacterTextSplitter } from 'langchain/text_splitter';
import { DocumentChunk } from './entities/document-chunk.entity';
@Injectable()
export class DocumentIngestionService {
private readonly logger = new Logger(DocumentIngestionService.name);
private readonly embeddings = new OpenAIEmbeddings({ model: 'text-embedding-3-small' });
private readonly splitter = new RecursiveCharacterTextSplitter({
chunkSize: 512,
chunkOverlap: 64,
});
constructor(
@InjectRepository(DocumentChunk)
private readonly chunkRepo: Repository<DocumentChunk>,
) {}
async ingestDocument(source: string, rawText: string, metadata: Record<string, string>) {
// Split into overlapping chunks to preserve context across boundaries
const chunks = await this.splitter.splitText(rawText);
this.logger.log(`Ingesting ${chunks.length} chunks from ${source}`);
for (const chunk of chunks) {
const [embedding] = await this.embeddings.embedDocuments([chunk]);
await this.chunkRepo.save({
source,
content: chunk,
embedding, // stored as pgvector column
metadata,
ingestedAt: new Date(),
});
}
}
}Before this service touched any document, I ran a content audit script that flagged contradictions (same heading, different answers across sources) and staleness (pages not updated in over 18 months). The client's content team spent three days resolving those before ingestion. That three days of non-AI work was the highest-leverage thing in the entire project.
// src/retrieval/query.service.ts
import { Injectable } from '@nestjs/common';
import { InjectRepository } from '@nestjs/typeorm';
import { Repository } from 'typeorm';
import { OpenAIEmbeddings } from '@langchain/openai';
import { OpenAI } from '@langchain/openai';
import { DocumentChunk } from '../ingestion/entities/document-chunk.entity';
@Injectable()
export class QueryService {
private readonly embeddings = new OpenAIEmbeddings({ model: 'text-embedding-3-small' });
private readonly llm = new OpenAI({ model: 'gpt-4o-mini', temperature: 0.2 });
constructor(
@InjectRepository(DocumentChunk)
private readonly chunkRepo: Repository<DocumentChunk>,
) {}
async answer(question: string): Promise<{ answer: string; sources: string[] }> {
// Embed the question and find nearest chunks via pgvector cosine similarity
const [queryEmbedding] = await this.embeddings.embedDocuments([question]);
const chunks: DocumentChunk[] = await this.chunkRepo
.createQueryBuilder('chunk')
.orderBy('chunk.embedding <=> :embedding', 'ASC') // pgvector operator
.setParameter('embedding', JSON.stringify(queryEmbedding))
.limit(5)
.getMany();
if (!chunks.length) {
return { answer: "I don't have documentation covering that topic yet.", sources: [] };
}
const context = chunks.map((c) => c.content).join('
---
');
const sources = [...new Set(chunks.map((c) => c.source))];
const prompt = `You are a support assistant. Answer the question using ONLY the context below.
If the context doesn't contain enough information, say so — do not guess.
Context:
${context}
Question: ${question}`;
const answer = await this.llm.invoke(prompt);
return { answer: answer.content as string, sources };
}
}The outcome after six weeks: support ticket volume for documentation questions dropped by 54%. The remaining 46% were genuinely novel questions the docs didn't cover — which was itself valuable signal that drove a documentation sprint. The team didn't lose their jobs; they stopped spending half their day copy-pasting from Confluence.
When AI Is NOT the Answer
This is the section that builds the most credibility in a consulting relationship, and also the one most consultants skip because they want to keep the scope large. I've told clients 'you don't need AI for this' more times than I can count. Here's what that actually looks like.
- If a rule-based system covers 95% of cases — write the rules. A decision tree in code is cheaper, faster, and more auditable than a model that achieves 97% at 10x the cost and complexity.
- If the data doesn't exist yet — build the data collection layer first. No model fixes absent data. A six-month collection phase before the AI phase is not a failure; it's good engineering.
- If the bottleneck is a broken process, not a slow one — fix the process. AI automating a broken workflow produces broken results faster.
- If the team won't trust the output — and in regulated industries or high-stakes decisions this is common — a human-in-the-loop workflow with better tooling often beats full automation on every measure that matters.
- If the volume doesn't justify it — if you're processing 50 documents a month, a well-designed form and a part-time VA costs a tenth of the infra and maintenance burden.
I've walked away from paid work because the client's actual problem was a communication breakdown between two teams. No model fixes that. Saying so early is the most valuable thing you can do for a client relationship.
Measuring Success Beyond 'It's Faster Now'
Speed is a proxy metric. What companies actually care about is cost, quality, and risk. 'Faster' matters only if the thing being sped up maps to one of those. Before any project goes live, I establish a measurement framework with the client that answers three questions.
1. What did this cost before, and what does it cost now?
This means counting actual hours, not estimated hours. I ask clients to log the manual workflow for two weeks before the project starts. That gives a baseline. After launch, I track the same workflow. The delta is the labour saving. Add infra and API costs to the AI side. That's your ROI number — not a projected one, a measured one.
2. Did quality go up, hold, or drop?
For the support triage project, quality meant: did the answer resolve the ticket without escalation? We tracked that rate before and after. It went up by 18 percentage points. But we also tracked 'incorrect answer rate' — cases where the AI confidently answered wrong. That number started at 7% and after two prompt and chunking iterations came down to 1.2%. Tracking both sides of quality is what earns trust from the team using the system.
3. What is the failure mode, and how often does it fire?
Every AI system has a failure mode. The question is whether it fails loudly or quietly. Quiet failures — where the system produces a confident wrong answer — are the dangerous kind. I instrument every pipeline with an uncertainty signal and a fallback path. If the retrieval confidence is below a threshold, the system escalates to a human rather than guessing. That fallback rate is itself a metric: if it's above 20%, the system needs more work before you call it production-ready.
// Simplified confidence gating — escalate instead of guessing
async function answerWithFallback(question: string, queryService: QueryService) {
const topChunks = await queryService.retrieveTopChunks(question, { limit: 5 });
// Cosine similarity scores come back as distances (0 = identical, 2 = opposite)
const bestScore = topChunks[0]?.similarityScore ?? 2;
const CONFIDENCE_THRESHOLD = 0.35; // tuned empirically per deployment
if (bestScore > CONFIDENCE_THRESHOLD) {
// Low confidence — escalate to human queue rather than hallucinate
await escalationQueue.push({ question, reason: 'low_retrieval_confidence', score: bestScore });
return { answer: null, escalated: true };
}
return queryService.generateAnswer(question, topChunks);
}What to Look for in an AI Consultant
If you're evaluating whether to hire someone for AI consulting work, the questions that matter are not about which models they've used. They are: Do they push back on requirements? Do they talk about data before they talk about models? Do they have a process for discovery, or do they jump straight to a demo? Do they have examples of projects they declined or descoped because AI wasn't the right fit?
The person you want is the one who is more interested in your actual problem than in selling you a particular technical approach. That's also the person most likely to deliver something that still works six months after the engagement ends.
Need an AI engineer for your project? I build RAG systems, LLM integrations, AI agents, and SaaS backends. Reach me at prayagbagthariya.vercel.app/hire-me or prayag129787@gmail.com.
Frequently Asked Questions
How long does a typical AI consulting engagement take before we see results?
For most operational automation problems, expect four to eight weeks from discovery to a working prototype in a staging environment. Production-ready systems with proper monitoring, fallback handling, and user training typically take three to four months. Timelines stretch when data quality work is needed upfront — and it almost always is. The spike prototype at week two will give you a much more accurate estimate than any initial proposal.
Our data is a mess. Does that mean AI can't help us?
Not at all — it just means data work comes before AI work. I've never walked into a client environment where the data was genuinely clean. The question is how much cleanup is required and whether that work is scoped into the project. Sometimes a two-week data normalisation sprint unlocks a problem that looked unsolvable. The risk is treating data quality as an afterthought rather than a prerequisite.
How do we avoid becoming dependent on one AI model or vendor?
Design your integration layer to be model-agnostic from day one. That means abstracting the LLM call behind an interface rather than calling OpenAI or Anthropic directly in your business logic. In NestJS I typically create a LLMProviderService that accepts a provider config at runtime — you can swap GPT-4o for Claude Sonnet or a self-hosted model without changing the application logic. Your embeddings and vector store choice matter more for lock-in than your generative model.
What's the difference between AI automation and AI consulting?
AI automation is the implementation work — building the pipeline, writing the integration, deploying the system. AI consulting is the upstream work that determines whether the automation is worth building at all. Good consulting includes workflow analysis, data assessment, build vs. buy decisions, risk identification, and success metric definition. You can get automation without consulting, but you're much more likely to automate the wrong thing or hit avoidable problems mid-build.
How do we measure ROI on an AI consulting engagement?
ROI = (labour hours saved x hourly cost + quality improvement value) minus (AI infra costs + consultant fees + internal team time). The hard part is establishing an honest baseline before the project starts. I ask clients to log the manual process in real-time for two weeks before we begin — not estimate it, log it. That baseline number is what makes the ROI calculation defensible to a board or finance team rather than being a projection someone made up in a pitch deck.
Continue reading
Get updates directly in your Google Search feed
Add this publication as a preferred source to highlight articles in Top Stories and AI Overviews.


