A leading sleep solutions retailer built an AI sales enablement assistant to give more than 6,000 store associates instant access to product knowledge, promotions, and coaching. Adoption grew fast. Visibility into production performance did not keep pace. Golden datasets and manual log reviews were too slow for live traffic, and thumbs-up/thumbs-down user feedback was too sparse to reveal complete visibility into production behavior. They needed an AI agent observability solution.
Picking a platform was the easy part. Grid Dynamics evaluated observability platforms against the retailer’s existing engineering workflows, telemetry, and governance processes, and chose Datadog LLM Observability because it treated the assistant’s LLM calls as first-class citizens inside traces, alerts, and dashboards the retailer already used, instead of routing that data into a separate tool nobody would check.
The harder part was deciding what to measure. Drawing on deep e-commerce delivery experience, the Grid Dynamics team defined a focused set of metrics tied to real business risk: failure-to-answer rates to expose knowledge base gaps, evaluator correctness to protect the assistant’s coaching and training integrity, and prompt injection detection to enforce security accountability. The team built custom LLM-as-a-judge evaluations through prompt engineering to turn those metrics into real signals, then instrumented the codebase to enrich every trace with LLM-specific context the platform didn’t capture by default.
That work changed how the retailer’s own reviewers operate. Instead of combing through thousands of log traces by hand, business reviewers now scan a dashboard that surfaces only what’s flagged as incorrect, cutting a full-log review down to a handful of exceptions.
Results in production:
- Lower latency and compute costs. End-to-end LangGraph tracing tracked latency and token usage across the assistant’s orchestration flow, helping engineers find and remove redundant LLM calls. The result: faster responses, less compute waste, and easier troubleshooting when something breaks.
- 93.9% evaluator accuracy. Roleplay evaluations verified as logically correct under deliberately strict scoring, catching cases where the coaching engine had assigned high scores to weak or incomplete answers before they reached a rep’s training record.
- 79.8% live answer rate. The remaining 20.2% of questions now feed a content roadmap that closes documented gaps in product knowledge, documentation, and internal resources.
- Real-time security detection. Prompt injection and instruction-override attempts are flagged as they happen, something the old workflow never surfaced.
- A defensible benchmark for what comes next. Every future model change has to clear this production baseline before it ships.
Manual reviews could not keep up with a 6,000-person workforce running on live AI. Continuous observability does.
See what AI agent observability could surface in your own AI rollout. Talk to Grid Dynamics about building a production baseline you can defend.
You might also like
