Teaching Machines to Listen

Invoca's AI Studio let customers train custom LLM models on their call data, building signals that could automatically detect outcomes specific to their business: appointments booked, sales converted, frustrations expressed. The feature shipped in early 2025. Accuracy was high. Customers who got it working loved it.

But getting it working was the problem.

The v1 workflow dropped users straight into manual training: review calls one at a time, mark them as true or false examples. Each round required evaluating 20 true and 20 false calls. It was meticulous, manual, and demanding. And it assumed users already knew exactly what they wanted to find. Many of them thought they did. Then they started reviewing calls and realized their understanding shifted with every edge case they hit. The workflow was simultaneously a training tool and a discovery tool, and it had only been designed to be the first.

I worked with PM Zach Cohen to scope a research effort. We interviewed four Invoca team members who spent their days guiding customers through AI signal creation. They lived inside the workflow's gaps every day, building signals on behalf of customers, interpreting ambiguous definitions, coaching people through a tool that was supposed to be self-service.

The pattern was immediate and unanimous. Invoca had quietly built an entire human infrastructure around the workflow's weaknesses. Every new AI Gold customer received 8 hours of dedicated time from analytics services. For larger enterprise accounts like T-Mobile, the arrangement was contracted hours yearly with a dedicated analyst handling all signal creation. The product worked, but only with a person standing next to it explaining what to do.

The reframe: the workflow had two distinct failure points, at opposite ends.

The front door was missing. What made a good use case for an AI signal? What didn't? The workflow had no answer. One customer trained for 13 rounds and still couldn't get acceptable accuracy. When an analyst stepped in, scoped the signal properly, and reached 90 to 95% accuracy by round four. The difference wasn't the technology. It was knowing what to aim at.

The back door was duct-taped shut. After launching a signal, the product gave an "estimated accuracy" score but couldn't tell users how it actually performed in production. Real-world accuracy could swing 5 to 10% from the training prediction. Analytics team members ran accuracy tests by hand, pulling random call samples, listening to them, recording results in spreadsheets.

The middle worked. Without guided entry and automated validation, the whole system depended on human scaffolding at both ends.

The concept validated cleanly. Research surfaced the "bookends problem" framing that became the product strategy lens. Stakeholders aligned on the four-stage approach. The interactive Figma Make prototype told the story across multiple stakeholder presentations and earned buy-in for the next phase.

I left Invoca before the concept moved into build. The work mattered beyond what shipped. Research exposed a structural problem (that the product's self-service promise was being subsidized by an invisible human layer) and the concept addressed it at the system level, not with patches. The bookends framing carried forward as a shared lens the analytics services team and PMs continued using when scoping signal work for new customers.

This is the hardest kind of project to put in a portfolio. The one that was right but didn't finish. If I were picking it up tomorrow, I'd push harder on two things. First, the validation step needed user research with external customers. Internal SMEs are real users but bring expertise that masks usability problems. The concept needs to work for the customer training their first signal without an analyst guiding them through it. Second, the LLM-as-a-Judge interface and validation feedback system were both still in active iteration. Those are the moments where design either builds trust or erodes it.