Large-language models have transformed how millions interact with products : from customer support to code generation to legal document analysis.

These new engagement models invite users through a meaningfully different product journey. Every application now speaks English. Understanding user behavior is essential to building great products.

How should a product manager gauge the customer experience? Lots of new challenges arise :

• First, the models are non-deterministic which means sometimes the same question phrased a bit differently produces unexpected results.
• Second, users can input anything they like which can lead to model drift & unexpected outputs potentially including might include PII (personally identifiable information) or other sensitive content.
• Third, how should a PM think about the user journey through a conversation? What is a successful interaction? It’s no longer straightforward clicking through five screens.

Context.ai is a product analytics platform for LLM-powered applications. Context.ai gives product builders an understanding of how users engage with their products and visibility into product performance, so they can build better experiences.

Context.ai detects the topics of conversations ingested for analytics, and then rates the user satisfaction for each topic. Product builders can understand how users navigate their product and identify stronger or weaker product performance areas.

In the video, a PM navigates through the UI to discover the most popular themes, delve through individual transcripts. Context also enables blocking & filtering of particular topics, tracks particular keywords , & manages PII.

Context.ai is integrated with Langchain to simplify deployment.

When we met Henry & Alex, the co-founders of Context, we met a team who had worked on some of the largest products globally, including managing conversations & safety on YouTube & working on some of Google’s most significant APIs. Their backgrounds, working to solve similar problems, map well to helping PMs understand the user journey through new LLM-enabled products.

We’re thrilled to co-lead Context.ai’s seed with Google Ventures. As LLMs & chat become a predominant new UX, instrumenting the user experience will become essential to shipping great products.

If you’re building an LLM-enabled product & looking to understand user behavior & improve those journeys, try Context.ai.