Analytics, Usage & Cost Visibility

End-to-end observability: trace IDs, token/cost accounting, per-org dashboards and alerts to understand usage, cost spikes, and ROI of AI features.

What Is This Feature?

Analytics is how you understand what your AI assistant is actually doing, how much it's costing, and whether it's delivering value. This feature covers the full observability stack: tracking every conversation and AI call end to end, calculating the cost of each request, rolling up usage by customer, and surfacing all of this in dashboards your team can act on. It's the foundation for product decisions, cost management, and customer billing.

Why It Matters to Your Business

Running an AI product is fundamentally different from running a traditional software product — AI calls cost money per request. Without detailed usage analytics, you can't answer basic but critical questions: Which customers are using the most resources? Is a particular assistant configuration more expensive than expected? Is overall cost growing faster than revenue?

Cost attribution. Know exactly which organization, which assistant, and even which type of query is driving your AI spend. This is essential for setting pricing, managing margins, and having informed conversations with high-usage customers.
Usage trends for product decisions. Are certain features being used heavily while others are ignored? Are response times trending up or down? Analytics turns these questions into answerable, data-backed insights.
Proactive cost management. Instead of discovering an unexpected bill at the end of the month, you see cost spikes as they happen and can act immediately — throttling a runaway integration, adjusting quotas, or reaching out to a customer.
Customer reporting. Some customers want to see their own usage data. Per-organization dashboards let them self-serve this information, reducing support load.

How It Works (No Technical Jargon)

Every conversation creates a chain of events that the analytics system tracks:

A request arrives. The system assigns it a unique tracking ID (called a trace ID) that follows the request through every step.
The AI model is called. The system records which model was used, how many "tokens" (units of text) were processed, and how long it took.
Cost is calculated. The token counts are mapped to dollar amounts using the current pricing from the AI provider. This happens automatically in the background.
The data is stored. Raw events are stored for short-term, detailed debugging. Aggregated daily totals per organization are stored long-term for trend analysis and billing.
Dashboards update. Within 24 hours of any activity, the dashboards reflect the latest usage and cost figures.

The Trace ID: Why It Matters

The trace ID is a thread that connects every piece of data about a single conversation. If a customer asks "why did this response come out wrong?" your team can find the trace ID and see the complete picture: what was retrieved from the knowledge base, what was sent to the AI, what the AI returned, and what post-processing happened. This makes debugging a specific bad response possible in minutes, not hours.

What You Can See in the Dashboard

For your team:

Total AI cost over time — daily, weekly, monthly, broken down by organization
Usage per organization — requests, tokens, cost, and trends
Top queries and assistants — what's being asked most, and by whom
Response quality indicators — latency, error rates, citation rates
Cost spike alerts — automated notifications when spending exceeds thresholds

Per-organization view (for customers):

Their own usage summary — how many conversations, how many requests
Cost breakdown if you choose to expose this
Historical trends over configurable date ranges

Cost Accuracy and Transparency

AI provider pricing changes periodically. The system maintains a pricing map that can be updated when providers adjust their rates, ensuring cost calculations stay accurate. If a particular request can't be mapped to a cost (e.g., due to a provider reporting anomaly), it's flagged as "pending" and recalculated once the data is available — you never silently lose cost data.

What to Expect on the Roadmap

The team is building:

End-to-end trace IDs attached to every request and AI call (estimated 2 weeks)
A cost aggregation job that calculates per-organization daily costs and a dashboard to display them (estimated 3 weeks)

Once live, you'll have the visibility to understand exactly what your AI platform costs, who's driving that cost, and whether you're getting the return you expect.