Production Monitoring & Alerting — Reliability & Incident Response

Reliability requires visibility. You can't fix what you can't see. Our monitoring and alerting infrastructure provides comprehensive observability into widget health, performance, errors, and user experience — enabling proactive incident detection and rapid response.

Centralized error tracking

Every exception is captured, enriched with context, and forwarded to error tracking services (Sentry, Rollbar, Bugsnag, Application Insights). Captured data includes:

• Full stack trace with source maps for readable file/line numbers
• User context (visitor ID, session ID, conversation ID — no PII)
• Device and browser information
• Breadcrumb trail of user actions leading to the error
• Network request history (recent API calls, response codes)

Performance monitoring and metrics

Key metrics tracked by percentile (p50, p75, p90, p95, p99):

• Time to First Byte (TTFB) — backend API response latency
• First Contentful Paint (FCP) — how fast the widget becomes visible
• Time to Interactive (TTI) — when users can send messages
• Message render time — how long AI responses take to display
• Knowledge base retrieval latency — search performance

p95/p99 metrics ensure edge cases affecting small percentages of users are addressed.

Performance budgets prevent regressions

Define acceptable performance thresholds:

• "Widget must load in < 2 seconds on 3G connections"
• "AI responses must render in < 500ms"
• "Knowledge base search must return in < 1 second"

Monitoring alerts when real-world performance violates budgets.

Alerting rules and escalation policies

Thresholds for actionable events:

• Error rate > 1% — triggers warning
• Error rate > 5% — triggers critical page
• API latency p95 > 3 seconds — triggers investigation
• > 100 failed authentication attempts in 5 minutes — possible attack

Alerts route through on-call systems (PagerDuty, Opsgenie) with escalation policies and attached runbooks.

Health check endpoints for uptime monitoring

• /health — returns 200 OK if service is operational
• /health/deep — verifies database connectivity, knowledge base access, and AI service availability

Health checks run every 30–60 seconds from multiple geographic regions. SLA compliance is tracked automatically.

Proactive anomaly detection

Machine learning models detect unusual patterns:

• Sudden traffic spikes (viral content or DDoS attack?)
• Unusual error rates (recent deployment caused regression?)
• Geographic anomalies (service degradation in specific region?)
• Temporal patterns (performance degrades every night at 2 AM — why?)