Sentinel
Planned AI observability platform that will monitor agent reliability and catch drift. Not built yet (October 2026)
Sentinel is a design on my roadmap. This page sets out the problem it targets and how I plan to build it. It reports no results, because there are none yet.
The Problem
AI systems degrade silently. A model that performed at 95% accuracy last month might be at 78% today — and nobody notices until customers complain. Most teams monitor infrastructure (CPU, memory, uptime) but not AI behavior (response quality, hallucination rates, retrieval relevance).
The gap is observability. Not infrastructure monitoring — AI-specific observability that tracks what matters: is the system still giving good answers?
The Architecture
Sentinel is a planned observability platform purpose-built for AI agent pipelines. It will connect to any LLM-based system via lightweight instrumentation and provide three layers of monitoring:
Every LLM call, tool invocation, and agent decision will be captured as a structured trace through LangFuse integration. Traces will include full input/output pairs, latency, token usage, and custom metadata. The instrumentation has to stay light enough that nobody switches it off.
Automated evaluators will run on sampled traces using Pydantic AI for structured output parsing. Evaluators will check: factual grounding, response relevance, instruction adherence, and format compliance. Each evaluation will produce a typed score that feeds into trend analysis.
When quality metrics cross configurable thresholds, Sentinel will fire alerts. Not just "something is wrong" — specific alerts like "retrieval relevance for the finance domain dropped below 85% over the last 4 hours." Dashboards will be built on FastAPI endpoints with real-time WebSocket updates.
Technical Decisions
Structured, typed outputs. When an evaluator assesses response quality, I need a typed score object — not a string that might say "7/10" or "seven out of ten" or "pretty good." Pydantic AI enforces output schemas, making downstream aggregation reliable.
Open source and self-hostable. Sentinel is designed for enterprises that cannot send trace data to third-party SaaS. LangFuse will run in the customer's own infrastructure with full data sovereignty.
Trace data is relational at its core — spans belong to traces, traces belong to sessions, sessions belong to users. PostgreSQL handles this naturally with JSONB columns for flexible metadata and proper indexing for time-series queries.
What I Will Measure
Once Sentinel is built, these are the numbers it has to earn before this page reports any:
- Share of LLM calls captured as traces
- Tracing overhead added to each call
- Time from a drop in answer quality to an alert
- False alerts per week