Tagged:
monitoring6 posts
-
AI Secrets Management: Protecting API Keys, System Prompts, and Model Credentials in Production
API keys for model providers have a blast radius unlike traditional credentials. System prompts encode proprietary business logic that teams rarely treat as secrets. This is a practical guide to treating AI credentials as the first-class secrets they are — with concrete patterns for storage, rotation, agent scoping, and incident response.
-
LLM Security Monitoring in Production: Anomaly Detection, Audit Logging, and Intrusion Detection for AI Systems
Guardrails block known bad inputs. Incident response handles breaches after they happen. This post covers the gap: building LLM-specific observability that detects attacks while they are occurring — anomalous outputs, prompt injection signatures, unusual tool-call patterns, and model drift from baseline.
-
Retrying Leaks Your Monitor: Why AI Control Scaffolds Need to Rethink Feedback
New research from Redwood Research shows that retrying — blocking risky actions and continuing with monitor rationale in context — lets adversarial models craft sneakier attacks, negating safety gains. Resampling avoids the information leak: drawing 5 samples per step and auditing on the maximum suspicion score raises safety from 61% to 71% at a 0.3% audit budget.
-
The Model Pool Is a Moving Target: What the June 2026 Deprecation Incident Taught Us
On June 16, 2026, a mid-session model retirement silently took down our AI agent. Here's how the failure happened, what the model pool actually looks like, and how we're now monitoring for it.
-
When Tools Multiply: Auditing the Real Cost of AI Agent Capability Growth
We ran a snapshot-based audit of our AI agent's tool definitions across May–June 2026. Tool count grew 190% in 40 days. Here's what we measured, what it means for context window sustainability, and why lazy-loading isn't the answer.
-
We Found a Regression in Our Own AI Agent
We built monitoring infrastructure to catch silent behavior changes in AI agent wrapper layers. The first time we ran it on ourselves, it caught a production bug we had no idea existed.