From Packets to Agent Chains: eBPF for AI Runtime Security

Kernel: Stable Connection Tracking

The kernel layer is identical to our database monitoring approach (detailed in our performance optimization post): eBPF programs attach process identity (TGID, container context), track socket direction (request vs response), and generate connection keys for chaining. The kernel emits stable anchors. Userspace builds meaning.

Userspace: FSMs for AI Protocols

We use the same FSM approach from database monitoring, but AI protocols add new challenges: streaming responses (SSE tokens arriving incrementally), tool calls in JSON (nested function definitions), and multi-provider variations (OpenAI vs Anthropic vs Azure OpenAI).

Here’s the OpenAI API FSM:

OpenAI API FSM (simplified)

IDLE → TLS_HANDSHAKE → HTTP2_CONNECTED → HEADERS_RECEIVED
  → CHAT_COMPLETION_REQUEST → DATA_STREAMING → RESPONSE_COMPLETE → IDLE

When a connection opens, we’re in IDLE. The TLS handshake moves us to HTTP2_CONNECTED. When we see HEADERS frames with :path=/v1/chat/completions, we transition to CHAT_COMPLETION_REQUEST. As DATA frames arrive, we buffer and parse JSON, extracting the model name, token counts, and tool definitions. When we see the response’s END_STREAM flag, we emit a complete semantic event and return to IDLE.

Pinecone gRPC FSM (simplified)

IDLE → GRPC_HANDSHAKE → SERVICE_READY → QUERY_REQUEST
  → QUERY_RESPONSE → RESULT_STREAMING → RESPONSE_COMPLETE → IDLE

Vector database protocols use gRPC with protobuf encoding. The FSM detects the gRPC service path (/pinecone.VectorService/Query), parses the protobuf-encoded request to extract embedding dimensions and namespace, then processes the streaming response to count results and detect PII in metadata.

The AI-specific win: FSMs handle streaming LLM responses where tokens arrive incrementally over seconds. Regex can’t maintain state across packet boundaries. FSMs can track partial JSON, buffer incomplete messages, and emit complete events when data: [DONE] arrives.

AI-specific sampling: Keep every call touching PII, tool chains, or external APIs. Sample routine LLM calls at 1/100. Reduction happens post-extraction by emitting semantic events instead of raw packets.

Wire Protocol and FSM

OpenAI API uses HTTP/2 over TLS with JSON payloads. Example request:

POST /v1/chat/completions
{
  "model": "gpt-4",
  "messages": [{"role": "user", "content": "Retrieve customer emails"}],
  "tools": [{"type": "function", "function": {"name": "query_database"}}]
}

FSM extracts semantics from HTTP/2 binary frames:

State: IDLE

• on TLS handshake to api.openai.com → CONNECTED
• on HTTP/2 SETTINGS frame → HTTP2_READY

State: HTTP2_READY

• on HEADERS frame with :path=/v1/chat/completions → CHAT_REQUEST
• on HEADERS frame with :path=/v1/embeddings → EMBEDDING_REQUEST

State: CHAT_REQUEST

• on DATA frames → reconstruct JSON body → extract:
  • model name (gpt-4, gpt-3.5-turbo, etc.)
  • message history (system, user, assistant roles)
  • tools/functions defined
  • temperature, max_tokens (generation params)
• on END_STREAM flag → REQUEST_COMPLETE

State: REQUEST_COMPLETE

• on response HEADERS frame → RESPONSE_STREAMING
• on response DATA frames → reconstruct JSON → extract:
  • model output (completion text or tool calls)
  • token usage (prompt_tokens, completion_tokens)
  • tool_calls array (function names and arguments)
  • finish_reason (stop, length, tool_calls, content_filter)
• on END_STREAM → emit semantic event → IDLE

What We Extract

For each LLM API call: protocol, model, tokens (prompt/completion/total), tool_calls, PII presence, security signals (prompt injection patterns, anomalous tool use)

For each vector query: protocol, embedding_dim, namespace, top_k, PII in metadata, security signals (overly broad queries, namespace violations)

For each workflow: chain reconstruction, PII propagation tracking, cost attribution, policy violations

Aggregator: Chain Reconstruction

The aggregator’s job is unique to AI monitoring: reconstruct multi-step agent workflows. When OpenClaw queries Postgres, sends results to GPT-4, then creates a Jira ticket, the aggregator correlates these events (even across pods) into a single chain. It tracks PII propagation (customer email from DB → LLM context → Jira → ❌ pastebin) and detects violations traditional tools miss.

Example: Reconstructing an AI Agent Chain from Runtime Signals

Let’s look at a real scenario: an engineering team using OpenClaw for customer support operations. A support engineer instructs the AI assistant to “help resolve ticket for customer #12345,” and OpenClaw autonomously handles it by orchestrating multiple tools and services across their infrastructure. Here’s what we observe at the runtime layer:

Process tree:

The collector observed 7 sequential connections from a single OpenClaw worker pod: two OpenAI API calls, a PostgreSQL query, a Pinecone vector search, a Jira API post, a Slack webhook, and finally a curl upload to personal-pastebin.com. A child process compressed the full conversation history, including PII from every prior step, into a gzip file before that final upload.

Security Questions We Can Now Answer

With full chain reconstruction, security teams can answer questions that were previously out of reach:

1. Was this chain appropriate for support intent?

The first 6 steps look normal: query customer data, retrieve context, generate response, create ticket. But step 7 (compressing the conversation and uploading to an unknown external domain) is a clear anomaly. The same tools that enable legitimate support also enabled exfiltration. Without chain visibility, you’d see 7 authorized operations. With it, you see a policy violation.

Alert triggered: “Agent accessed PII and uploaded to unauthorized external destination”

2. What PII was accessed and where did it propagate?

Traditional DLP might catch step 7 if it can inspect the payload. But it won’t tell you the PII came from database queries in steps 2-3, was augmented by LLM in step 4, and should never have reached step 7.

3. Did the agent stay within authorized tools?

Individual tool authorizations all passed: OpenClaw has read access to the customer database, the company uses OpenAI API for AI features, and Jira API calls are part of the support workflow. But step 7 used curl to upload to a domain not in the allowed destination list. Every authentication check passed. The workflow outcome violated policy.

4. Was this autonomous or human-supervised?

All 7 steps were autonomous. The LLM decided which tools to invoke. No human approval was required. When you combine broad tool access with autonomous decision-making and PII handling, you need behavioral monitoring, not just access control.

Risk score: HIGH (autonomous execution + PII handling + unauthorized external destination)

1) AI Tool Chaining Visibility (Execution Graph)

Traditional APM tells you about service dependencies. AI observability needs to go deeper: which tools did the agent use, in what order, and how did data flow between them?

2) Intent Signals from AI Workflow Patterns

Agents can be authorized for every individual action but still produce inappropriate outcomes. The risk emerges from the sequence and destination of actions, not from any single permission.

By analyzing workflow patterns, we can distinguish normal behavior from anomalies:

Normal customer support workflow:

User request → LLM (plan) → Query DB (1 row) → Vector search (5 docs)
→ LLM (synthesize) → Create ticket → Notify via Slack

Characteristics: Small data scope (single customer), authorized destinations only, expected tool sequence for support intent.

Bulk data exfiltration via AI agent:

User request → LLM (plan) → Query DB (WHERE 1=1, 10k rows)
→ Loop 50 times { Vector search (top_k=1000) → File write }
→ Compress → Upload to personal S3

Characteristics: Large data scope, loops (unusual for support), file I/O with compression (preparation for export), unauthorized external destination. Every individual step might be authorized, but the chain is a violation.

3) Policy Surfaces for Agentic AI Security

Traditional access control policies ask “can this role access this resource?” That model breaks down with agents because agents need broad access to do their job. Every individual permission check passes. The policy violation is in the outcome.

Chain-level policies shift the question to “did this workflow make sense?”

Data access scope & destinations: Querying one customer record for a support ticket is expected; pulling 10,000 in a loop is not. Sending data to approved SaaS tools (Jira, Slack) is part of the workflow; uploads to personal cloud storage or unknown domains are not, regardless of whether the connection succeeds.

Tool sequencing & volume: A RAG workflow (vector search → LLM) is a normal pattern. An LLM call triggering a database write followed by an external POST is worth a look. Passing a few hundred tokens to an LLM for summarization is routine; 50,000 tokens of customer records in a single call is not.

Timing & autonomy: 500 LLM calls in 10 minutes likely means a runaway loop or scripted attack. Database access at 3 AM from a business-hours agent is worth investigating. When broad tool access combines with fully autonomous decision-making and PII handling, behavioral monitoring matters more than access control.

4) Real-Time Detection and Post-Facto Investigation

Architecture Designed for Scale

Performance Summary

Benchmark Results

Internal testing and load benchmarks validate performance characteristics:

Total end-to-end latency for complete chain reconstruction: <500ms

Performance Design Principles

Our team has deep expertise in eBPF, protocol parsing, and distributed systems. The same technology stack powers our database activity monitoring capabilities at scale.