“What was allowed?” vs. “What actually happened?”
A common postmortem pattern:
The IAM policy said the service could read the bucket. The audit log confirmed an API call to read the bucket. Nobody could tell me which container, on which node, started by which deployment, on behalf of which user, made that call, or what it did with the data afterwards.
That gap, between what is allowed, what is logged, and what actually happened, is where most runtime security questions live. Closing it is an engineering problem: stitching workload-attributed eBPF runtime data to IAM, IdP, CloudTrail, and database catalogue telemetry to produce runtime identity attribution for every observed query and API call.
This post is the technical walkthrough of that stitching. The join keys, the sources that contribute each part of the chain, the structured fact you can build out the other end, and the detections that fall out of it.
Why Permissions and Logs Are Not Enough
Most security programs reason about runtime behavior through indirect evidence:
- IAM policies describe what an identity could do.
- CloudTrail / control-plane logs describe what API calls were made to the cloud provider.
- Database audit logs describe what the DB engine saw, from the DB’s perspective.
- Application logs describe what the developer chose to emit.
- Okta / IdP logs describe authentication events.
- Posture snapshots describe configuration at a point in time.
Each is useful. None of them, alone or together, answers the question that matters after an incident or before an access review:
Which workload, running which code, under which identity, touched which data, and where did that data go?
The reasons are concrete:
- IAM tells you what is allowed, not what was used. A policy with
s3:GetObject on a wide prefix says nothing about whether the workload reads one object per day or one million. Permission breadth ≠ usage. - Okta tells you who authenticated, not what the workload did downstream. Once a token is minted and passed to a service, Okta has no view into what queries that service ran, or whether the token’s scope matched what the workload actually needed.
- CloudTrail captures control-plane activity, but most of the interesting data-plane behavior is missing.
GetObject shows up. SELECT email, ssn FROM customers WHERE ... against an RDS instance does not, because that traffic flows directly between the workload and the database engine. - Database audit logs show the DB user, not the real caller. A pooled
app_user connection from a shared service may serve dozens of upstream identities. The DB sees one principal; the truth is fan-in. - Application logs are inconsistent. They depend on what developers chose to instrument, are routinely silenced for performance, and rarely capture identity propagation across hops.
None of these reconstruct the chain. They each describe one slice, in one vocabulary, with one identity model.
Starting Where the Protocol Intelligence Layer Ended
We have already covered the first half of this story in eBPF Data Monitoring: Building a Protocol Intelligence Layer for Production Traffic. That post is the prerequisite: kernel capture, connection tracking, protocol FSMs, wire-level parsing, DNS resolution, Kubernetes pod-and-service enrichment, PII detection. Assume all of that is in place.
What that pipeline produces, and where this post picks up, is a structured statement like:
Pod checkout-7f8a (namespace prod, service checkout) ran SELECT email, phone FROM customers WHERE id = ? against RDS endpoint customers-db-prod.…rds.amazonaws.com 412 times in the last hour. The result columns matched the PII classifier for email and phone.
That is a query-level fact, attributed to a workload. It is also where most “we have eBPF visibility now” deployments stop. The interesting security questions begin one layer up:
- Which identity was that pod running as? Was it the identity the workload was designed to run as?
- Was the access allowed broadly, or used narrowly? Was the IAM surface wider than the observed usage?
- Which human request, if any, set this in motion? Or was the workload acting on its own?
- Did the runtime behavior match the workload’s declared purpose, or drift from it?
The kernel does not answer any of these. Neither does the protocol layer. They are an enrichment and correlation problem against telemetry that lives entirely outside the data plane: IAM, IdP, control-plane audit, cloud metadata, schema and grant catalogues, request traces. The rest of this post is about that join.
The Sources That Fill In the Identity and Permission Gaps
The protocol intelligence layer already gave us the workload, the query, the destination, and the data class. The sources below add the parts it cannot see on its own: identity, permission, and intent. Each closes a specific blind spot. None is optional if the goal is end-to-end attribution.
- Workload-identity mapping (IRSA / Workload Identity / Pod Identity): the bridge from “pod with service account X” to “calls made as IAM role Y”. The kernel never sees this binding; it lives in the cloud provider’s token-exchange path. Without it, every observed cloud SDK call is anonymous at the principal level.
- IAM, evaluated as effective policy: not just the attached role document. Inline policies, managed policies, permission boundaries, SCPs, resource policies, and condition keys all combine into the real permission surface. The distinction matters because “broader than usage”, the privilege-drift signal, only makes sense against the effective surface, not the attached one.
- CloudTrail (management and data events where enabled): the control-plane corroboration of what the workload’s IAM identity actually called. Two reasons it earns its place even when eBPF is in front of the same traffic. First, it captures activity that never touched a workload at all: console actions, cross-account
AssumeRole chains, partner-account access, automation outside the cluster. Second, the delta between CloudTrail and eBPF is itself a signal. eBPF-observed calls that do not appear in CloudTrail point at telemetry gaps or unsigned access paths. CloudTrail entries with no workload-side counterpart point at access that bypassed the application entirely. - Cloud metadata (RDS, S3, KMS, VPC, account tags): classifies the other end of an observed connection. The protocol layer already extracted that a Postgres query went to
customers-db-prod.…rds.amazonaws.com. The cloud-metadata join turns that into instance class, parameter group, encryption posture, account, environment tag, and whether this resource is in scope for a given compliance boundary. - Okta / IdP: who authenticated, when, from where, with what MFA posture, into which app. On its own this is just session telemetry. Joined to a frontline service’s request log via session ID or JWT subject, it becomes the start of the upstream-identity chain that the workload layer eventually consumes.
- Database catalogue, not just PII detection: schema, column types, classification tags, and role grants. The protocol layer can flag that the bytes contained an email. The catalogue tells you which DB role had
SELECT on that column to begin with, and whether the app_user the query came in as is over-granted relative to what the application needs. - Service mesh / app traces (OTel spans, request IDs, JWT claims, baggage): identity propagation across internal hops. This is the most fragile source, because it depends on application instrumentation. It is also the only thing that lets you walk an observed DB query back through N service boundaries to the originating user request. Without it, the chain breaks at the first pooled-connection or shared-service-account boundary, which in most production systems is the very first hop.
Combined with eBPF, these sources produce a runtime fact like:
Pod checkout-7f8a (Deployment checkout, namespace prod) running as service account checkout-sa → IAM role prod-checkout → opened a TLS connection to RDS instance customers-db-prod → authenticated as DB user checkout_app → executed SELECT email, phone FROM customers WHERE id = ? 412 times in the last hour → on behalf of upstream Okta user alice@... via request req-abc → result rows contained columns classified as email, phone.
Every clause comes from a different source. Removing any one of them breaks attribution.
Detections That Fall Out of the Joined Chain: Privilege Drift, Identity Mismatch, Pooled-Connection Fan-In
Once you can reconstruct that chain, a class of questions that were previously unanswerable becomes routine:
- Identity usage: For each IAM role / service account, which workloads actually exercised it, against which resources, how often? Roles with zero usage over 30 days are deprecation candidates.
- Privilege drift: Compare the effective policy surface to the observed usage. A role with
s3:* on a bucket but only ever observed calling GetObject on one prefix is over-permissioned by a measurable amount, and a least-privilege policy can be auto-generated from the observed call set. - Identity vs. workload mismatch: A workload using a role it was not designed to use (e.g. the analytics pod assuming the checkout role via a misconfigured trust policy) shows up as an unexpected (workload, role) pair.
- Data access vs. purpose: Tag workloads with their intended data classes. A workload labelled
billing reading from a table classified as health_records is a deviation, regardless of whether the IAM policy formally allowed it. - Shared-DB-user fan-in: The DB sees one user; eBPF sees N upstream workloads. You can quantify how much identity is being lost at the pooled-connection boundary and decide whether per-workload DB credentials are worth the cost.
- Behavioral deviation: Same workload, same identity, sudden change in query shape, table set, or volume. Useful as a signal, not on its own a verdict.
The pattern is the same in every case: the IAM/Okta/log view tells you what was possible; the eBPF view tells you what the kernel saw; the join tells you what happened and whether it matched intent.
Honest Limitations
Limitations specific to the identity/permission/correlation layer, not the capture layer:
- Identity propagation is only as good as the application makes it. If upstream services do not propagate a user-identifying header, JWT, or trace context, no amount of runtime telemetry will recover the human identity. The chain breaks at the first hop that drops context, and most production systems have at least one such hop.
- Effective IAM evaluation is non-trivial. Resource policies, SCPs, permission boundaries, session policies, and condition keys interact in ways that
simulate-principal-policy and similar APIs get close to but not always right. “Permission used” is a measurable thing; “permission needed” is a model, and the model has error bars. - CloudTrail and eBPF do not agree on time. CloudTrail events arrive with delivery delay and ordering quirks; eBPF events are real-time per host. Joining them requires tolerance windows, which means a fraction of joins will be ambiguous or miss outright. Plan for it; do not assume a clean inner join.
- eBPF coverage stops at the host. Serverless (Lambda, Fargate, Cloud Run), managed databases, third-party SaaS: you cannot run eBPF inside any of them. CloudTrail, provider audit logs, and VPC Flow Logs are partial substitutes, and the substitution is rarely lossless. Calls made entirely outside the cluster show up only in control-plane sources, with no workload-side corroboration.
- Pooled connections collapse identity by design. A single
app_user DB connection serves many upstream callers. Even with perfect trace propagation upstream, the DB sees one principal. Attribution past that boundary is reconstructed from request-time correlation, not observed in the DB session. The reconstruction has gaps. - Behavioral baselines are noisy. Workloads change. Deployments shift query patterns. Any “deviation from intent” signal needs hysteresis, change-aware suppression, and a way for engineering to declare intent. Without those, it produces alert noise rather than signal.
None of these are reasons to skip the work. They are reasons to publish assumptions alongside conclusions and to be explicit about what is reconstructed versus assumed.
What the Stitched Chain Lets a Security Team Do
Once the join exists, the operational surface changes:
- Access reviews run against observed usage, not attached policy. The output of the join is a per-(role, resource, action) usage frequency over a window. Reviewers approve what was used, not what was theoretically allowed.
- Trust policies get narrowed by data, not guessed at. The set of (workload, role) pairs observed at runtime is the trust policy. Anything outside that set is either a misconfiguration or a use case nobody documented.
- Least-privilege policy generation becomes mechanical. The set of distinct (action, resource) pairs a role actually exercised over N days is the candidate policy. Compare to the effective policy; the difference is the safe-to-remove surface.
- DB grant audits stop relying on the DB user alone. With the application identity stitched to the DB session, you can audit “which application accessed which classified column”, not just “which DB user did”.
- Sensitive-data access has a real subject. Instead of “an
app_user query returned 412 emails”, the alert names the workload, the IAM role, the upstream service path, and (where instrumentation cooperates) the originating end-user request.
The join keys, in order of how often they are needed:
None of these joins is exotic. The work is plumbing: getting each source into a shape that survives the join, handling delivery skew (CloudTrail vs. eBPF), and tolerating the places where instrumentation drops context. The result is a single chain that ties a kernel-observed byte stream to an authenticated human (or an explicit absence of one), through every layer in between.
Key Takeaways
- eBPF is a substrate, not an outcome. A protocol intelligence layer on top of eBPF produces workload-attributed query and API events. Security-relevant questions begin one layer up.
- Identity, permission, and intent live outside the data plane. Workload identity bindings (IRSA / Workload Identity / Pod Identity), effective IAM evaluation, CloudTrail, cloud metadata, Okta / IdP, OTel traces, and DB catalogues are each required to close a specific blind spot.
- The work is six joins, not one platform. Workload ↔ identity, identity ↔ permission used, workload ↔ cloud resource, service hop ↔ user request, DB session ↔ application identity, column ↔ classification.
- Privilege drift becomes measurable as the set difference between effective permission and observed usage. A least-privilege policy candidate is the observed-usage subset.
- Pooled connections collapse identity at the DB boundary; attribution past that point is reconstructed from upstream traces, with documented gaps.
- Be explicit about fragility: identity-propagation gaps, effective-IAM error bars, CloudTrail vs eBPF time skew, host-only eBPF coverage, and baseline noise on intent-deviation signals.
TL;DR
- A protocol intelligence layer on top of eBPF gives you workload-attributed, query-level events. That is the starting point, not the destination.
- The identity, permission, and audit context needed to make those events actionable lives in IAM, IdP (Okta), CloudTrail, cloud metadata APIs, DB catalogues, and trace systems, not in the kernel.
- This post walks through the six joins (workload ↔ identity, identity ↔ permission used, workload ↔ cloud resource, service hop ↔ user request, DB session ↔ application identity, column ↔ classification) and the keys each one uses.
- The output is a single structured fact per observed query/API call that enables identity-usage audits, privilege-drift detection, identity-vs-workload mismatch alerts, pooled-connection fan-in analysis, and intent-deviation signals.
- We are explicit about where the join is fragile: identity propagation, effective-IAM error bars, CloudTrail vs eBPF time skew, host-only eBPF coverage, pooled connections, and baseline noise.
