Falco in Production: Runtime Security Without Drowning in Alerts

Falco takes about ten minutes to install. One Helm chart, and within a minute you have alerts flowing.

Then the second week arrives, the channel has four thousand messages in it, and someone quietly mutes it. By month three nobody remembers Falco is running.

This is the part the getting-started guides skip. Deploying runtime security is trivial; operationalising it is the actual work — and it’s mostly about deciding what you don’t want to hear about.

What Falco Actually Is

Falco is a CNCF graduated project, originally built by Sysdig. It parses Linux system calls from the kernel as they happen, runs that stream against a rules engine, and alerts when something matches.

A rule looks like this (simplified — the shipped version carries a few more guards):

- rule: Terminal shell in container
  desc: A shell was used as the entrypoint/exec target in a container
  condition: >
    spawned_process and container
    and shell_procs and proc.tty != 0
  output: >
    A shell was spawned in a container
    (user=%user.name container=%container.name image=%container.image.repository)
  priority: NOTICE
  tags: [container, shell, mitre_execution]

Falco adds container and Kubernetes metadata to the raw kernel event, so the alert tells you which pod, which namespace, which image — not just which PID.

Two drivers are supported today: the modern eBPF probe (the default, built on CO-RE, no kernel module needed) and a kernel module. Beyond syscalls, a plugin system pulls in other sources — Kubernetes audit logs, CloudTrail, Okta.

Why This Layer Exists

You probably already have scanners. Falco does something they structurally cannot.

Every layer above runtime evaluates an artefact. Falco evaluates behaviour.

That distinction matters because a workload can pass every pre-deployment gate and still be compromised at runtime — through a vulnerability nobody had a signature for, a leaked credential, or a supply-chain path that looked clean at scan time. That’s the same argument I made about why an SBOM alone won’t stop the next Log4j: knowing what’s in the box doesn’t tell you what the box does once it’s running.

Falco is the layer that notices a perfectly clean container suddenly pulling a binary and executing it.

The Noise Problem, Stated Honestly

Here’s the thing most write-ups won’t tell you, and which the Falco project itself says plainly:

«The maturity level of the rules… does not directly reflect their potential for generating noise in the adopters’ environment. This is due to the unique and constantly changing nature of each environment, especially in cloud environments, making it challenging to accurately predict the impact of rules.»

Read that again, because it’s the honest core of the whole topic. Nobody can tell you in advance which rules will be noisy for you. Not the project, not a vendor, not a blog post. Your normal is yours alone.

Consider “Terminal shell in container” — a genuinely valuable detection. Now consider what routinely spawns shells in a real cluster:

• your operators running kubectl exec to debug
• CI jobs using debug containers
• init containers whose entrypoint is /bin/sh -c
• health checks implemented as shell scripts
• backup jobs, migration jobs, cron jobs

The rule isn’t wrong. It’s describing something suspicious in general that is completely ordinary in your cluster. Multiply this across dozens of default rules and you get four thousand messages a week.

The noise is not a bug. It’s the unavoidable gap between generic detection logic and your specific environment. Which means closing that gap is your job, and it’s the job the tooling can’t do for you.

The Rules Maturity Framework

The project ships a classification that’s genuinely useful once you know it exists.

maturity_stable — the default set. Broad, system-level detections aligned with MITRE ATT&CK: remote code execution, container escapes, network pivots, privilege escalation, credential theft. Start here, and only here.

maturity_incubating — more specific detections. More value in the right context, considerably more effort to evaluate.

maturity_sandbox — experimental. Broader or narrower coverage, highest engineering cost.

The recommended progression from the docs:

1. Run stable rules only
2. Tune them against your real traffic, while measuring performance overhead
3. Once false positives are low and consistently so, add incubating or sandbox rules
4. Write custom rules early for what’s unique to you
5. Explore plugins if other event sources fit your ecosystem

Note what the framework does not claim: maturity describes adoption readiness, not quietness. A stable rule can still flood you.

Tuning Without Blinding Yourself

Four moves, in order of how much they help.

1. Watch before you alert

Run Falco and send everything to a log — not to a channel anyone is expected to read. Give it a full week, including a deploy, an incident, and a weekend.

You are not looking for attacks. You are building an inventory of your normal. That week of data is worth more than any amount of reasoning about which rules “should” be noisy.

2. Write exceptions, don’t disable rules

The reflex when a rule fires constantly is to switch it off. That’s how coverage silently disappears.

Falco supports exceptions and rule overrides precisely so you can carve out the known-good case while the detection stays alive for everything else:

# loaded AFTER the default rules file
- rule: Terminal shell in container
  exceptions:
    - name: known_debug_images
      fields: container.image.repository
      comps: in
      values: [company/debug-toolkit, company/migration-runner]
  override:
    exceptions: append

Two details that bite people here. The override block is not optional — without it Falco treats the entry as a full redefinition of the rule and errors out because desc, condition and output are missing. And load order matters: your custom file has to be listed after falco_rules.yaml in rules_files, otherwise the override applies to a rule that doesn’t exist yet.

Now the rule still fires for a shell in your payment service. It just stops firing for the debug image your team deliberately runs.

The docs also ask you to be specific — prefer actor and target over a single broad field, so an exception can’t be reused as a hiding place. Many default rules additionally expose user_known_* macros designed to be overridden for exactly this purpose, which is often the cleanest hook.

The discipline: every exception is a documented statement about your environment. Write down why, so the next person doesn’t have to guess whether it was a decision or an accident.

3. Route by severity, not by default

Sending everything to one Slack channel is the single fastest way to make Falco useless. Split it:

falcosidekick exists for exactly this — it takes Falco output and fans it out to more than 50 different destinations. Falco itself writes to stdout, a file, syslog, a spawned program or an HTTP endpoint; the routing logic lives in the forwarder.

The test is simple: would you wake someone at 3am for this? If not, it doesn’t belong on a pager. And if nothing would wake anyone, you have monitoring, not detection.

4. Write custom rules for your crown jewels

The docs recommend this early, and it’s the highest-signal work you can do.

Generic rules describe generic threats. But you know things Falco can’t: which paths hold your secrets, which service is the crown jewel, what its execution pattern looks like on a normal Tuesday.

- rule: Access to payment signing key
  desc: Anything reading the payment service signing key outside the signer
  condition: >
    open_read and fd.name startswith /etc/payments/keys
    and not proc.name in (payment-signer)
  output: >
    Signing key accessed by unexpected process
    (proc=%proc.name container=%container.name)
  priority: CRITICAL
  tags: [custom, crown_jewel]

One rule like this beats twenty generic ones. It has near-zero false positives because it encodes knowledge only you have — and if it fires, something is genuinely wrong.

One trap worth knowing: proc.name is truncated at 16 characters by the kernel, not by Falco. Match a longer binary name and the condition silently never fires. Use proc.exepath when the name is long.

The Performance Conversation

Falco processes a stream of kernel events, so its cost scales with syscall volume. A busy node doing millions of syscalls per second costs more than an idle one.

When the agent can’t keep up, it drops events — which is both a performance symptom and a detection gap. The docs have a dedicated troubleshooting page for exactly this, which tells you how common it is.

Three practical notes:

• Measure on your own workloads. Published benchmarks tell you about someone else’s cluster.
• Capture selectively. You don’t need every syscall to detect what you care about; tuning what you collect is usually cheaper than scaling the node.
• Treat the budget as real. The project frames security monitoring as having a limited budget in practice — because it does. An agent that degrades your workloads gets removed, and then you have no detection at all.

What “Done” Looks Like

The docs describe end-to-end operationalisation as detection triage plus pre-defined runbooks in your incident response workflow. That’s the honest bar, and it’s higher than “alerts are arriving”.

A realistic definition of working:

• The alert stream is small enough that a human still reads it in month three
• Every rule that pages someone has a runbook saying what to do
• Exceptions are documented, with a reason attached
• Somebody periodically verifies the pipeline still works end to end — Falco logging, transport, destination, triage
• Custom rules exist for the assets you’d actually lose sleep over

That last point about verification matters more than it sounds. A detection pipeline that silently broke three months ago looks identical to a quiet environment.

The Uncomfortable Truth

An alert nobody reads is worse than no alert, because it creates the illusion of coverage. You pass the audit question — “do you have runtime detection?” — while having none in practice.

So the real measure isn’t how many rules you enabled. It’s whether, when something happens, anyone notices within an hour.

Getting there means enabling fewer rules than you could, tuning them harder than feels necessary, and accepting that a quiet, trustworthy signal beats comprehensive coverage that everyone ignores.

Falco is a genuinely good tool, and the runtime layer is one almost nobody else covers. It just doesn’t come finished — it comes as raw capability that your environment has to shape.

If you’re building the surrounding platform, the same principle runs through secure-by-default GKE and platform engineering on Kubernetes: the control that works is the one people don’t have to remember to use — or in this case, the one they don’t learn to ignore.

The Bottom Line

Falco sees what your scanners and admission controllers structurally cannot: behaviour, as it happens. That makes it worth running.

But the default ruleset is a starting point, not a configuration. The project says outright that nobody can predict which rules will be noisy in your environment, which means the tuning work is not optional and not something you can outsource to a guide.

Observe for a week. Write exceptions instead of disabling rules. Route by severity so the pager stays meaningful. Add custom rules for the things only you know matter.

Aim for a signal your team still trusts in six months. That’s the whole game.