# Felix Linux dataplane — Design Felix's Linux dataplane is a **single codebase** (`InternalDataplane` in `dataplane/linux/`) switched between **iptables, nftables and eBPF** modes — `BPFEnabled`/`NFTablesMode` select which managers and behaviours are wired into the *same* object. All three modes share the manager/driver model, the main event loop, the `OnUpdate`/`apply()` cycle, and the restart-and-resync doctrine. This doc owns that shared architecture (all three modes) plus the `*tables` (iptables/nftables)-specific parts — the `Table` abstraction, rule generation and dispatch chains, IP sets. eBPF mode reuses the shared architecture but adds its own managers (notably `bpfEndpointManager`), BPF maps and packet path, documented in the `bpf-*` family (start at [`bpf-overview.md`](./bpf-overview.md)); a BPF change typically needs *both* docs. Windows is a separate dataplane, covered here only as a contrast. Read this file before editing `felix/dataplane/linux/`, `felix/iptables/`, `felix/nftables/`, `felix/generictables/`, `felix/rules/`, `felix/ipsets/`, `felix/markbits/`, `felix/routetable/`, `felix/routerule/`, or `felix/vxlanfdb/` (plus the `bpf-*` family for BPF-specific files). The input boundary — the protobuf messages the dataplane receives — is the other end of the contract in [`calc-graph.md`](./calc-graph.md); the [dataplane API section below](#the-dataplane-api-calc-graph--dataplane-contract) is where that contract is written down. Build/test commands are in [`felix/CLAUDE.md`](../CLAUDE.md); the whole-Felix overview is in [`felix/DESIGN.md`](../DESIGN.md). ## Conventions - "WEP"/"HEP" = workload/host endpoint. "Local" = hosted on this node. - "`*tables`" means the legacy netfilter dataplane, **iptables or nftables** — any non-BPF Linux dataplane Felix can program. Where a statement is backend-specific the backend is named. - "**Manager**" and "**driver**" are the two layers this doc keeps distinguishing — see Layer roles. File paths are repo-relative; type/function/field names are cited, line numbers omitted. ## Layer roles The calc graph → managers (convert) → drivers (reconcile) layering, and the dataplane's other jobs — reacting to interface changes, detecting drift, reporting programming status — are described in [`DESIGN.md` §1](../DESIGN.md#dataplane-managers-and-drivers). This section covers where the manager/driver boundary flexes. The split is not rigid. Some managers are **vertically integrated** (manager and reconciliation logic in one object); others are split with shared reconciliation logic in a driver. The `bpfEndpointManager` is the extreme case — a combined manager+driver whose reconciliation is complex enough that the two roles are tightly coupled. When you can, prefer the split: let a driver absorb the resync complexity behind a declarative "here is the desired state" API and keep the manager simple (see [Restart, resync and mark-and-sweep](#restart-resync-and-mark-and-sweep)). ### Review notes for this section - A new feature that programs kernel state belongs in a manager (conversion) and, if it needs reconciliation, a driver — not smeared across the event loop. Prefer the declarative manager + reconciling driver split; reach for the integrated `bpfEndpointManager` style only when the coupling is genuinely irreducible. ## The OnUpdate / apply() split Every manager implements the `Manager` interface (`dataplane/linux/int_dataplane.go`): ```go type Manager interface { OnUpdate(protoBufMsg any) CompleteDeferredWork() error } ``` (Extended interfaces: `ManagerWithRouteTables`, `ManagerWithRouteRules` expose route syncers to the loop; `UpdateBatchResolver` lets a manager resolve cross-manager state before any programming begins.) **`OnUpdate` must be cheap and must not perform kernel I/O** (no netlink call, `iptables-restore`, subprocess fork, or BPF map write). It *may* push desired state into the in-memory `ipsets`/ `generictables.Table` objects, which *queue* the change (the `Manager` doc-comment permits this; `masqManager.OnUpdate` does exactly this via `AddMembers`/`UpdateChain`). The pattern: stash the message and **mark the affected resources dirty**; all reconciliation and kernel mutation happen later in `CompleteDeferredWork()`, walking the dirty set. Why the split: 1. **Batching is far cheaper.** Many operations cost the same for one change or many — legacy iptables ships the *whole* ruleset to userspace and back per change; an IP-set update forks a subprocess. Doing the heavy work once per `apply()` over a coalesced batch (Felix throttles `apply()` under load so batches grow) is the difference between keeping up and not. 2. **It separates safe-anytime work from must-wait work.** In-memory updates are safe before in-sync; touching the kernel is not (see [resync](#restart-resync-and-mark-and-sweep)). ### Review notes for this section - Any kernel access (netlink call, `iptables-restore`, subprocess fork, BPF map write) in an `OnUpdate` is a bug. It belongs in `CompleteDeferredWork`. Flag it. - `OnUpdate` that does work proportional to anything other than the size of the single message is suspect — the point is to be cheap and defer. ## The apply() event loop `InternalDataplane.apply()` (`dataplane/linux/int_dataplane.go`) is the throttled reconciliation cycle. The message-receive loop fans each protobuf message out to every manager's `OnUpdate`, then schedules an `apply()`; under load, applies are rate-limited so messages coalesce into bigger batches. `apply()` runs in a deliberate order, because dataplane resources have dependencies on each other — most importantly **iptables rules reference IP sets**, and the kernel refuses both to create a rule referencing an unknown IP set and to delete an in-use IP set. The order is: 1. Clear `dataplaneNeedsSync`; it will be re-set if anything fails. 2. **Resolution pass** — give every manager (via `UpdateBatchResolver`) a chance to resolve cross-manager state from the batch. This can produce an update from one manager to another (e.g. an endpoint manager telling the BPF endpoint manager about a HEP) that **must** land before either starts programming. 3. **Programming pass** — call `CompleteDeferredWork()` on every manager. 4. **XDP** — applied inline right after the programming pass (not a queued resync), every `apply()`; only its `QueueResync` is gated on `forceXDPRefresh`. This is Felix's **legacy** XDP — untracked-policy XDP layered on iptables mode (`xdpState`), no longer enhanced — not the modern XDP path in the proper BPF dataplane (see the `bpf-*` design family). Don't confuse the two. 5. Handle any popped **refresh timers** by **queueing resyncs**: `forceRouteRefresh` resyncs the route tables, the routing rules, **and the VXLAN FDBs**; `forceIPSetsRefresh` resyncs the IP sets. 6. **Create/update IP sets** — but **defer IP set deletions** until after the tables are updated. 7. Update VXLAN FDB and link-address entries. 8. **Update route tables and routing rules in parallel** (their own goroutines), to overlap their latency with the table work. 9. **Update `*tables`**, now that referenced IP sets exist. 10. **Delete the deferred IP sets**, now that no rule references them. 11. **Join** the parallel route/rule work before returning. If any step fails, the manager/driver keeps its pending state, the loop sets `dataplaneNeedsSync`, and `apply()` will run again. The [failure philosophy](#failure-philosophy) below governs how far a single failure is allowed to halt the rest of the cycle. The IP-set ordering in steps 6/9/10 is **the dataplane half** of the cross-layer "never reference an IP set before it's programmed" invariant; the calc graph's flush order is the other half (see [IP sets](#ip-sets)). ### Review notes for this section - A change to the `apply()` ordering must preserve: IP set creates before `*tables`; IP set deletes after `*tables`; the resolution pass before the programming pass. Re-ordering these silently breaks the dependency invariants and only fails under specific timings. - New per-`apply()` work that isn't gated on a dirty flag (i.e. runs even when nothing changed) erodes the throttling benefit. Walk the dirty set, don't rescan the world every cycle. ## Failure philosophy Error handling in the loop is, by the maintainers' own assessment, a relatively weak area of the architecture — treat changes here with care. The governing principles: - **Minimise blast radius.** Where there is no ordering constraint, a failure programming one resource must not abort the rest. If programming route A fails, still attempt B, C, D; keep A dirty; retry on the next cycle. Most drivers work this way. - **Block when continuing would violate a dependency or a security interlock.** Sometimes you *must* stop: if resource A can't be programmed and the next manager will program something that depends on A — or proceeding out of order would break a security interlock (e.g. opening traffic before its policy is in place) — the loop has to hold off rather than press on. - **Fail closed, and ultimately panic, where limping on is dangerous.** It is not safe to keep running if `*tables` programming **consistently** fails, because that can leave the node open. After retries are exhausted, Felix gives up and panics rather than run indefinitely in an unknown, possibly insecure state. ### Refresh timers vs error-triggered resync These are distinct mechanisms with overlapping effect: - **Error-triggered resync**: a failed `apply()` leaves work dirty and re-runs soon. The re-run is paced by the leaky-bucket `applyThrottle` (with a ~10s `retryTicker` as a backstop), not by exponential backoff; exponential backoff is only used *inside* the iptables `Table.Apply()` loop. This recovers from transient failures. - **Periodic refresh** (`forceRouteRefresh`, `forceIPSetsRefresh`, XDP): on a timer, queue a resync on the drivers even when nothing is known to be wrong. This is the belt-and-braces defence against **drift Felix didn't cause and wasn't told about** — the drift-detection job noted in [`DESIGN.md` §1](../DESIGN.md#dataplane-managers-and-drivers). Most drivers resync everything in one pass; the legacy ipsets driver satisfies a periodic refresh *incrementally* over several apply loops, because re-listing every set at once is too slow (see [IP sets](#ip-sets)). ### Review notes for this section - A change that lets `apply()` press past a failure must confirm it isn't crossing a dependency or security interlock — the case where blocking is mandatory. - A change that swallows a `*tables` programming error (rather than keeping it dirty / eventually failing loudly) risks leaving the node silently open. The fail-closed-then-panic behaviour for persistent `*tables` failure is intentional; don't soften it without a strong argument. ## Restart, resync and mark-and-sweep This doctrine shapes every driver and is the part most often gotten wrong when adding a feature that creates kernel state. **Felix must be restartable at any moment** (upgrade, config change, crash) and, on restart, **resync with the dataplane and converge to the current desired state with minimal disruption** — including cleaning up resources created by a *previous* Felix instance whose datastore state may have been completely different. The restarted Felix has **no memory** of what the old one did and gets **no** "resource X was deleted" event for state the old Felix created but the new datastore no longer wants. Two consequences: 1. **Cleanup is deferred until in-sync.** The dataplane does not touch the kernel until it receives the first datastore in-sync signal, and the first `apply()` after that is what triggers cleanup. If it swept earlier it would delete state it simply hadn't been told about yet. (This is why the calc graph must never fabricate in-sync — see [`calc-graph.md` → In-sync semantics](./calc-graph.md#in-sync-semantics).) Several drivers are architected so the *first* `Apply()`/resync call performs the read-back-and-reconcile. 2. **Every driver must identify its own resources.** To mark-and- sweep — keep what's still wanted, delete the orphans — a driver must be able to look at the live kernel state and decide "this one is mine and unwanted" **without** the original datastore state and **without** a delete edge-trigger — including state written by an *earlier Felix version*, whose naming or format may have changed across the upgrade. Therefore: **any feature that creates a new kind of kernel resource must, up front, design how a freshly-restarted Felix will recognise that resource as Calico's, for later cleanup.** This is a first-class design question, not an afterthought. ### How each subsystem identifies "ours" The kernel subsystems differ a lot in what they support, which is why the identification mechanism differs per driver: | Subsystem | How Felix recognises its own state | |---|---| | iptables (`iptables/`) | A **rule comment with our prefix and a hash of the input rule**. Needed because `iptables-save` output does **not** round-trip — the kernel re-canonicalises some constructs and the tools reformat others (the one concrete documented case is TCP-flag matches, per `iptables/actions.go`; MARK/CONNMARK are rendered to round-trip). The comment lets Felix (i) identify Calico rules even outside Calico-owned chains, and (ii) detect drift: read-back hash ≠ desired hash ⇒ reprogram. (Does not defend against malicious tampering that preserves the comment — out of threat model.) | | nftables (`nftables/`) | Inherited the iptables hash/prefix approach for porting ease. Inside Calico's own table it isn't strictly needed — **if it's in our table, it's ours** — but it is what picks out state a previous iptables-nft Felix left in the shared tables (see [cleaning up after the other backend](#cleaning-up-after-the-other-backend)). | | ip rules (`routerule/`) | No marking support. Identified by **the tables they jump to being Felix-owned**. Imperfect — a config change can confuse it. | | routes (`routetable/`) | Heuristic, because the first implementation didn't uniformly use the route `proto` field (it should have): **in a Felix-owned table ⇒ ours; carries our proto ⇒ ours; points down a `cali`-owned veth ⇒ ours**; etc. The classifier is the `OwnershipPolicy` interface — `MainTableOwnershipPolicy.RouteIsOurs`/`IfaceIsOurs` in `routetable/ownershippol/`. (Not to be confused with `RouteClass`, which is a same-CIDR conflict tie-breaker among *desired* routes, not an ownership test.) | | IP sets, iptables chains, veths | Identified by **name prefix** (`cali`...). | ### Review notes for this section - **A key review question for any PR that creates new dataplane state:** "How does a freshly-restarted Felix, with no memory and possibly a totally different datastore, recognise this as ours to sweep — with no delete event and no prior state?" If the PR doesn't answer it, it has a latent leak across restart. The menu of mechanisms is the table above (hash-comment / owned-table / owned-proto / name-prefix); pick one and wire in the read-back. - A change that makes a driver mutate the kernel before in-sync defeats the deferred-cleanup safety and risks deleting not-yet-seen state. - Among Felix's *own* producers, `(RouteClass, ifaceName)` is the ownership key: `RouteTable.SetRoutes()` **replaces** the whole desired set for that pair, so two managers that write to the same pair silently delete each other's routes. Each independent producer therefore needs its own class — which is why the same-subnet routes (shared parent device) and the IPAM-block drop routes (shared `InterfaceNone`) are split per encapsulation type. ### Fail closed while Felix isn't running Recognising "our" resources is not only about cleanup — it is also **security-critical**, because the dataplane has to keep doing the right thing in the windows when Felix is **down, restarting, or behind**. The motivating race: - Calc-graph updates can be delayed (Felix busy, restarting, or crashed). - The CNI plugin creates a new workload's `cali*` veth and plugs in the pod **before** that endpoint round-trips through the datastore back to Felix — so for a moment the interface exists but Felix has no policy for it. Therefore the iptables/nftables dispatch chains (the [dispatch trie/map](#rules-generation-dispatch-chains-and-mark-bits)) **fail closed**: they **drop traffic to/from any `cali*` interface that isn't explicitly allow-listed**. This is a Felix responsibility, not the CNI plugin's: the CNI plugin relies on Felix having *already* programmed that catch-all drop rule, so a freshly-plugged veth carries no traffic until Felix programs its policy — and the rule stays in the kernel even if Felix is stopped entirely. The general doctrine, which applies to any dataplane change: **think about what happens if Felix crashes or stops at this exact point.** Existing, already-secured traffic must keep flowing; anything that can't yet be secured properly must fail closed, not fall open. ## The dataplane API (calc graph → dataplane contract) This section is the one place the calc-graph→dataplane contract is written down; [`calc-graph.md`](./calc-graph.md) links here for the consumer view, and documents the producer-side ordering machinery (the `EventSequencer`). - **Transport.** The interface is the set of `proto.*` messages in [`felix/proto/`](../proto/), delivered to managers' `OnUpdate`. It exists because Felix was split Python→Go along this seam during a rewrite; the protobuf encoding is an artefact of that history. - **Mostly internal, one external consumer.** The API is essentially internal to Felix, with **one friendly out-of-tree consumer: the VPP dataplane**, which consumes the protobuf-encoded stream. There is **no formal stability guarantee**, but changes aren't made willy-nilly — the convention is to give the VPP team a heads-up ("we're refactoring this, would it break you?") before changing the wire shape. - **Ordering guarantees the dataplane may rely on.** The `EventSequencer` flushes in dependency-safe order, so the dataplane may assume **references arrive before referents and are removed after them**: IP sets before policies, policies/profiles before the endpoints that reference them, VTEPs before routes. The authoritative ordering and its rationale are in [`calc-graph.md` → Flush order is the dependency contract](./calc-graph.md#flush-order-is-the-dependency-contract). - **In-sync.** No kernel mutation before the first in-sync; see [Restart, resync and mark-and-sweep](#restart-resync-and-mark-and-sweep). - **The exceptions.** The "references before referents" rule has sanctioned exceptions for genuinely-missing resources (e.g. the fail-safe deny-all profile), documented on the producer side in [`calc-graph.md` → The missing-resource tension](./calc-graph.md#the-missing-resource-tension); a dataplane that consumes a pass-through signal owns the receiving half. ### Review notes for this section - A change to a `proto.*` message shape (field added/removed/ repurposed, message semantics changed) is a change to this contract: update this section and [`calc-graph.md`](./calc-graph.md), and consider the VPP consumer — flag it for a heads-up. - A dataplane change that starts relying on a *new* ordering property not guaranteed by the flush order is a contract change, not a local change — it must be backed by an `EventSequencer` guarantee, not by incidental current behaviour. ## The `*tables` Table abstraction The iptables (`iptables/table.go`) and nftables (`nftables/`) `Table` types, over the shared `generictables.Table` interface, own reconciliation of `*tables` chains and rules. The model: - **Felix owns named `cali-*` chains outright** and reconciles them to the desired contents. Into kernel-owned chains it **inserts** (or appends) only its own jump rules and recognises them by hash comment (see [identification](#how-each-subsystem-identifies-ours)). - **Desired vs actual is reconciled by rule hash.** Each rule carries a hash of its intended content (`generictables.RuleHasher`); on resync Felix reads back, compares hashes, and reprograms only what differs — giving minimal-delta, non-disruptive updates. - **`iptables-restore` is used for performance and atomicity.** Applying the whole update through one `iptables-restore` is much faster than issuing individual `iptables` calls, and lands as a single atomic per-table transaction (see the goals comment in `iptables/table.go`). Note this does *not* make full rewrites free: Felix still computes the hash delta and reprograms only changed rules, partly to avoid resetting iptables packet/byte counters on rules that didn't change. - **nftables rebuilds the table atomically after repeated failures.** When `Apply()` has failed several times in a row, the nftables backend recreates the table in case the change it is trying to make is incompatible with the current state. The `delete table` and `add table` go in the *same* transaction as the rewrite of every chain and rule, and the sets and maps in the table are marked for reprogramming at the same time. A failed attempt therefore leaves the existing ruleset alone rather than stripping the base chains and their hooks, which would leave traffic unfiltered. iptables has no equivalent; its retry loop only backs off. ### iptables vs nftables: parity and divergence The two backends are kept behaviourally aligned, but parity is a **deliberate decision, not an automatic requirement**: - A new feature that's trivially the same function on both ⇒ implement on both; that's the path of least resistance. - A feature with significant per-backend differences, or absent from one backend, ⇒ a design decision to agree, not a default. Expect more nftables-only (and BPF-only) features over time. - nftables is gradually migrating to **nft-native constructs** (e.g. match-action maps) for performance/scale wins, leaving iptables users no worse off. ### nftables flowtable offload (nftables only) `NFTablesFlowTableOffload` programs a `flowtable` object plus a `flow offload` rule in filter FORWARD, handing established flows to the kernel's software fast path. The fast path fires from the **ingress** hook of a member device and short-circuits straight to the output device, so an offloaded flow **never reaches FORWARD or POSTROUTING**. Anything Felix renders into those hooks stops applying for the life of the flowtable entry. Two invariants follow: - **The offload rule sits ahead of the dispatch jumps.** FORWARD jumps to `cali-flow-offload` on `RELATED,ESTABLISHED` only, so `NEW` and `INVALID` packets still traverse policy and pay for neither the jump nor the chain. Per-endpoint chains accept established traffic before any NFLOG rule, so policy attribution in flow logs is unaffected; connection byte counts come from `nf_conntrack_acct` rather than from Felix. - **The offload rule excludes endpoints whose features need those hooks, by IP.** `flowtableExclusionManager` in [`flowtable_mgr.go`](../dataplane/linux/flowtable_mgr.go) maintains the `no-flow-offload` IP set, holding the IPs of endpoints with DSCP marking (rendered into mangle POSTROUTING) or a connection or packet rate limit (rendered into the endpoint's filter chain), and the offload rule in `cali-flow-offload` matches neither source nor destination in that set. Bandwidth QoS is enforced by tc on the veth, which the fast path still traverses, so it does not disqualify an endpoint. Keeping the endpoint's veth out of the flowtable device set is **not** a substitute: the offload rule creates the entry whichever devices the flow uses, and the fast path fires from the *ingress* device, so a plain pod talking to a connection-limited pod still short-circuits at its own veth. Only the reply direction would be protected. Membership is also gated on the interface being up, in both the endpoint manager (workload veths) and [`flowtable_mgr.go`](../dataplane/linux/flowtable_mgr.go) (overlay and pattern-matched host devices). nft rejects the whole transaction if a flowtable names a device the kernel doesn't have, which takes down the entire table. ### Cleaning up after the other backend A node can switch backends across a Felix restart, so Felix sweeps every dataplane it isn't currently programming. There are three, and `iptablesBackend: Auto` can move a node between the last two on its own, since detection keys off kube-proxy's chains and so depends on boot ordering. The shared tables (`filter`/`nat`/`mangle`/`raw`) exist twice in the kernel, once per iptables backend; this doc calls them the **legacy copies** and the **nftables copies**. | Dataplane | Swept by | Needed when | |---|---|---| | Calico's own nftables table | that table's `Table`, which deletes it wholesale (safe: nothing else writes there) | iptables mode, either backend | | the nftables copies of the shared tables, where `iptables-nft` writes | `nftables.IPTablesNFTCleanup` | nftables mode, or iptables mode on iptables-legacy | | the legacy copies of the shared tables, where `iptables-legacy` writes | `iptables.Table` pinned to iptables-legacy, `CleanupOnly` | nftables mode, or iptables mode on iptables-nft | Four constraints on the sweep: - The nftables copies are read over **netlink**, not with `iptables-nft-save`, which refuses a table holding anything iptables can't express — a neighbouring tool's native nft rules are enough. Only **base chains** are read for rules, the only chains Felix ever inserted into, which keeps the dumps off kube-proxy's thousands of chains. The chain list is one dump of the whole family (netlink has no per-table filter), so the sweep is paced by the refresh interval rather than run every apply. - A `CleanupOnly` table never panics: the backend it names may have no kernel support here, in which case nothing of ours is in it. It retries on a latch rather than every apply, so an unreadable backend doesn't burn a retry budget each time round the loop. - The legacy tables are built only when **both** sets of binaries are present. Otherwise `FindBestBinary` falls back to the default `iptables`: with the legacy binaries missing the sweep would land on the backend Felix is programming, and with the nft ones missing the tables Felix programs are themselves the legacy copies. The same check gates the nft view in iptables mode. - They are also built only when the legacy modules are already loaded, since `iptables-legacy-save` would autoload them onto a node running pure nftables. `environment.DetectBackend` shares that check, so backend detection never probes a backend the kernel hasn't got. Cleanup **never terminates**. A shared table can be written again at any point (kube-proxy restarting, say), so one clean read proves nothing about the next. ### Review notes for this section - A PR that renders a new rule into filter FORWARD or mangle POSTROUTING for a subset of endpoints has to decide whether those endpoints can still be offloaded. If the rule has to run, add them to the `no-flow-offload` set, or return from `cali-flow-offload` for the traffic it matches, and cover it in `fv/flowtable_test.go`. - A PR adding `*tables` rule semantics must first decide the iptables/nftables story explicitly (both? nft-only? — see above), and should carry FV coverage in the relevant mode(s) (`make fv` and/or `make fv-nft`). - A change to how rules are hashed or how Calico rules are recognised touches the restart/resync identification mechanism — review it against [mark-and-sweep](#restart-resync-and-mark-and-sweep), not just the happy path. - A change to what Felix writes into the shared tables needs a matching change to the other mode's cleanup, or a node that switches backends keeps the stale rules indefinitely. ## Rules generation, dispatch chains and mark bits [`felix/rules/`](../rules/) is the `*tables` rule-rendering layer: it converts lists of Calico-internal rules/endpoints/etc into concrete `*tables` rules. - **Dispatch chains avoid O(n-endpoints) per-packet cost.** Evaluating an iptables rule costs on the order of a microsecond (rough estimate; no in-repo benchmark), so a flat chain of 100+ per-endpoint dispatch rules is a real per-packet tax. This really is per-*packet*, not per-connection: the conntrack accept lives in the per-endpoint chain, downstream of dispatch, so even established flows traverse the dispatch chains; it cannot be hoisted above dispatch without breaking setups that carry non-Calico rules. Felix builds a **shallow (typically single-level) tree of dispatch chains**, binning endpoints by the next character after the common interface-name prefix (`sortAndDivideEndpointNamesToPrefixTree` / `buildSingleDispatchChainTree` in `rules/dispatch.go`): roughly one branch per distinct next-character rather than one rule per endpoint. This is the `*tables` analogue of the BPF fast-path discipline: keep per-packet work sub-linear in the number of local endpoints. The dispatch chains also **fail closed**: a `cali*` interface that isn't in the trie is dropped, which is the security-critical default that protects not-yet-known workloads (see [Fail closed while Felix isn't running](#fail-closed-while-felix-isnt-running)). - **Mark-bit allocation.** Marks are a scarce, shared resource. `MarkBitsManager` (`felix/markbits/`, e.g. `NextSingleBitMark`) allocates bits for `*tables` modes from the configured range. (BPF uses a *fixed*, congested range whose individual bits are managed in a BPF header file — see the BPF design family.) An allocation that exhausts the range, or collides with bits another subsystem expects, is a startup/runtime failure. ### Connection transition logging `LogConnectionTransitions` (`*tables` modes only; enum, `Disabled` by default, `FirstResponseAfterLog` enables) extends the policy `Log` action with one follow-up log per logged connection, recording the first observed response — `-est` (first genuine reply), `-rst` (TCP RST, i.e. connection refused) or `-icmp-err` (related ICMP error) — so that a user reading the kernel log can tell whether a logged connection was ever answered. No follow-up log means no response was seen. Mechanism and invariants: - **One connmark bit** (`MarkConnStateLog` in `rules.Config`) means "matched a Log rule; no response seen yet". It is allocated from `MarkBitsManager` *conditionally* (only when the feature is enabled, so the endpoint-mark block isn't shrunk otherwise) and *before* the endpoint-mark block grab. Reserved from packet-mark space because the Log rules below also use it as a scratch packet mark, and so user `CONNMARK --save-mark` rules can't corrupt it. - **Setting the bit** (`rules/policy.go`): a rendered `Log` rule is followed by a `CONNMARK` rule with the same match. When `LogActionRateLimit` is set, both must instead hang off a single rate-limit decision, so that a connection gets a response log iff its initial log was emitted: clear the bit in the packet mark; re-evaluate the match under the limit and set it; then LOG and CONNMARK, each matching that bit. The leading clear is load-bearing — an earlier `Log` rule (or a user restore-mark rule) may have left the bit set, which would log a packet the limiter rejected. The scratch marks can't be used for this: the rule's own match may depend on them (`matchBlockBuilder`). Untracked (raw-table) policies skip the whole thing; CONNMARK has no effect on NOTRACKed packets. - **Checking the bit**: `appendConntrackRules` (`rules/endpoints.go`) prepends, ahead of the RELATED,ESTABLISHED accept rules, a jump to the shared `cali-log-conn` chain matching on the connmark bit AND ctstate RELATED,ESTABLISHED. The ctstate match is load-bearing: same-direction packets (e.g. a retransmitted SYN) still carry the bit but are ctstate NEW and must not be logged as a response. The first packet that reaches the check in RELATED/ESTABLISHED state is necessarily the first response. This lands in every tracked endpoint chain (workload and host, filter and mangle) except the preDNAT mangle chains, which skip it: mangle `cali-PREROUTING` accepts RELATED,ESTABLISHED first, so it could never match there. One static-chain special case: replies to workload→host connections leave via filter `cali-OUTPUT` and RETURN at the workload-interface rules without traversing any per-endpoint chain, so the same check rule is inserted at the top of `cali-OUTPUT`. - **The `cali-log-conn` chain** (`rules/static.go`, emitted into both the filter and mangle static chains, per IP version) has three mutually exclusive branches — TCP RST, ICMP-protocol + ctstate RELATED, catch-all established — each a LOG / clear-bit / RETURN triplet. LOG is non-terminating, so each branch must RETURN before the next branch's LOG to keep it to exactly one log per packet. The ICMP branch matches the ICMP protocol explicitly so only real ICMP errors get the `-icmp-err` log (conntrack associates ICMP errors with the *original* connection's entry, which is why they carry its connmark; other RELATED flows, e.g. helper children, fall through to the established branch). The chain needs no rate limiting of its own: a packet only gets here if the policy LOG fired, so these logs are already bounded by `LogActionRateLimit`. - **Log prefixes** come from the dedicated `LogConnectionTransitionsPrefix` param (default `calico-response`), used verbatim — the chain is static and shared by all policies, so `LogPrefix`-style `%`-specifiers cannot be resolved and are rendered literally (the API doc says so) — with the base truncated so suffix plus the `": "` appended by the Log action fit the backend limit (29 chars iptables, 126 nftables). - **RST visibility**: netfilter hooks run before the TCP stack sees a packet (forwarded traffic never touches the host TCP stack at all), and conntrack only *classifies* an RST (valid → ESTABLISHED reply; bad-seq → INVALID, handled by the existing INVALID rule), so a genuine refused-connection RST always reaches the check rule. - **Toggle caveat** (documented, accepted): disabling and re-enabling the feature with a different resulting bit layout can produce at most one spurious log per pre-existing connection; stale connmark bits are otherwise inert once the rules are gone. ### Review notes for this section - A PR that adds a dispatch rule per endpoint to a flat chain reintroduces the linear per-packet cost the tree structure exists to avoid. Keep new per-endpoint matching inside the tree. - A PR that claims a new mark bit must allocate it through `MarkBitsManager` against the configured range, and must not assume a specific bit is free. The Enterprise build allocates further bits from the same range, so a bit that looks free here may collide there — check downstream before adding one. ## IP sets [`felix/ipsets/`](../ipsets/) reconciles kernel IP sets against desired membership. Three behaviours look odd until you know the kernel constraints behind them: - **Temp-set-and-swap.** Atomically changing the capacity or metadata of an **in-use** IP set is only possible by building a temporary set and swapping it in. That's why updates that change set parameters go via a temp set. - **Deletions are deferred until after `*tables` updates.** You cannot delete an in-use IP set, so a set being removed must first have every referencing rule removed — which the `apply()` ordering (delete IP sets *after* `*tables`) guarantees. - **Deletions are rate-limited/queued** because each `ipset destroy` is surprisingly slow (~40ms) and serialised in the kernel: Felix caps deletions per iteration (`MaxIPSetDeletionsPerIteration = 1`, rescheduling with a ~100ms floor), so a big policy teardown of thousands of sets doesn't stall the whole dataplane on cleanup. - **Periodic resyncs are incremental**, for the same reason: each set is read back with its own `ipset list `, so re-listing every set per refresh is far too slow. A refresh lists only the *names*, repairs any set that vanished or appeared unexpectedly in the same apply, and queues the survivors' contents for re-checking a time-boxed batch per apply, paced by the same ≤100ms reschedule as deletions. The queue has two tiers: **must** (start-of-day and error-forced) is drained fully before Felix trusts the dataplane, **background** (periodic refresh) is spread over apply loops. On nodes with many sets it may never fully drain, so the sweep becomes a continuous rolling scan whose effective re-check period is one sweep — which can exceed `IpsetsRefreshInterval`. That is the intended trade-off, not a pacing bug. **The cross-layer invariant: never reference an IP set before it is programmed.** This is enforced jointly by the calc graph and the dataplane, and a change to *either* alone breaks it: - the **calc graph** always emits an IP set before any policy that references it (the `EventSequencer` flush order), and - the **`apply()` ordering** creates IP sets before `*tables` and deletes them after. ### Review notes for this section - A change to IP-set creation/deletion ordering, in *either* the calc graph flush order or `apply()`, must preserve the joint "reference only after programmed / delete only after dereferenced" invariant. Reason about both layers together. - A change that deletes IP sets eagerly (not deferred / not rate-limited) can hit in-use errors or stall the dataplane on a big teardown. - A change to the incremental resync must preserve three invariants: (a) the start-of-day full resync completes before the first writes; (b) a desired set found missing from the name listing is repaired in the same apply as the sweep that finds it; (c) a set whose write failed is re-checked at **must** priority before Felix writes to it again. ## Routing drivers [`felix/routetable/`](../routetable/), [`felix/routerule/`](../routerule/) and [`felix/vxlanfdb/`](../vxlanfdb/) are drivers in the sense above: managers (`vxlanManager`, `ipipManager`, `wireguardManager`, `noEncapManager`, …) compute desired routes/rules/FDB entries and hand them off; these drivers reconcile against netlink. They follow the same doctrine as the rest of the dataplane — start-of-day resync, minimal-disruption deltas, mark-and-sweep of orphans — but with the **weakest identification story** (see the [identification table](#how-each-subsystem-identifies-ours)): routes by a heuristic blend of owned-table / owned-proto / points-down-a-cali-veth, ip rules only by the tables they jump to. That makes ownership classification and resync correctness the delicate part of any change here. > The deep netlink-level design of route resync (grace periods for > CNI races, conntrack cleanup on IP moves, etc.) is large enough to > warrant its own future sub-design; this section covers only how > the route drivers fit the dataplane architecture. ### Review notes for this section - A change to route/rule ownership classification can make Felix either delete routes it shouldn't (too greedy) or leak routes a previous Felix left (too timid). Review against the restart scenario, not just steady state. ## Dual stack The manager/driver topology **mirrors the kernel's structure**. Because the Linux kernel runs IPv6 largely as a separate plane, Felix instantiates a **second copy** of the managers and drivers for IPv6, and the two are ships in the night — IPv4 and IPv6 updates can often run in parallel for that reason. The standard failure here is touching only the IPv4 instance. Legitimate asymmetries are rare on the `*tables` path. Two notes: - **BPF is the exception** to the two-separate-planes model: its entrypoint programs receive both families and fan out internally to v4/v6 sub-programs. - **nftables** could in principle have used a single shared v4/v6 table, but was kept as two to track the iptables structure for porting ease. ### Cross-cutting components that feed the per-family managers Not everything is duplicated per family. Some components are deliberately **single instances that feed the per-family managers**. The live migration monitor (`dataplane/linux/live_migration.go`) is the model: one FSM per workload, driven by GARP detection and timers that have nothing to do with IP family, whose state changes are then pushed into the endpoint managers — which own the per-family route programming. For those components the dual-stack trap is not "did you duplicate the code" but **"did you fan out to every family's manager"**. A singleton holding a *single* reference to its downstream manager serves IPv4 only and fails silently — the IPv6 manager never learns the state, so its routes keep their default behaviour. Prefer a list plus an explicit registration call (`registerListener`) over a single-valued field: a missing family then shows up as a missing call at the construction site in `int_dataplane.go`, alongside the `RegisterManager` call it belongs with. ### Review notes for this section - A dataplane change must be applied symmetrically to the IPv4 and IPv6 manager/driver instances unless there's a specific reason not to. "Works on v4, forgot v6" is a recurring bug. - A cross-cutting (non-per-family) component that pushes state into the per-family managers must be wired to **all** of them. Check the construction site in `int_dataplane.go`: every per-family manager the component needs should have a matching registration call. A single-valued reference field where a list belongs is the smell. - Dual-stack behaviour needs dual-stack tests. A per-family manager's own unit tests may already run for both families and still pass while the family is disconnected upstream; the registration itself and at least one FV path need explicit IPv6 coverage. ## Windows (contrast) Windows has its own dataplane (`dataplane/windows/`) and a full design is out of scope here, but it's worth recording what carries over and what doesn't, because it sharpens the Linux model: - **What's the same:** the overall architecture. Windows still defers programming until in-sync, still aims for minimal-disruption reconciliation, still resyncs at start of day. - **What differs:** the HCN/HNS dataplane has **no IP-set primitive**, so the Calico IP-set model doesn't fit. The Calico→kernel conversion that on Linux is split across managers and drivers is instead done by **calc-graph-like flattening functions inside the Windows dataplane**, converting policies + IP sets into fully-flattened HCN rules. It's isolated there deliberately. - **A sharp edge:** the HCN dataplane causes **traffic disruption on *any* rule change**, so the "don't flap already-correct resources on resync" rule (below) matters even more on Windows than on Linux. ## How to write a manager (and common failure modes) The recurring ways dataplane changes go wrong, and the recipe that avoids most of them. **Start simple: dirty-flag + start-of-day resync.** For a low-traffic manager, don't try to handle each individual update incrementally. Code the start-of-day resync first, driven by a single `dirty` bool: 1. Initialise `dirty = true` so the resync runs on the first in-sync `apply()`. 2. On any relevant `OnUpdate`, stash the data and set `dirty`. 3. In `CompleteDeferredWork`, if `dirty`, reconcile the whole thing and clear it. That's often all a low-traffic manager needs. Even when it's not the most efficient possible design, it's a **100%-correct** one — enough to write the `felix/fv` FV tests against and optimise later. **The hard part is a non-disruptive resync** — reconciling without flapping resources that are already correct. There's no shortcut: - Design in a simple **discriminator** for your resources up front (a `cali` prefix/comment where supported is ideal) so you can tell yours apart — this is the same requirement as [mark-and-sweep](#restart-resync-and-mark-and-sweep). - On resync: read back the kernel state; reconcile it against desired; classify into **already-correct / out-of-sync / to-delete**; **mark already-correct as done and don't touch it**; make minimal least-disruptive changes to out-of-sync resources (some disruption may be unavoidable — that's OK *as long as only out-of-sync objects are touched*); sweep the orphans. **The shortcut, where the kernel offers it: atomic full-state replacement.** If the kernel can atomically replace the whole resource set and GC orphans itself, you can skip read-back entirely: compute the full desired state and hand it over atomically. This is **often the API a driver presents to its manager** even when the kernel underneath doesn't work that way — the driver absorbs the read-back/reconcile/cleanup so the manager just declares desired state. (This is exactly why the manager/driver split pays off.) BPF map versioning (`felix/bpf/maps/maps.go`) shows the discipline that goes with this. **By default a map is rebuilt from desired state at upgrade time**, exactly like every other resync — no special handling. The copy/migrate path is reserved for maps whose contents are **sourced by the BPF programs themselves**, not by Felix — in practice the **conntrack map**. For those, `PinnedMap.EnsureExists`/`Upgrade` repins the live map aside to `_old`, builds the new-layout map in the normal pin path, copies entries across (`CopyDeltaFromOldMap`/`copyFromOldMap`), and drops `_old` — crash-safe because a restart mid-migration finds `_old` and rolls forward. The migration logic is fiddly; apply it only to maps that genuinely can't be rebuilt. ### The failure modes, distilled 1. **Delta-only, no from-scratch resync.** The biggest blind spot. A manager that only handles individual updates and never reconciles from scratch will leak or diverge across restart. Write the start-of-day resync first. 2. **Kernel work in `OnUpdate`.** Breaks batching and the safe-before-in-sync boundary. Defer to `CompleteDeferredWork`. 3. **Disruptive resync.** Flapping already-correct resources on every resync — connectivity glitches. Classify and touch only the out-of-sync. 4. **No restart-time identification.** Creating a resource type with no way for a fresh Felix to recognise it ⇒ orphans leak forever. 5. **Forgetting nftables, or BPF mode.** `*tables` rule changes that only land on one backend; or forgetting that BPF mode reuses some of this code (e.g. parts of `felix/rules/`). 6. **Touching only IPv4.** See [Dual stack](#dual-stack). ### Review notes for this section - A new manager without a from-scratch resync path (dirty-flag or atomic-replace) is incomplete, regardless of how well it handles deltas. Ask to see the resync. - A resync that rewrites or re-adds resources unconditionally (rather than diffing and touching only what's wrong) will glitch connectivity — push back. ## Siblings that move with this doc - [`calc-graph.md`](./calc-graph.md) — whenever the [dataplane API contract](#the-dataplane-api-calc-graph--dataplane-contract) changes.