<!--
Copyright (c) 2026 Tigera, Inc. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0
-->

# IPAM CNI plugin

This sub-design covers the CNI plugin's use of IPAM: `cni-plugin/pkg/ipamplugin/` (the `calico-ipam` binary's implementation) and `cni-plugin/pkg/k8s/` (annotation handling). The
cross-component picture - data model, consumers, repo split - lives in the [index](./DESIGN.md). Handle-ID conventions are defined in
[`./ipam-core-library.md`](./ipam-core-library.md) and referenced here, not re-derived.

The CNI plugin ships two binaries: `calico` (main CNI) and `calico-ipam` (used when `conf.IPAM.Type == "calico-ipam"`). The dispatcher in
[`cni-plugin/cmd/calico/calico.go`](../../cni-plugin/cmd/calico/calico.go) routes by binary name. `calico` runs the `calico-ipam` code in its own process instead of exec'ing the `calico-ipam` binary.

## ADD flow

Invariants:

- **No partial state on failure.** Lock acquisition, the allocation call, and dual-stack family pairing each have a failure mode that must roll back, not leave half-applied state
  for the next CNI invocation to trip on. Under `calico-ipam`, addresses that a WorkloadEndpoint for another sandbox lists are the exception (KubeVirt pods with VM address
  persistence aside): a failed ADD leaves them on whichever of the pod's handles holds them, since the next ADD claims them back, and releases only the unlisted addresses on its
  own handle. A WorkloadEndpoint that names this same sandbox keeps nothing, so its addresses are released as that sandbox's DEL would.
- **Lock then clock.** The 90s allocation timeout starts *after* the host-wide IPAM lock is acquired (see [The IPAM lock](#the-ipam-lock)). Otherwise lock contention burns the
  budget before any allocation work runs.
- **IPAM dies with the plugin.** A runtime that gives up on a sandbox kills `calico`. A separate `calico-ipam` process would outlive it and could later move the pod's address
  onto the abandoned sandbox, whose cleanup DEL then frees it out from under the live one. Under a meta-plugin such as Multus the runtime kills the meta-plugin instead, and
  `calico` keeps running, so the next invariant has to hold as well.
- **A stale ADD never takes a newer sandbox's address.** An ADD refuses to move an address whose allocation was stamped after the ADD started, since only a sandbox kubelet
  created later could have claimed it. It fails with `ErrCodeSandboxSuperseded` instead of falling back to a fresh address, because networking the abandoned sandbox would
  replace the live one's host interface. Both times come from the node's wall clock.
- **Only `ErrCodeAddressNotOurs` earns a fresh address.** `calico-ipam` returns it when an address it is asked to move from the pod's old sandbox isn't the pod's to take (another
  pod or a non-pod handle owns it, it is in cooldown, or its pool or block no longer admits it), and when a requested `ipAddrs` or `IP=` address has a pool or block that doesn't
  admit it. The plugin then assigns a fresh address for each family the pod wants that no reused address covers, even one the old sandbox never had, and lists IPv4 first. Any
  other reuse error fails the ADD, as does this code for an `ipAddrs` pod.

`cmdAdd` in [`cni-plugin/pkg/ipamplugin/ipam_plugin.go`](../../cni-plugin/pkg/ipamplugin/ipam_plugin.go) is the entry point. The flow is straightforward client construction →
pool resolution → lock → `AutoAssign` → return. The design-relevant steps are:

- **KubeVirt detection.** A pod name prefixed `virt-launcher-` triggers a VMI fetch. When `IPAMConfig.KubeVirtVMAddressPersistence` is enabled, the handle ID switches from
  pod-scoped to VM-scoped (`vmipam.CreateVMHandleID`) and existing IPs on that handle are reused. This is what makes the IP follow the VM across pod recreations and live migration.
  The two handle paths must stay separate - folding them collapses VM persistence.
- **Pool resolution precedence.** Per-pod annotation > per-namespace annotation > conf default > all enabled pools. `utils.ResolvePools` accepts pool names or CIDRs. The precedence
  is user wire surface; reordering it is a deprecation cycle, not a refactor.
- **Lock-then-clock.** Acquire the host-wide IPAM lock before the 90s allocation timeout starts. See [The IPAM lock](#the-ipam-lock).
- **Boundary mask.** `AutoAssign` returns block-masked CIDRs (so callers can program per-block routes); the CNI plugin narrows to `/32` (`/128`) at its boundary. Don't push the
  narrowing back into the library.

Static / BYO IP and pool / floating-IP annotations are handled inside `cni-plugin/pkg/k8s/k8s.go` before this flow runs - see [Annotations](#annotations).

**Review notes**

- KubeVirt VM persistence relies on the handle ID being VM-scoped, not pod-scoped. Don't fold the two handle paths together.
- Pool resolution is hot path. `ResolvePools` was hand-optimized in https://github.com/projectcalico/calico/pull/9891; preserve the fast path.
- Don't log `stdinData` - it may contain `K8sAuthToken` / `K8sClientKey`. See `cni-plugin/pkg/k8s/k8s.go`.
- A conflict on a requested static IP is resolved by moving the address to this sandbox's handle under the pod's own identity, never by releasing it. Release-by-address cannot
  express "only if this pod owns it".
- The owner precondition comes from CNI_ARGS (kubelet). Anything a pod spec can set, including the addresses `getPodIPs` projects from the pod, is attacker-controlled.

## DEL flow

Invariants:

- **DEL is idempotent.** Any subset of block, handle, or allocation may already be gone. "Not found" is success.
- **Release by both handle forms.** DEL releases the sandbox's container-ID handle, then the workload-ID handle (`<namespace>.<pod>`). Dropping either leaves its IPs for the
  kube-controllers GC to reclaim instead of freeing them on DEL.
- **A stale sandbox's DEL keeps what the WEP lists.** When the WEP's container ID is empty or doesn't match the DEL's, it releases only the unlisted addresses on its own handle, because a later
  ADD may have moved the listed ones there, and never falls through to the normal release: `IPsByHandle` skips a block it can't read, so finding no listed address there proves
  nothing. A failed ADD's cleanup of its own handle follows the same rule. The workload-ID handle stays for the matching DEL to release. The matching DEL releases only its
  own handles, so a listed address left on a failed sandbox's handle waits for a later ADD to move it, or for the [kube-controllers GC](./ipam-gc.md) to free it once the pod
  is gone. KubeVirt pods with VM address persistence, and IPAM types other than `calico-ipam`, release as usual.
- **KubeVirt DEL is not "release immediately."** Clear owner attrs first; release only when the VM/VMI is gone and all attrs are empty.

`cmdDel` in the same file. The standard path releases the container-ID handle (`GetHandleID`), then the workload-ID handle (`<namespace>.<pod>`) unconditionally: that one
is not derived from the container ID, so it can hold an allocation at the same time as a container-ID handle.
`ErrorResourceDoesNotExist` at this stage is success, not failure - any subset of block / handle / allocation may already be gone.

KubeVirt with persistence is the exception: the IP must survive pod deletion so it follows the VM through live migration. The plugin fetches the VMI, checks `DeletionTimestamp`,
enumerates via `IPsByHandle`, and clears owner attrs through `SetOwnerAttributes` under preconditions. `ReleaseByHandle` runs only when the VM/VMI itself has `DeletionTimestamp`
*and* every IP's owner attrs are empty - that's the handshake that lets Felix race the active/alternate swap without losing the IP.

**Review notes**

- Always release by the container-ID handle and by the workload-ID handle. Dropping the workload-ID call leaks IPs across CRI container-ID changes and across upgrades
  from v2.x-era allocations.
- DEL must be idempotent. A retry that fails because the block is gone is a bug, not a feature.
- KubeVirt DEL is not "release immediately." The `SetOwnerAttributes` preconditions exist because Felix can race the active/alternate swap; if the precondition fails, the CNI
  plugin retries.
- Don't expand "not found" tolerance into "any error tolerated." It's specifically `ErrorResourceDoesNotExist` and the not-found-by-handle case.

## Annotations

User-facing wire surface - breaking parsing or semantics here is a user-visible incident. Handled in [`cni-plugin/pkg/k8s/k8s.go`](../../cni-plugin/pkg/k8s/k8s.go). Namespace
annotations are read as defaults; per-pod annotations override.

| Annotation | What it does |
|---|---|
| `cni.projectcalico.org/ipAddrs` | JSON array of IPs. Each IP allocated via `AssignIP` with `IntendedUse: Workload` - goes through IPAM, tracked normally, and the containing pool's `allowedUses` is enforced (an IP from a non-workload pool is rejected). An address the pod already holds under an earlier sandbox's handle is moved to the new handle rather than released. |
| `cni.projectcalico.org/ipAddrsNoIpam` | JSON array of IPs. **Bypasses IPAM entirely**, no allocation record. Requires `feature_control.ip_addrs_no_ipam=true`. |
| `cni.projectcalico.org/ipv4pools`, `ipv6pools` | Pool names or CIDRs scoping the allocation. Feeds `utils.ResolvePools` ahead of the conf default. |
| `cni.projectcalico.org/ipFamilies` | `["IPv4","IPv6"]` - controls which families to assign. |
| `cni.projectcalico.org/floatingIPs` | JSON array of external IPs to DNAT to the pod. Stored as `IPNAT{InternalIP, ExternalIP}` on the WEP; Felix consumes downstream. Requires `feature_control.floating_ips=true`. |

`ipAddrs` and `ipAddrsNoIpam` look similar; the code paths diverge. `ipAddrs` goes through `AssignIP`, so the IP appears in an `IPAMBlock` and the GC will see it. `ipAddrsNoIpam`
skips IPAM, the IP is whatever the user wrote, and nothing else in Calico knows the allocation exists - the user is responsible for not picking conflicts.

**Review notes**

- Annotation names and values are user wire surface. Don't rename, don't tighten parsing, don't reject inputs that previously worked. If the semantics need to change, that's a
  deprecation cycle, not a refactor.
- `ipAddrsNoIpam` and `floatingIPs` are feature-gated for a reason. Don't quietly drop the feature-gate check.
- Per-pod overrides per-namespace overrides conf default. Don't reorder the precedence.
- `floatingIPs` are consumed by Felix via the WEP `IPNAT` list - changes here that affect format need to land with the Felix side, not after.

## Platform differences

`ipam_plugin.go` is shared; the platform splits are file-level GOOS shims and a small set of args set differently on Windows.

- Lock file: Linux defaults to `/var/run/calico/ipam.lock` ([`ipam_lock_linux.go`](../../cni-plugin/internal/pkg/utils/ipam_lock_linux.go)); Windows to
  `c:\CalicoWindows\ipam.lock` ([`ipam_lock_windows.go`](../../cni-plugin/internal/pkg/utils/ipam_lock_windows.go)).
- Binary: Windows enters through `node/cmd/calico-ipam`.
- Windows forces `StrictAffinity=true` (HNS can't route remote affinity blocks). `WindowsUseSingleNetwork` collapses to `MaxBlocksPerHost = 1`. `HostReservedAttrIPv4s` reserves the
  first three and last IP of each block under the literal handle `windows-reserved-ipam-handle`. The reserved-handle name is parsed elsewhere - don't rename it.

## Dual-stack

If the request asks for both v4 and v6, `AutoAssign` returns one `IPAMAssignments` per family. The CNI plugin treats a half-success as a failure: if v4 came back populated and v6
came back empty (or vice versa), release the successful family immediately and surface a partial-fulfilment error to kubelet. Otherwise the pod silently runs single-stack, which is
worse than failing the ADD: the user asked for dual-stack and didn't get it.

**Review notes**

- Half-success must release the successful family. Leaving a single-family allocation in place when the request was dual-stack is an IP leak from kubelet's perspective - the pod
  won't come up, but the IP is still claimed.
- The `ipFamilies` annotation can narrow the request to one family. Don't surface "missing the other family" as an error when the user asked for only one.

## The IPAM lock

`AcquireIPAMLockBestEffort` in [`cni-plugin/internal/pkg/utils/ipam_lock.go`](../../cni-plugin/internal/pkg/utils/ipam_lock.go) takes a `flock` on the path from
`conf.IPAMLockFile`, falling back to the platform default. The lock is per-host and held for the duration of a `calico-ipam` ADD or release, including the keep-aware
release that the `calico` plugin runs in-process. It isn't reentrant, so nothing may hold it while calling into `calico-ipam`.

What it protects against: two `calico-ipam` invocations on the same node racing each other. CAS on `IPAMBlock` would technically serialize the persisted state, but pre-allocation
work (resolving pools, choosing a block, claiming a fresh affinity) is multi-step and the second invocation would otherwise waste a full retry budget bouncing off the first.
Holding the lock single-flights per-host allocation; the CAS protocol handles cross-host conflicts.

Lock-then-clock: the 90s allocation timeout starts **after** the lock is acquired, not before. Previously the timer started before the lock wait so a busy node could burn the
entire budget on lock contention and then fail the ADD with no allocation attempt. Fix landed in https://github.com/projectcalico/calico/pull/11824.

**Review notes**

- Lock first, then start the timeout. The clock starts after `AcquireIPAMLockBestEffort` returns, not before.
- The lock is best-effort: if acquisition fails (permissions, missing dir) the function logs and returns a no-op unlock so allocation proceeds. Don't promote this to a hard failure
  - the CAS layer is still correct, the lock is a contention optimization.
- The lock is host-local. It doesn't protect against another node racing for the same block; the BlockAffinity two-phase claim does that.
- Don't take the lock around the libcalico client construction or other slow setup. Hold it only across the allocation or release calls.

## Pod handle form

`PodHandleID` builds it and `ParsePodHandleID` parses it, both in `libcalico-go/lib/ipam`. `ParsePodHandleIDAnyNetwork` parses it without the network name, for the IPAM GC, and accepts only runtime-shaped (64 hex character) container IDs.

| Form | Written by | Accepted |
|---|---|---|
| `<network>.<container-id>` | `GetHandleID`, and `upgrade/migrate.go` for host-local migration | yes |
| `<namespace>.<pod>` | pre-v3 workload-ID handles | no, there is no container ID to hand over from |
| `<network>.vmi.<namespace>.<vm>` | KubeVirt VM-scoped handles | no, those persist across sandboxes by design |

A handle may carry a trailing carriage return, from the host-local migration file-format change in
[cni-plugin#821](https://github.com/projectcalico/cni-plugin/issues/821).

## Keep in sync with

- [`./ipam-core-library.md`](./ipam-core-library.md) - handle IDs are defined there; the CNI plugin is one consumer of that convention.
- [`./ipam-datastore.md`](./ipam-datastore.md) - block / handle state model that DEL idempotency relies on.
- [`../../cni-plugin/pkg/ipamplugin/`](../../cni-plugin/pkg/ipamplugin/) - a `DESIGN.md` stub in that directory will point back here.
