# IPAM core library This sub-design covers the core IPAM library in `libcalico-go/lib/ipam/`. It is the in-process API every IPAM caller goes through (CNI plugin, kube-controllers, `node`, `calicoctl`, operator). The cross-component picture - data model, consumers, repo split - lives in the [index](./DESIGN.md). ## Public API surface The library's contract is `Interface` in [`interface.go`](../../libcalico-go/lib/ipam/interface.go); that file is the source of truth for method signatures and is not restated here. A few methods carry design-relevant constraints worth calling out: - **`AutoAssign`** returns block-masked CIDRs, not `/32` (or `/128`). Callers narrow at the boundary. This is load-bearing for the CNI plugin's per-block route programming. - **`AssignIP`** enforces the target pool's `allowedUses` when `AssignIPArgs.IntendedUse` is non-empty: it fails if the pool containing the requested IP does not permit that use, using the same allowed-use check pool qualification applies for `AutoAssign`. Callers that leave `IntendedUse` empty are exempt (back-compat). This closes a gap where a specific-IP request (e.g. the CNI `ipAddrs` annotation) could draw from a pool not sanctioned for its use. - **`GetUtilization`** reports `Capacity`, `InUse`, `Reserved` and `Available` per pool and per block. `InUse` (allocated) and `Reserved` (covered by an `IPReservation`) overlap when an address was allocated before it was reserved, so `Capacity` is not their sum plus `Available`; `Available` counts addresses that are neither, and is the only one of the four that answers "how many can still be handed out". Consumers must read it rather than deriving it. Pool-level counts span the whole pool CIDR, including space no block covers yet - a reservation over unblocked space is still unassignable - so they are computed as a set operation (pool minus reservations minus blocks, via `go4.org/netipx`) rather than summed from the blocks. Reservations may overlap and nest arbitrarily, which is why a set is needed and not a sum over CIDRs. - **`NumReservedIPsInCIDR`** is the pool-level reserved count on its own, for callers that already hold the `IPReservation`s and would rather not pay for a list of every allocation block. kube-controllers uses it for `ipam_ippool_reserved` from syncer-fed reservations (see [ipam-gc](./ipam-gc.md#metrics)). It takes the resources, not CIDRs, so that a variant can take a second kind of reserving resource without reshaping its callers. - **`ReleaseIPs`** takes `ReleaseOptions` with a sequence number; every release path must plumb it through (see [CAS retry and sequence numbers](#cas-retry-and-sequence-numbers)). - **`SetOwnerAttributes`** is KubeVirt-only and swaps owner attributes under preconditions, without releasing and re-allocating. Felix's live-migration monitor is the only non-CNI caller. - **`MoveIPToHandle`** re-points an allocated address at a different handle in one block CAS, without releasing it, so the address is never unowned and never enters cooldown. `ExpectedOwner` is required and has to name a namespace and a pod, so only the address's current owner can move it; handing an address to a different workload is a release and a fresh assignment. - **`GetIPAMConfig` / `SetIPAMConfig`** read and write the v1 `IPAMConfig` / v3 `IPAMConfiguration` singleton. Field-level bounds are enforced by the CRD schema in k8s mode, but the cross-field rules live only in `SetIPAMConfig` - a direct CRD write can persist a config that violates them, which the library rejects on read (see [IPAMConfig](#ipamconfig)). **Review notes** - New CRDs / new API methods need operator RBAC; see https://github.com/tigera/operator/pull/4775 (May 2026). Easy to miss on backports. - Don't leak `crd.projectcalico.org/v1` types through new public APIs. The `lib/v3` -> `lib/internalapi` rename (https://github.com/projectcalico/calico/pull/11870) exists to keep that boundary clean. - `AutoAssign` returning block-masked CIDRs is load-bearing for the CNI plugin's routing. Don't quietly switch to `/32`. - Anything that makes an address unassignable has to be discounted by `GetUtilization` as well as by the allocation path, or the reporting surfaces over-count free addresses. The two must be fed from the same set of reserved CIDRs: allocation and the per-block counts share the `addrFilter`, and the pool-level counts use the same CIDRs as a set. - There is one implementation of the reserved-set arithmetic, in [`accounting/reserved.go`](../../libcalico-go/lib/ipam/accounting/reserved.go), beside the shared pool counts, block attribution and leak predicate. `GetUtilization` and `NumReservedIPsInCIDR` are both thin callers of it. Don't grow a second copy in a consumer - a reporting surface that disagrees with `calicoctl ipam show` is worse than no surface. Addresses that allocation skips for only some pools are counted through the tracker's reserved classes (`setExtraReserved`), so a block's counts follow its pool's class wherever it moves. The classes only count: the allocation path must skip those addresses itself, and `NumReservedIPsInCIDR`, which takes a CIDR rather than a pool, sees none of them. ## AutoAssign and host affinity Three invariants frame this section: - **Two-phase claim.** A new block goes `pending → confirmed`. Pending affinities are treated as absent for ownership and routing; only confirmed affinities participate. This is what makes the claim race resolvable. - **`MaxBlocksPerHost` caps claims, not allocations.** Once a node reaches the cap it can still fill blocks it already owns. The cap is the only gate on new-block claims. - **`StrictAffinity=true` is the only way to suppress non-affine fallback.** Anything that needs "borrow nothing" semantics must set it, not invent a parallel switch. `AutoAssign` is the hot path. Entry point is `autoAssign` in [`ipam.go`](../../libcalico-go/lib/ipam/ipam.go). The walk splits at three chokepoints: `prepareAffinityBlocksForHost` resolves the node, asks pool qualification which pools the request may use, and lists existing affinities; `findOrClaimBlock` walks affine blocks and lazily reconstructs an `IPAMBlock` if an affinity exists but the block doesn't (recovery for a node that crashed mid-claim); `findUsableBlock` claims a new block via the two-phase `pending → confirmed` protocol or, with `StrictAffinity=false`, falls back to `randomBlockGenerator` over non-affine blocks. A few non-obvious design points: - The random generator seeds from a hostname hash so retries hit the same blocks in the same order. Two nodes competing for the same pool deterministically diverge instead of thrashing. - An empty affine block becomes eligible for reclaim by another node after `EmptyBlockMinReclaimAge` (1 minute). The same value is what makes [empty-block release](./ipam-gc.md#empty-block-release) safe. - The block cap is `min(global MaxBlocksPerHost, request-level)`, defaulting to 20 if both are zero. Once a node reaches it, `allowNewClaim` is forced false; existing blocks still fill. - Pool qualification ([`pool_qualification.go`](../../libcalico-go/lib/ipam/pool_qualification.go)) is the one place that decides which pools a request may draw from, and which of the node's affine blocks IPAM may release: AutoAssign and EnsureBlock use its full rule set, AssignIP a narrower one. A new eligibility criterion goes there, not in a caller. Assignment and IP release (`ensureConsistentAffinity`) both ask it whether the node keeps an empty block. Pools a request names skip the selectors and `AssignmentMode`, nothing else. Code outside the package that must agree with that choice, such as reporting which nodes are at their block cap, asks `PoolClaim.MayUse` rather than re-deriving the rule. - `GetEnabledPools` skips a pool that lacks `Allocatable=True` when an allocatable or terminating pool covers its CIDR. The IP pool controller resolves overlap asynchronously, so without this a pool created over an active one is allocatable until the controller's first status write - unbounded while kube-controllers is down. A pool already marked `Allocatable=True` is taken at its word and never tested against the others. Administratively disabled pools never mask, including while they are terminating, which matches the controller's trie. **Review notes** - The `pending -> confirmed` two-phase claim is intentional; don't "optimize" it away. It's what makes claim races resolvable. See https://github.com/projectcalico/calico/pull/6003, https://github.com/projectcalico/calico/pull/1712. - Treat pending block affinities as if absent when deciding ownership or advertising routes. - Pre-existing blocks may lack `AffinityType`; default to `"host"` on read. https://github.com/projectcalico/calico/pull/11179 was a crash from this assumption. - Pool resolution is in the hot path. `ResolvePools` was hand-optimized in https://github.com/projectcalico/calico/pull/9891 - preserve the fast path. - Overlap is enforced at pool selection, not only by the IP pool controller. Don't reduce the gate back to a bare `Allocatable=False` check; that reopens the window a new overlapping pool is allocatable in. - `MaxBlocksPerHost` defaults are a recurring doc/code drift point (https://github.com/projectcalico/calico/issues/9462). If you change the default in code, update the docs in the same PR. ## CAS retry and sequence numbers Two invariants frame this section: - **Every IPAM write is CAS.** No write bypasses `backend.Client.Update`. The retry loop is bounded; on non-conflict errors it surfaces, not swallows. - **Per-ordinal sequence numbers detect ABA.** A release that doesn't match the stored sequence number is rejected. Every release path must plumb the sequence number through; dropping it reopens the pod-reuse race. All `IPAMBlock`, `BlockAffinity`, and `IPAMHandle` updates are CAS on resource version via `backend.Client.Update`. The library wraps every write in a bounded retry loop (`datastoreRetries = 100`, declared in [`ipam.go`](../../libcalico-go/lib/ipam/ipam.go)). On `ErrorResourceUpdateConflict` the loop re-fetches and retries; on any other error it exits immediately and surfaces to the caller. Each `IPAMBlock` carries a `SequenceNumber` that [`updateBlock`](../../libcalico-go/lib/ipam/ipam_block_reader_writer.go) increments on every write, and per-ordinal `SequenceNumberForAllocation[ord]` records the block's sequence number at allocation time. A `ReleaseOptions.SequenceNumber` from the caller is compared against the stored value; on mismatch the library returns `ErrorBadSequenceNumber` and the IP is **not** released. This prevents the GC from freeing an ordinal that was reallocated to a new pod between scan and release - the new pod's allocation bumps the block sequence number, so stale release options won't match. Block release is parallelised per block via a semaphore sized at `GOMAXPROCS`. **Review notes** - Every release path must plumb the sequence number through. A release path that drops it on the floor is a regression. See `ReleaseOptions.SequenceNumber`. - Don't mutate the in-memory `*model.AllocationBlock` (KVPair) before persisting. Update the value, call `updateBlock`, then update auxiliary state (handle, host info). https://github.com/projectcalico/calico/pull/12697 was exactly this bug: partial mutation persisted via a retry loop after the actual write was skipped. - The `Deallocated` / `Unallocated` queue cycling is load-bearing for IP-reuse delay. New code that punches the bitmap directly skipping the queue breaks rate-limited reuse (relevant to https://github.com/projectcalico/calico/issues/12638). - Older blocks may not have per-ordinal sequence numbers. New code must tolerate the absent case (default + heal-forward), not crash. ## IP release and cooldown Releasing an IP and freeing it for reuse are two distinct steps, separated by a configurable cooldown. This sits on top of the `Unallocated` FIFO queue: the queue cycles freed ordinals so the longest-idle IP is reused first, and the cooldown adds a wall-clock floor on how soon any released IP can come back. - **Release marks the IP, it does not free it.** `release` / `releaseByHandle` ([`ipam_block.go`](../../libcalico-go/lib/ipam/ipam_block.go)) clear the handle association and stamp the allocation's `ReleasedAt` with the current time. The ordinal stays in `Allocations` - the IP is no longer tied to a workload, but it is not yet available for reallocation. An IP in this state is "in cooldown". - **`garbageCollect` deallocates IPs whose cooldown has elapsed.** It moves an ordinal to `Unallocated`, clears its sequence number, and prunes the now-unreferenced attribute, but only once `ReleasedAt` is older than `IPCooldownSeconds`. With `IPCooldownSeconds=0` the IP is deallocated on the next GC pass. This is the only place ordinals move to `Unallocated`. - **GC runs implicitly on every read by the IPAM client.** `blockFromBackend` calls `garbageCollect` whenever a block is loaded. On write paths (`AutoAssign`, `release`, `releaseByHandle`, `SetOwnerAttributes`) the reclamation folds into the same CAS write, so a new allocation can reuse IPs that finished cooling down in one transaction. On read-only paths the GC'd view is computed and then discarded - the caller sees cooled-down IPs as not-yet-reusable, but nothing is persisted. - **Blocks with no write activity need a backstop.** Because read-only GC doesn't persist, a block that sees no further allocation or release never gets rewritten, so its cooled-down IPs would never be deallocated. The kube-controllers GC closes that gap; see [ipam-gc](./ipam-gc.md#cold-ip-garbage-collection). **Review notes** - Release and deallocation are distinct states. Code that counts "allocated" IPs has to decide whether cooldown counts as allocated, released, or its own state - don't silently fold it into one. `calicoctl ipam check` treats it as its own state. - A release call against an IP already in cooldown is not an error - it's already released. Don't return an error for the idempotent case. - The cooldown floor is on top of the `Unallocated` FIFO, not a replacement for it. Both exist; don't remove the queue cycling thinking the timestamp covers it. ## Handle IDs A handle ID is an opaque string from the library's point of view, but its **format is a convention every IPAM caller has to follow** because `calicoctl datastore migrate` parses the tunnel prefixes to rewrite them on node rename. Conventions in use: | Caller | Handle ID format | |---|---| | CNI workload (default) | `.` via `ipam.PodHandleID`. For the default network, `` is `k8s-pod-network`. | | CNI workload (KubeVirt persistent) | `.vmi..` via `vmipam.CreateVMHandleID`, so live-migrated VMs keep the same handle. Hashed if it would exceed 128 characters. | | IPIP tunnel | `ipip-tunnel-addr-` | | VXLAN tunnel | `vxlan-tunnel-addr-`; IPv6 variant is `vxlan-v6-tunnel-addr-` (note the `-v6-` infix, not a suffix) | | WireGuard tunnel | `wireguard-tunnel-addr-`; IPv6 variant is `wireguard-v6-tunnel-addr-` | | Windows-reserved | literal `windows-reserved-ipam-handle` | | LoadBalancer | `lb-`, where `` is the sha256 of `--` (lowercased), truncated to the DNS1123 limit. Built by `createHandle` in `kube-controllers/pkg/controllers/loadbalancer/loadbalancer_controller.go`. Not to be confused with the `virtual:load-balancer` affinity string. | The CNI plugin also keeps a separate **workload-ID** form (`.`) and releases by both on DEL so that allocations made before a CRI container-ID change can still be found - see [`./ipam-cni.md`](./ipam-cni.md). **Review notes** - The GC and `calicoctl ipam check` classify tunnel allocations by `AttributeType` and `Node` spec respectively, not by the handle prefix - so a handle-format change doesn't touch them, but it does need an `AttributeType` for the GC to recognize it. - A new tunnel handle prefix needs migration code in `calicoctl datastore migrate` (`calicoctl/calicoctl/commands/datastore/migrate/migrateipam.go`), which remaps tunnel-type handles on node rename. That parser's prefix list is v4-only today (`ipip-tunnel-addr-`, `vxlan-tunnel-addr-`, `wireguard-tunnel-addr-`), so it already skips the `*-v6-tunnel-addr-` handles - add v6 prefixes there if you touch it. - Skipping the workload-ID release on CNI DEL leaks IPs whose container-ID changed under CRI. Don't drop the second release call. - Every change to an allocation's `(handle, active owner, alternate owner)` tuple goes through `updateAllocationOwnership`, which re-points the ordinal at a new attribute entry. Editing an entry in place rewrites the ownership of every other address sharing it, since `findOrAddAttribute` de-duplicates entries by value. ## IPAMConfig `IPAMConfig` is a singleton CR. Defaults are applied on read by `GetIPAMConfig` when the CR is missing, so callers can rely on "there is always a config". Writes come from the operator reconciling user-facing config and from end users editing the CR directly (v1 `IPAMConfig` or v3 `IPAMConfiguration`). Field-level bounds are enforced by the CRD schema in k8s mode (kubebuilder markers on `IPAMConfigurationSpec`: `MaxBlocksPerHost` range, the `KubeVirtVMAddressPersistence` enum), so direct kubectl writes don't bypass those. But the cross-field rules below live only in `SetIPAMConfig`, not in the CRD schema - a client that writes the CR directly can persist a config that violates them, which the library then rejects on read: - `StrictAffinity=false` + `AutoAllocateBlocks=false` is rejected (would mean "never allocate anywhere", which is never what the user wants). - `MaxBlocksPerHost > 0` requires `StrictAffinity=true`. Fields: | Field | Effect | |---|---| | `StrictAffinity` | Disables non-affine fallback in `AutoAssign`. Windows forces true; see [ipam-cni](./ipam-cni.md#platform-differences). | | `MaxBlocksPerHost` | Per-host cap on the number of affine blocks. 0 means default (20). Once a host hits the cap, `allowNewClaim` is forced false; existing blocks still fill. | | `AutoAllocateBlocks` | When false, `AutoAssign` will never claim a new block - only allocate from blocks the host already owns. | | `KubeVirtVMAddressPersistence` | Default for whether KubeVirt VM addresses survive VM restart / migration. Auto-detection is on by default. | | `IPCooldownSeconds` | Minimum age of a released IP before it can be reused. Release stamps the IP's `ReleasedAt`; `garbageCollect` only deallocates it once this many seconds have passed. 0 deallocates on the next GC pass. Capped at 1200. See [IP release and cooldown](#ip-release-and-cooldown). | **Review notes** - The default-when-missing behavior is load-bearing. New required fields need a default plus heal-forward; don't add a field that crashes when absent. - `MaxBlocksPerHost > 0` only makes sense with `StrictAffinity=true`; the validator enforces this. If you relax the validator, you also need to define what "borrow blocks but cap our own" means - it currently isn't defined. - Field-level checks belong in kubebuilder markers (enforced by the CRD in k8s mode); the cross-field rules live only in `SetIPAMConfig`. A direct kubectl write skips the latter and persists config the library rejects on read, so a new cross-field rule must live in the library, not on a single caller. - A new field needs the v3 type, conversion, and operator reconciliation in addition to v1, or it isn't reachable through user-facing config. ## Error taxonomy Sentinel errors are defined in [`ipam_errors.go`](../../libcalico-go/lib/ipam/ipam_errors.go) and at the top of `ipam.go`. The design points worth carrying here are the error classes, not the names: - **Transient.** `ErrorResourceUpdateConflict`, `errBlockClaimConflict`, `errStaleAffinity`. The retry loop swallows these; callers never see them. Exiting the retry loop without consuming a transient is a bug. - **Validation.** `IPAMConfigConflictError`, `ErrStrictAffinity`, `ErrNoQualifiedPool`, `ErrBlockLimit`. Surface to callers; user-visible. New validation rules attach here. - **Stop-and-re-evaluate.** `ErrorBadSequenceNumber`, `errBlockNotEmpty`. The release path treats these as "skip this allocation and try again next sync", not as failure. A caller that catches and proceeds anyway defeats the sequence-number / empty-block protocols. Internal sentinels (`noFreeBlocksError`, `errBlockClaimConflict`, `errStaleAffinity`) are not part of the public API; translate to the exported `Err*` values before returning. **Review notes** - `mustBeEmpty=true` on `ReleaseBlockAffinity` is a hard precondition. The caller verifies emptiness; the GC's two-consecutive-empty-observations check is what gates this. - A new release code path that catches `ErrorBadSequenceNumber` and proceeds anyway defeats the protocol. Skip and re-evaluate. ## Keep in sync with - [`./ipam-datastore.md`](./ipam-datastore.md) - the CAS protocol and sequence-number scheme are defined together with the backend. - [`./ipam-cni.md`](./ipam-cni.md) - the CNI plugin duplicates the handle-ID convention; changes here that affect the format need to land there too. - [`../../libcalico-go/lib/ipam/`](../../libcalico-go/lib/ipam/) - a `DESIGN.md` stub in that directory points back here.