tailscale/cmd
Tom Proctor 95a135ead1
cmd/{containerboot,k8s-operator}: reissue auth keys for broken proxies (#16450)
Adds logic for containerboot to signal that it can't auth, so the
operator can reissue a new auth key. This only applies when running with
a config file and with a kube state store.

If the operator sees reissue_authkey in a state Secret, it will create a
new auth key iff the config has no auth key or its auth key matches the
value of reissue_authkey from the state Secret. This is to ensure we
don't reissue auth keys in a tight loop if the proxy is slow to start or
failing for some other reason. The reissue logic also uses a burstable
rate limiter to ensure there's no way a terminally misconfigured
or buggy operator can automatically generate new auth keys in a tight loop.

Additional implementation details (ChaosInTheCRD):

- Added `ipn.NotifyInitialHealthState` to ipn watcher, to ensure that
  `n.Health` is populated when notify's are returned.
- on auth failure, containerboot:
  - Disconnects from control server
  - Sets reissue_authkey marker in state Secret with the failing key
  - Polls config file for new auth key (10 minute timeout)
  - Restarts after receiving new key to apply it

- modified operator's reissue logic slightly:
  - Deletes old device from tailnet before creating new key
  - Rate limiting: 1 key per 30s with initial burst equal to replica count
  - In-flight tracking (authKeyReissuing map) prevents duplicate API calls
    across reconcile loops

Updates #14080

Change-Id: I6982f8e741932a6891f2f48a2936f7f6a455317f


(cherry picked from commit 969927c47c3d4de05e90f5b26a6d8d931c5ceed4)

Signed-off-by: Tom Proctor <tomhjp@users.noreply.github.com>
Co-authored-by: chaosinthecrd <tom@tmlabs.co.uk>
2026-03-11 10:25:57 +00:00
..
addlicense all: remove AUTHORS file and references to it 2026-01-23 15:49:45 -08:00
build-webclient all: remove AUTHORS file and references to it 2026-01-23 15:49:45 -08:00
checkmetrics all: remove AUTHORS file and references to it 2026-01-23 15:49:45 -08:00
cigocacher cmd/cigocacher: make --stats flag best-effort (#18761) 2026-02-19 16:06:12 +00:00
cloner types/ptr: deprecate ptr.To, use Go 1.26 new 2026-03-05 20:13:18 -08:00
connector-gen all: remove AUTHORS file and references to it 2026-01-23 15:49:45 -08:00
containerboot cmd/{containerboot,k8s-operator}: reissue auth keys for broken proxies (#16450) 2026-03-11 10:25:57 +00:00
derper all: use Go 1.26 things, run most gofix modernizers 2026-03-06 13:32:03 -08:00
derpprobe all: remove AUTHORS file and references to it 2026-01-23 15:49:45 -08:00
dist all: remove AUTHORS file and references to it 2026-01-23 15:49:45 -08:00
distsign all: remove AUTHORS file and references to it 2026-01-23 15:49:45 -08:00
featuretags all: remove AUTHORS file and references to it 2026-01-23 15:49:45 -08:00
get-authkey all: remove AUTHORS file and references to it 2026-01-23 15:49:45 -08:00
gitops-pusher cmd/gitops-pusher: fix precedence when id token env var is empty 2026-01-30 17:37:11 -07:00
hello all: remove AUTHORS file and references to it 2026-01-23 15:49:45 -08:00
jsonimports all: remove AUTHORS file and references to it 2026-01-23 15:49:45 -08:00
k8s-nameserver all: remove AUTHORS file and references to it 2026-01-23 15:49:45 -08:00
k8s-operator cmd/{containerboot,k8s-operator}: reissue auth keys for broken proxies (#16450) 2026-03-11 10:25:57 +00:00
k8s-proxy types/ptr: deprecate ptr.To, use Go 1.26 new 2026-03-05 20:13:18 -08:00
mkmanifest all: remove AUTHORS file and references to it 2026-01-23 15:49:45 -08:00
mkpkg all: use Go 1.26 things, run most gofix modernizers 2026-03-06 13:32:03 -08:00
mkversion all: remove AUTHORS file and references to it 2026-01-23 15:49:45 -08:00
nardump all: remove AUTHORS file and references to it 2026-01-23 15:49:45 -08:00
natc all: use Go 1.26 things, run most gofix modernizers 2026-03-06 13:32:03 -08:00
netlogfmt all: remove AUTHORS file and references to it 2026-01-23 15:49:45 -08:00
nginx-auth all: remove AUTHORS file and references to it 2026-01-23 15:49:45 -08:00
omitsize all: remove AUTHORS file and references to it 2026-01-23 15:49:45 -08:00
pgproxy all: remove AUTHORS file and references to it 2026-01-23 15:49:45 -08:00
printdep cmd/printdep: add --next flag to use rc Go build hash instead 2026-01-27 14:49:56 -08:00
proxy-test-server all: remove AUTHORS file and references to it 2026-01-23 15:49:45 -08:00
proxy-to-grafana all: remove AUTHORS file and references to it 2026-01-23 15:49:45 -08:00
sniproxy all: use Go 1.26 things, run most gofix modernizers 2026-03-06 13:32:03 -08:00
speedtest all: use Go 1.26 things, run most gofix modernizers 2026-03-06 13:32:03 -08:00
ssh-auth-none-demo all: remove AUTHORS file and references to it 2026-01-23 15:49:45 -08:00
stunc all: remove AUTHORS file and references to it 2026-01-23 15:49:45 -08:00
stund types/ptr: deprecate ptr.To, use Go 1.26 new 2026-03-05 20:13:18 -08:00
stunstamp all: use Go 1.26 things, run most gofix modernizers 2026-03-06 13:32:03 -08:00
sync-containers all: remove AUTHORS file and references to it 2026-01-23 15:49:45 -08:00
systray all: remove AUTHORS file and references to it 2026-01-23 15:49:45 -08:00
tailscale all: use Go 1.26 things, run most gofix modernizers 2026-03-06 13:32:03 -08:00
tailscaled ipn/ipnlocal, feature/ssh: move SSH code out of LocalBackend to feature 2026-03-10 17:27:17 -07:00
testcontrol all: remove AUTHORS file and references to it 2026-01-23 15:49:45 -08:00
testwrapper cmd/testwrapper: make test tolerant of a GOEXPERIMENT being set 2026-03-06 14:05:35 -08:00
tl-longchain all: remove AUTHORS file and references to it 2026-01-23 15:49:45 -08:00
tsconnect go.toolchain.branch: switch to Go 1.26 2026-03-04 21:57:05 -08:00
tsidp ipn/ipnlocal, feature/ssh: move SSH code out of LocalBackend to feature 2026-03-10 17:27:17 -07:00
tsshd all: remove AUTHORS file and references to it 2026-01-23 15:49:45 -08:00
tta all: use Go 1.26 things, run most gofix modernizers 2026-03-06 13:32:03 -08:00
vet all: remove AUTHORS file and references to it 2026-01-23 15:49:45 -08:00
viewer all: use Go 1.26 things, run most gofix modernizers 2026-03-06 13:32:03 -08:00
vnet all: remove AUTHORS file and references to it 2026-01-23 15:49:45 -08:00
xdpderper all: remove AUTHORS file and references to it 2026-01-23 15:49:45 -08:00