JWT "jwt expired" the moment it was issued: clock skew between containers, and the claims that hide it

verbatim error{"error":"invalid_token","error_description":"jwt expired"} # issued at 10:00:00 UTC, validated at 10:00:01 UTC, exp = 10:00:30

Problem

A freshly minted token rejected by the very next service:

{"error":"invalid_token","error_description":"jwt expired"}
# issued at 10:00:00 UTC, validated at 10:00:01 UTC, exp = 10:00:30

The token was valid for 30 seconds (short-lived by design for a service-to-service call). The validator's clock was 35 seconds behind the issuer's. Every token born after the skew appeared was dead on arrival.

Root cause

exp and iat are absolute UTC timestamps, and "now" is whatever the validating machine thinks. Two time domains drifted:

1. Containers without an NTP-synced host or with a suspended host VM — laptops sleep, VMs pause, clocks fall behind and jump forward on resume. 2. The silent variant: nbf (not-before) set to "now" by the issuer; a validator whose clock is ahead rejects the token for a few seconds as "not yet valid" — same class, opposite direction.

The 30-second expiry amplified a 35-second skew into a 100% failure rate; with a 15-minute token, the same skew produced only mysterious first-request failures.

fix preview — first 3 of 4 lines (bash), truncated:

date -u +"%s"   # run on issuer and validator; the delta is your answer
# inside k8s:
kubectl exec -it issuer-pod -- date -u +"%s"
… 1 more line in the fix

🔒 the fix — including 2 code blocks — is members-only. $1/mo unlocks everything.