Incident record
SEV2 · Load balancer, ingress, and service mesh
Illustrative example
Intermittent 502 errors on checkout after an ingress controller upgrade.
Detect
An error-rate alert fires on checkout. The load balancer, the ingress, and the pods all report healthy.
Mitigate
We roll the ingress controller back to the previous version as a temporary measure. Errors return to baseline.
Find the cause
Proxy logs show the new version keeps idle connections to services open longer than the mesh proxies do, so some requests reuse connections the mesh has already closed.
Escalate and fix
We report the changed default to the ingress project, align the idle timeouts in Git so each hop closes idle connections before the next one does, and reapply the upgrade through the change process.
Prevent recurrence
We add an alert on upstream connection resets and a timeout check to the upgrade runbook, then review the incident with your team.
Owner · Todea platform team, from alert to review
Your team · Kept informed at each step; approved the change window
Logged in the shared tracking system: one history for compliance, incident review, and planning.