Skip to content

Platform engineering

The eMudhra CA Compromise: Why Patching Kubernetes Nodes Isn't Enough

How the eMudhra compromise affects Kubernetes nodes, container images, application runtimes, and CA bundles, and where updates are needed across each layer.

Todea Engineering

Cloud Native Practice

23 min read
#kubernetes#security#pki#tls#supply-chain#platform-engineering
The eMudhra CA Compromise: Why Patching Kubernetes Nodes Isn't Enough

In October 2026, a researcher discovered that eMudhra, a certificate authority, had generated eight certificate-signing private keys that were single-byte numbers (4, 56, 66, 115, 155, 156, 230, and 237) instead of random 384-bit values, so anyone could recover them from the public CA certificates in under a second. Anyone who recovered an affected TLS signing key could issue certificates for domains they did not control. An attacker who also intercepted or redirected traffic could then impersonate legitimate services to clients that still trusted the compromised issuers. Your own certificates do not have to come from eMudhra for your connections to be at risk. If a client accepts the forged certificate, the attacker can receive the credentials and request data sent over that connection and return altered responses.

Major browsers received updates to block the affected certificates, but those protections do not automatically extend to Kubernetes. Nodes, container images, application runtimes, and cluster configuration can each supply trusted certificates or apply their own verification rules. Remediation depends on which ones a running component actually uses.

Where Kubernetes keeps its trust

A Kubernetes cluster does not use a single shared trust store. A container runtime normally relies on certificates from the node, applications may include their own bundles inside images, and language runtimes can provide another set. Additional CA bundles can also be stored in Kubernetes configuration and mounted into workloads.

These copies are updated separately. A running process may also retain certificates loaded before an update, so correcting a file does not necessarily correct the application's trust decisions. The following sections explain what to update at each layer and when a restart or reload is required.

The exposure concerns connections that trust the affected public issuers. A connection restricted to a private cluster or mesh CA is not automatically exposed by this incident.

The diagram compares a typical image pull with an application connection. The application may use one or more of the sources shown, depending on its TLS configuration.

Node and workload trust follow separate update paths. Each running client must load any changed certificates.

1. The node: container runtime and control plane

When Kubernetes downloads an image, the kubelet asks the container runtime, such as containerd or CRI-O, to retrieve it. The runtime connects to the registry and verifies its HTTPS certificate using the node's trusted CAs. A registry-specific CA configured for containerd or CRI-O is added to that pool rather than replacing it, so the node bundle still applies to those registries.

If that trust configuration still accepts the compromised issuers, an attacker on the network path could impersonate the registry. This happens before the workload starts, so updating certificates inside the workload's image does not protect the download.

Install the distribution's CA update or replace the node with a fixed node image. Then restart affected processes that cached the old certificates. This includes the container runtime and any relevant node or control-plane components. A newly started diagnostic tool can use the updated bundle while a long-running service still uses the old one.

The control plane also needs attention. In kubeadm clusters, the API server and controller manager mount certificate directories from the node. These stores can be used for external HTTPS connections. For example, the API server uses system roots for an admission webhook when its configuration omits an explicit caBundle; a supplied bundle must be reviewed separately.

Use the supported static-Pod restart procedure for kubeadm control-plane components. Deleting their mirror Pod objects through the API does not restart the underlying containers. Roll through a replicated control plane one node at a time and verify readiness before continuing.

Also deploy applicable Kubernetes and runtime releases containing the certificate-verification fix. For managed Kubernetes, follow the provider's control-plane and node-image rollout guidance. Provider updates do not automatically replace your application images or custom CA bundles.

2. The image: the container's own trust store

Updating a node's CA bundle does not update the certificates inside its container images. Many images include their own /etc/ssl/certs, containing certificates installed when the image was built.

The disclosure site provides test endpoints that present certificates from affected issuers. The following command checks the emSign endpoint using a particular curl image:

kubectl run curl --image=curlimages/curl:8.22.0 --restart=Never --rm -i -- \
  -sS -o /dev/null -w '%{http_code}\n' https://dv-g3b-emudhra.we-have-a-problem.com/

The comparison below uses two images on the same node with an updated CA package:

curlimages/curl:8.22.020260611-r0Accepted: HTTP 200
curlimages/curl:8.23.020261020-r0Rejected: curl: (60)

The older image still accepts the affected certificate because its own bundle predates the fix. For images you maintain, update the CA package, rebuild, and redeploy. For vendor images, use a release containing the correction and confirm the running Pods have adopted it.

This checks curl's trust configuration. An application in the same container may use a different bundle or runtime, and rejecting this endpoint does not prove that every alternative certificate chain is blocked.

3. The runtime: the application's certificate checks

Applications do not always use the image's /etc/ssl/certs. Some runtimes include their own trusted certificates. Others also contain code that rejects the compromised signing keys, even if the trust store still contains the affected roots.

Go is particularly relevant to Kubernetes because its control-plane components and many controllers are written in Go. Go 1.27.3 and 1.26.10 added a blocklist to crypto/x509 that rejects certificate chains containing the compromised keys.

A test client built with three Go versions illustrates the difference. With the same outdated CA bundle, the results were:

go1.26.9AcceptedAccepted
go1.26.10Rejected: x509: no valid chains builtAccepted
go1.27.3Rejected: x509: no valid chains builtAccepted

The updated clients reject the affected certificate while still accepting the control connection. This protection is compiled into the application: deploy a release built with a patched toolchain, or rebuild and redeploy it yourself. A version declared in go.mod does not establish which toolchain built the deployed binary. Check the binary's build metadata as well as its behavior.

Java, Node.js, Python, and Rust applications also need runtime-specific checks. Review the runtime's security guidance and any separately bundled CA packages, such as Python's certifi. Updating the operating system's certificates does not necessarily update those copies.

4. The cluster configuration: CA bundles in Kubernetes objects

Kubernetes objects can hold CA bundles that workloads and controllers use to verify connections. Updating a node or container image does not replace these stored copies.

Review the places where your applications receive trusted certificates:

  • ConfigMaps and Secrets, including Argo CD's argocd-tls-certs-cm, Secrets containing ca.crt, and bundles referenced by Istio or Kasten.
  • Kubernetes API configuration, including CA bundles in admission webhooks, APIServices, and CRD conversion webhooks.
  • Application configuration, including CA settings in custom resources, Helm values, and bundles distributed by trust-manager.

Use the SHA-256 certificate fingerprints in the disclosure site's affected root list to identify those roots. Review explicitly trusted affected intermediates too. Do not remove unrelated private cluster or mesh trust anchors merely because they are stored in the same type of Kubernetes object.

If you use trust-manager, check its default CA package and any repository or tag overrides. Charts through v0.25.0 install an affected package by default; v0.26.0 installs a corrected package. Updating the default package does not remove affected certificates supplied through additional Bundle sources.

Available fixes and upgrade guidance

The tables below list the required action for Kubernetes components, managed clusters, and cloud-native tools as of the November 9, 2026 snapshot. Consult the linked project guidance for newer versions before planning a rollout. Custom bundles, helper programs, and other image variants still require separate checks.

Kubernetes and container runtimes

KubernetesUse v1.37.2, v1.36.6, v1.35.10, or the historical v1.34.13 fix. Prefer a supported minor release. Update node trust stores separately; the binary update does not remove certificates from the images.
containerdInstall your distribution's CA update and restart containerd. Where available, use a vendor package rebuilt with fixed Go, such as Docker's containerd.io 2.4.2-3. Check the upstream release notes before selecting a newer version.
CRI-OUpdate the node's CA package and restart CRI-O. Use a fixed vendor build or the next patched release for your supported branch. OpenShift users should follow Red Hat's rollout guidance.
etcdUse v3.7.3, v3.6.16, or v3.5.35 on the appropriate branch. Review custom CA configuration and any additional programs using the image's CA bundle.
k3dSelect a fixed k3s image explicitly, such as rancher/k3s:v1.37.2-k3s1, or the matching patched release for your minor version. This updates the nodes; it does not update the k3d CLI's own TLS checks.
k3s, RKE2Use k3s v1.37.2+k3s1, v1.36.6+k3s1, or v1.35.10+k3s1, or the matching RKE2 +rke2r1 release. Historical 1.34 fixes also exist; move unsupported branches to a maintained version.

Managed Kubernetes

A provider update being available does not establish that it has reached your cluster. Check the control plane, node image, and rollout status, then address customer-managed workloads separately.

Amazon EKSRoll managed, self-managed, and Karpenter nodes to a patched AMI. AWS identifies AL2023 v20261023 as the trust-store fix; the snapshot recommends v20261031 for the additional containerd rebuild. Use Bottlerocket v1.68.0 or later and verify Auto Mode replacements completed.
Google GKEUse the exact control-plane and node versions in Google's security bulletin. Ubuntu fixes for 1.33, 1.36, and 1.37 were still pending in the recorded snapshot. Follow the provider's supported manual upgrade path and rebuild application images separately.
Azure AKSRoll Linux node pools to image 202610.23.0 or a later patched image using the node-image upgrade process. Verify Windows nodes received the updated certificate trust lists, particularly where automatic updates are restricted.
Red Hat OpenShiftSelect the fixed z-stream for your minor version and available update channel. Include the rebuilt CRI-O where applicable, including 4.21.41 on 4.21. Confirm the machine-config rollout completes; do not infer 5.x coverage from a 4.x advisory.
IBM CloudApply the November 2 worker fix packs, which include RHCOS coverage. Earlier October 27 packs cover Ubuntu and RHEL 9 workers. Confirm every worker has been updated or replaced.
NebiusUpgrade node groups to infrastructure version 78 or a later patched version, then verify the replacement nodes. Application images and mounted CA bundles still need separate updates.
Oracle OKEConfirm the node's package matches the relevant Oracle Linux advisory, including ELSA-2026-88550 or ELSA-2026-88459 where applicable. Do not treat an image's November build date alone as proof of remediation.
Alibaba ACKConfirm the corrected ca-certificates package for the node OS and use ACK's supported update or node-replacement procedure. Restart the runtime after an in-place update. Check provider guidance before treating a new image as fixed.
DigitalOcean, Akamai, OVHcloud, Scaleway, Civo, TencentAsk the provider which node images and managed components contain the fix. Verify the running node package and runtime, then use the supported rollout procedure. Lack of a public advisory does not establish that a service is vulnerable or fixed.
Naver Cloud, NHN Cloud, KT Cloud, Samsung SDSConfirm the fixed node image or CA package with the provider and check the deployed versions. Use the provider's supported rollout process, including restarts where required.

Service mesh, gateways, and networking

IstioUse 1.31.4 or 1.30.8 from the snapshot, or a later supported patch. The fixes began in 1.31.3 and 1.30.7. Check custom egress CA bundles separately; 1.29.10 does not add the Go blocklist.
LinkerdUse edge-26.11.1 or a later patched edge release and update the CLI. Private mesh mTLS is not exposed through public roots when its trust anchors contain only your private CA. Review public-CA use and custom bundles separately.
Buoyant Enterprise for Linkerd (BEL)Upgrade to the fixed release for your supported BEL branch and follow Buoyant's rollout guidance. Review custom bundles separately. A proxy dependency finding alone does not establish exposure of mesh mTLS that trusts only private mesh roots.
EnvoyUse v1.40.2, v1.39.6, v1.38.9, or v1.37.11 on the appropriate branch. Update any bundle supplied through trusted_ca; an image update does not replace that custom bundle.
Envoy GatewayUse v1.10.0 after reviewing upgrade changes. On v1.9.3 or v1.8.6, configure the corresponding fixed Envoy image, such as distroless-v1.39.6 or distroless-v1.38.9, and roll the proxies.
CiliumUse 1.20.4, 1.19.10, or 1.18.16 on your supported branch. Keep agent, operator, and cilium-envoy images aligned with the chart, including any tag or digest overrides.
ConsulUpgrade Consul, consul-k8s, and consul-dataplane to the listed 2.1.0 releases. For terminating gateways, configure CAFile with a corrected bundle and verify the gateway loads it.
KumaUse 3.0.1, 2.14.7, 2.13.12, or 2.11.20 on the applicable branch. If a custom Corefile uses DNS-over-TLS forwarding, review the CoreDNS helper's TLS verification and trust store separately.
TraefikUse v3.7.17 or v2.11.61 on the applicable branch; v3.7.16 also contains the fix. Replace older images and review any custom backend CA configuration.
NGINX Gateway FabricUpgrade to 2.8.1 and confirm both the controller and NGINX data-plane Pods run the updated images. Check any custom CA bundles used for upstream TLS.
ingress-nginx (retired)Plan migration to a maintained controller. Review upstream TLS verification and proxy-ssl-secret CA bundles while migrating. Disabled upstream certificate verification is not a mitigation for this incident.
Kong Ingress ControllerUse KIC 3.5.14 or 3.4.21 on the applicable branch. Update Kong Gateway separately and review custom CA bundles; the controller's fix does not cover the gateway's outbound connections.
Kong Gateway EnterpriseUse Kong's supported configuration to load a corrected outbound CA bundle, including lua_ssl_trusted_certificate where applicable. Verify all relevant TLS paths and adopt a fixed vendor image when available.
ContourUse Contour v1.33.8 with a fixed Envoy image such as distroless-v1.38.9. Check supplied CA Secrets and the Envoy trust configuration separately; absence of a controller bundle does not establish gateway coverage.
Emissary-ingressUse a corrected ca_secret for applicable outbound TLS connections. Verify whether a stable fixed release is available before upgrading. Do not assume 4.2.0-rc.0 also corrected the image's CA bundle.
Calico v3.32 (tigera-operator v1.42)Upgrade tigera-operator to v1.42.7 to deploy the fixed components, or apply the v3.32.3 manifests for a manifest-based installation. Confirm all Calico images rolled out.
Calico v3.33 (tigera-operator v1.44)Track the next patched 3.33 release and update effective CA stores in the meantime using a supported method. For new deployments, the snapshot lists 3.32.3 as a fixed option; do not downgrade an existing cluster without a supported plan.
CoreDNS, external-dnsUse CoreDNS 1.15.0 and external-dns v0.24.0, subject to cluster compatibility. Check the deployed DNS image rather than assuming a Kubernetes upgrade changed it. Review custom TLS-forwarding and provider trust settings.

Certificates, identity, and policy

cert-managerUse v1.21.3 or v1.20.5 on the applicable supported branch. Move older unsupported branches to a maintained release and check CA bundles configured for external issuers separately.
trust-managerUse chart v0.26.0, or configure the matching fixed Trixie or Bookworm package from the advisory. Check both repository and tag overrides. Verify Bundle targets update, then restart consumers that cannot reload certificates.
SPIREUse v1.16.0 after reviewing its breaking changes, or the fixed Go backports v1.15.5 and v1.14.9. The backport images retain an affected root. Review https_web federation and custom trust bundles separately.
DexUse v2.46.1. If startup templates fetch remote HTTPS data, check the bundled gomplate helper and the trust store it uses. Keep a corrected CA bundle available to every helper involved.
KeycloakUse 26.8.2, or verify the digest of the corrected 26.8.1-2 respin. Rebuild custom images with an updated JDK and OS trust store, and review additional truststores you supply.
Vault Secrets OperatorUse VSO 1.6.2 and confirm the controller rollout. Prefer the corrected default image, or address the remaining root in 1.6.2-ubi. Review CA bundles configured for connections to Vault.
External Secrets OperatorUpgrade the controller and chart to 2.14.0, then verify rollout completion. Check CA bundles configured for external secret providers independently of the controller image.
KyvernoUse a corrected trust store for existing installations and restart Pods after changes. The global.caCertificates.data mount replaces one bundle file; verify that no additional certificate directory restores the affected roots. Prefer a fixed stable release when available.
OPAUse v1.22.2 or v1.21.4 on the applicable branch. Update default or static images as deployed, and review any custom trust configuration for outbound connections.
GatekeeperCorrect every CA file or directory the process loads, then restart the Pods. Replacing one bundle file may be insufficient. Adopt a supported release built with a fixed Go toolchain when available.
cosignUse the v3.1.4 release binary or a later verified build for image-signing workflows. If using the container image, preserve its corrected bundle and review certificate-file or directory overrides. Verify the artifact actually used by the pipeline.
policy-controllerProvide a corrected effective trust store and restart the webhook Pods. Check all loaded CA files and directories, then adopt a fixed release when available. A merged Go update does not change an existing deployment.
FalcoUse Falco 0.45.2 with chart 9.2.2. Check pinned helper images too, including falcoctl 0.14.3 and driver-loader 0.45.2, and confirm the updated components are running.
oauth2-proxyUse v7.15.6. Prefer the corrected default image, or address the Alpine variant's remaining affected root. Check custom provider and upstream CA bundles separately.

Delivery and supply chain

Argo CD (with Argo Workflows and Argo Rollouts)Use Argo CD 3.6.1, 3.5.7, or 3.4.13; Rollouts 1.10.2 or 1.9.3; and Workflows 4.1.6 or 4.0.15. Keep the corrected bundle available to helper tools and review argocd-tls-certs-cm and other overrides.
HelmUpdate the trust store on every workstation or runner executing Helm. Check subsequent release notes and verify the downloaded binary's build toolchain before treating it as fixed.
FluxUse Flux 2.10.0, or the exact suffixed controller rebuild tags documented for 2.9 and 2.8. Plain older tags were not necessarily replaced. Review the 2.10 RBAC and notification API changes before upgrading.
Tekton PipelineUse v1.18.0 or the applicable maintained backport v1.15.5, v1.12.3, or v1.9.9. For Git resolvers using the Git CLI, correct the CA store that Git actually loads and restart the resolver Pods.
HarborUse v2.16.0 after following the supported upgrade path and reviewing the Redis-to-Valkey change. The listed v2.14.6 backport has the Go fix but retains affected roots in its image bundle.
Actions Runner ControllerUpgrade the controller and applicable scale-set charts to 0.15.1. Update the runner images used by the scale sets and any job containers separately; the controller update does not replace their runtimes or CA bundles.

Container builds, registries, and scanning

BuildKitRoll standalone or in-cluster builders to moby/buildkit:v0.34.2 or a later patched release. Update buildx-managed builders through the appropriate driver workflow. Changing the local Docker CLI does not update an existing remote builder.
Trivy, GrypeUse Trivy 0.76.1 and Grype 0.121.1, or later verified patches. Update scanner images and version pins in image-build and admission workflows.
ORASUpdate the trust store wherever ORAS runs, or rebuild v1.3.6 with a patched supported Go toolchain. Verify the resulting binary. Restricting destination registries alone does not prevent interception of an allowed connection.
crane (go-containerregistry)Rebuild v0.22.1 with a patched supported Go toolchain, or use a later verified release. Correct the runtime CA store too, then verify both the downloaded binary and any pinned container image.

Observability, data, and platform

Prometheus, Alertmanager, blackbox_exporterUse Prometheus 3.16.0, 3.15.1, or 3.13.5; Alertmanager 0.34.2; and blackbox_exporter 0.29.1 as applicable. Review Alertmanager's CORS change and any custom TLS settings before rollout.
Prometheus node_exporterReview any configured TLS client authentication and keep its CA bundle limited to the intended clients. Correct applicable trust stores and adopt a verified fixed build when available. An image finding alone does not identify a reachable TLS path.
Grafana 13.1, 13.0 and 12.4Use 13.1.9, 13.0.12, or 12.4.14 on the appropriate branch. Check separately installed plugins and custom CA bundles rather than assuming the server update covers them.
Grafana 13.3 and 13.2Use 13.3.0 or 13.2.5 as applicable and preserve the corrected bundle. Check plugin binaries and their CA overrides separately, then adopt plugin updates as they become available.
Loki, Tempo, Mimir, ThanosUse Loki 3.7.9 or 3.6.19; Tempo 3.1.1, 3.0.5, or 2.10.10; Mimir 3.2.3 or 3.1.8; and Thanos v0.42.5, or later verified patches on those branches. Review custom TLS stores.
OpenTelemetry CollectorUse 0.163.2 or a later patched core, contrib, or Kubernetes distribution. Update explicit image pins and confirm the running Collector Pods use the new image.
OpenTelemetry OperatorUse operator 0.162.0 and explicitly select Collector 0.163.2 or later through spec.image or the operator's Collector-image setting. Check injected auto-instrumentation images separately, including the documented Python update.
JaegerUse v2.23.0 or a later verified release and review custom TLS bundles. Verify the deployed image rather than assuming a floating tag has refreshed an existing Pod.
Fluentd, Fluent Bit, VectorUse Fluentd's v1.19.4-1.2 image or a later corrected build, Fluent Bit 5.1.5, and Vector 0.61.0 as applicable. Replace stale image tags and check mounted CA bundles used by outputs.
VeleroPrefer a fixed stable release when available and check plugins separately. For an interim trust-store update, account for all CA files and directories loaded by Velero and node-agent, then restart them. Do not replace only one file without verifying the result.
LonghornOn 1.12, use v1.12.2. On 1.13, follow the next patched release and applicable interim guidance. Do not move from 1.12.2 to unpatched 1.13.0 merely because its version number is higher.
Veeam KastenUse 9.0.9 or later and confirm every Kasten component was replaced. Review the ConfigMap referenced by cacertconfigmap.name. Follow Veeam's upgrade procedure rather than manually modifying trust stores inside its containers.
Crossplane (core)Use core 2.4.3, 2.3.7, 2.2.7, or 1.20.14 as applicable. Check each installed provider and function image independently, including its configured CA bundles.
Cluster APIUse Cluster API 1.14.4 or 1.13.8 and the corresponding clusterctl version. Check CAPA, CAPZ, and other installed infrastructure providers against their own releases and trust configuration.
Karpenter (AWS)Use 1.15.1 or 1.14.3 as appropriate. The historical 1.9.3 build has the Go fix but retains an affected root. Controller upgrades do not replace existing worker AMIs; plan that rollout separately.
KEDAUse a corrected effective trust store or a verified rebuild while awaiting a fixed release. Check scaler HTTPS connections and custom bundles. Network egress restrictions can reduce exposure but do not repair certificate verification.
Dapr, NATSUse Dapr 1.19.1, 1.18.10, 1.17.19, or 1.16.21, and nats-server 2.15.2 or 2.14.9 as applicable. Inspect supplied TLS CA files and additional components separately.
Kubeflow PipelinesUse 2.18.0 or 2.17.3 as applicable. Rebuild pipeline-step images with updated OS certificates, Python dependencies, certifi, and pip where used. Verify their effective trust configuration independently of the backend.
KubeRayUse KubeRay 1.7.2 or a later verified release and review the operator trust configuration. Update Ray head and worker images separately; the operator upgrade does not update their Python runtimes or CA bundles.
KServeUse verified controller rebuilds with a patched Go toolchain and update model-server runtimes and CA bundles. Check every serving image. Restricting egress alone does not stop interception of allowed HTTPS connections.
Kubernetes add-ons: metrics-server, VPA, cluster-autoscalerUse metrics-server v0.9.1, VPA 1.8.1 or 1.7.3, and the autoscaler release matching your Kubernetes minor: 1.37.0, 1.36.2, 1.35.3, or 1.34.6. Follow provider guidance for managed add-ons.
Kubernetes add-ons: kube-state-metricsAdopt a patched release when available and verify the binary and image bundle. Its usual API-server connection trusts the private cluster CA, so assess actual trust configuration before treating a scanner finding as an exploitable path.

Conclusion

The eMudhra compromise shows why a Kubernetes security update can remain incomplete after the nodes have been patched. Containers can retain older CA bundles, applications can use separate runtime stores, and cluster configuration can supply certificates that survive both updates. Running processes may also need a restart before any correction takes effect.

For each affected client, identify how it verifies certificates, apply the relevant update, and confirm that it rejects the compromised chains while legitimate connections still work. The outcome to verify is the behavior of the running platform, including the images, configuration, and helper programs it actually uses.

Frequently asked

Five questions we keep getting about the eMudhra compromise.

Is our service mesh mTLS affected?

No. Mesh mTLS trusts only the mesh's private CA. Check where the mesh trusts public roots: egress TLS, JWKS fetches, and SPIFFE federation over https_web.

Is updating ca-certificates on the nodes enough?

No. Running processes keep the old roots until restarted. Images, some runtimes, ConfigMaps, and Secrets carry their own copies that need separate updates.

Which Kubernetes versions include the fix?

v1.37.2, v1.36.6, v1.35.10, and v1.34.13; 1.33 and older have none. The component images still trust the roots, so protection comes from the Go blocklist.

Has anyone actually been attacked?

No exploitation has been reported. Most clients in a cluster do not check Certificate Transparency, though, so attacks on them cannot be ruled out.

Will our vulnerability scanner catch it?

Partly. Trivy and Grype flag the Go standard library, but ca-certificates fixes carry no CVE, so stale bundles can scan clean. Check them by fingerprint.

For help assessing exposure and planning remediation, talk to Todea about a Platform Readiness Review.