Microsoft has published a report today detailing a never-before-seen series of attacks against Kubeflow, a toolkit for running machine learning (ML) operations on top of Kubernetes clusters.
The attacks have been going on since April this year, and Microsoft says its end-goal has been to install a cryptocurrency miner on Kubernetes clusters running Kubeflow instances exposed to the internet.
According to Yossi Weizman, a security researcher with Microsoft’s Azure Security Center, the company has detected these types of attacks against “tens of Kubernetes clusters” running Kubeflow.
But while the number of hijacked clusters is small in comparison to previous Kubernetes attacks, the profits for crooks and the financial losses to server owners are most likely much higher than other attacks seen before.
“Nodes that are used for ML tasks are often relatively powerful, and in some cases include GPUs,” Weizman explained.
“This fact makes Kubernetes clusters that are used for ML tasks a perfect target for crypto mining campaigns, which was the aim of this attack.”
Attacks began in April this year
Microsoft says it’s been tracking these attacks since April when it first saw them get underway and documented the first attack wave, before crooks expanded their focus from general-purpose Kubernetes instances to ML-focused clusters running Kubeflow.
As it learned more from its investigation into the early attacks, Microsoft now says it believes the most likely point of entry for the attacks are misconfigured Kubeflow instances.
In a report today, Microsoft said that Kubeflow admins most likely changed the Kubeflow default settings and exposed the toolkit’s admin panel on the internet. By default, the Kubeflow management panel is exposed only internally and accessible from inside the Kubernetes cluster.
Kubernetes threat matrix for the atacks on Kubeflow instances
Image: Microsoft
Weizman said that since April, a cryptomining gang has been scanning for these dashboards, accessing the internet-exposed admin panels, and deploying new server images to Kubeflow clusters, with these images focused on running XMRig, a Monero cryptocurrency mining application.
How to detect hacked Kubeflows
In case server administrators may want to investigate their clusters for any hacked Kubeflow instances, Weizman provided the following steps.
- Verify that the malicious container is not deployed in the cluster. The following command can help you to check it:
kubectl get pods –all-namespaces -o jsonpath=”{.items[*].spec.containers[*].image}” | grep -i ddsfdfsaadfs
- In case Kubeflow is deployed in the cluster, make sure that its dashboard isn’t exposed to the internet: check the type of the Istio ingress service by the following command and make sure that it is not a load balancer with a public IP:
kubectl get service istio-ingressgateway -n istio-system