Crypto-mining attacks in containerised environments are not new. Researchers have found different kinds of crypto mining activities running inside misconfigured containers. Microsoft, in April, had disclosed large-scale crypto-mining attacks against Kubernetes clusters which were discovered by Azure Security Center, thus helping users protect Kubernetes clusters from security threats.
Azure Security Center routinely searches for and does research on new attack vectors against Kubernetes workloads. Recently, a new crypto-mining attack was found lately by ASC, which targeted Kubeflow – a machine learning toolkit for Kubernetes. ASC witnessed the deployment of a malicious image from a public repository on several different clusters. By examining the image’s layers, it found that the image was running an XMRIG miner.
Nodes which are utilised for ML tasks are usually relatively powerful CPUs. This makes Kubernetes clusters an ideal target for crypto-mining campaigns. Using the processing power of the clusters, the hackers wanted to mine cryptocurrency, which in this case, was Monero (XMR).
Hackers Love Mining Monero
In Feb 2018, it was found that over half a million computers were powering a botnet (a web of malicious bot scripts) which mined about 9,000 Monero coins, amounting to $3.6 million at the time over a period of 9 months.
According to cybersecurity organisation Trend Micro, a hacker group known as Outlaw had started breaching Linux-based enterprise systems and data centres to take over computing power and mine Monero coins, an attack typically called as crypto-jacking.
“A couple of years ago, coin mining was a bubbling story. There were many threats that used infect”ed machines to mine cryptocurrencies at the expense of the victim. Mining coins on someone else’s machine could provide the attacker with free CPU resources from each infected system, so there was no need to steal directly from the victim. The infected machine would also deliver the block rewards from the mining operations into the attacker’s wallet,” Attila Marosi, Senior Threat Researcher, SophosLabs tells us.
According to experts, the reason why hackers prefer Monero over any other digital asset is primarily due to two reasons: One is that Monero mining can take place on non-specialised machines like CPUs, unlike Bitcoin which requires ASICs to mine the asset (or getting rewarded for validating transactions) on the blockchain. This means it is low capital intensive as hackers just have to breach others’ CPU machines to run mining scripts (rather than buying expensive ASICs themselves to mine cryptocurrency).
The main feature why malicious hackers and cybercriminals love Monero is that the cryptocurrency uses an obfuscated public blockchain where an outside observer cannot establish the source, the transaction amount or destination address with the help of Ring Signatures. Monero transactions are anonymous, making it easier for hackers to evade attempts to track funds as well as circumvent sanctions imposed on certain countries like Iran or North Korea.
How Kubeflow Machine Learning Framework Was Exploited
Kubeflow has expanded and become a well-known framework for performing machine learning tasks in Kubernetes. Kubeflow is a containerised service, meaning the many tasks run as containers in the cluster. So if somehow attackers gain access to Kubeflow, they have many ways to run the malicious image in clusters.
Kubeflow framework consists of several different services. Some of those services cover frameworks for training models, Katib and Jupyter notebook server, and others. The framework is divided into various namespaces, which are a collection of Kubeflow services, and resources created within a namespace are isolated to that namespace.
Kubeflow lets users create a Jupyter notebook server and enable them to choose the image for the notebook server, including a choice to use a custom image. This image does not have to be a legitimate notebook image, and so attackers can run their own image using this feature.
Attackers exploited this feature to deploy a malicious container from a real Jupyter notebook. The code, in this case, ran from the notebook server, which is a container with a mounted service account.
Exposed Dashboard To Internet Led To The Attack
According to Microsoft, Kubeflow presents its UI functionality through a dashboard, which is exposed by Istio ingress gateway and is default-accessible just internally, and users should utilise port-forward to access the dashboard, which tunnels the traffic through the Kubernetes API server.
In some cases, users may change the setting of the Istio Service to Load-Balancer, which exposes the Service, (i.e istio-ingress gateway in the namespace istio-system) to the internet. Microsoft said that some users preferred to do it for convenience to directly access the dashboard without the need of tunnelling through the Kubernetes API server. By opening the service to the Internet, users may access the dashboard directly. But, this operation also enables insecure access to the Kubeflow dashboard, which permits anyone to execute operations in Kubeflow, also including the deployment of new containers in Kubeflow cluster.
“If attackers have access to the dashboard, they have multiple methods to deploy a backdoor container in the cluster,” according to Yossi Weizman Security Research Software Engineer, Azure Security Center.
The cyber attacker utilised an exposed dashboard (Kubeflow dashboard in this case) for obtaining access to the cluster. The attacker headed to move laterally and deployed the container using the mounted service account which contained the cryptocurrency miner.
According to experts, only a few professionals have in-depth experience when it comes to container management services like Kubernetes, and therefore security can be ignored. Consequently, Kubernetes environment needs to be configured properly to be secure, assuring nothing is exposed for cyber attackers. Microsoft has released instructions on how to check if your cluster is impacted and what security measures can be taken in future to prevent such attacks.
Provide your comments below
