{"id":100749,"date":"2019-08-08T14:34:21","date_gmt":"2019-08-08T21:34:21","guid":{"rendered":"https:\/\/www.paloaltonetworks.com\/blog\/?p=100749"},"modified":"2023-09-08T05:55:44","modified_gmt":"2023-09-08T12:55:44","slug":"kubernetes-penetration-test","status":"publish","type":"post","link":"https:\/\/origin-researchcenter.paloaltonetworks.com\/blog\/2019\/08\/kubernetes-penetration-test\/","title":{"rendered":"Kubernetes Penetration Test Report: Insights and Twistlock Response"},"content":{"rendered":"<p><span style=\"font-size: 12pt;\"><span style=\"font-weight: 400;\">The Cloud Native Computing Foundation (CNCF) late last year commissioned a penetration test to identify unknown security vulnerabilities and design weaknesses in Kubernetes. The <\/span><a href=\"https:\/\/github.com\/kubernetes\/community\/blob\/master\/wg-security-audit\/findings\/Kubernetes%20Final%20Report.pdf\" rel=\"nofollow,noopener\" ><span style=\"font-weight: 400;\">final report<\/span><\/a><span style=\"font-weight: 400;\"> is posted <\/span><a href=\"https:\/\/github.com\/kubernetes\/community\/tree\/master\/wg-security-audit\" rel=\"nofollow,noopener\" ><span style=\"font-weight: 400;\">in the working group's repository<\/span><\/a><span style=\"font-weight: 400;\">.<\/span><\/span><\/p>\n<p><span style=\"font-size: 12pt;\"><span style=\"font-weight: 400;\">When done well, penetration tests provide methods for improving software security quality. The Kubernetes test was thorough and well designed. It resulted in dozens of findings, including identification of many new vulnerabilities and recommendations that new security feature enhancements be implemented. The Kubernetes project just opened <\/span><a href=\"https:\/\/github.com\/kubernetes\/kubernetes\/issues\/81146\" rel=\"nofollow,noopener\" ><span style=\"font-weight: 400;\">#81146<\/span><\/a><span style=\"font-weight: 400;\"> as a single tracker to follow progress of the 37 issues that were identified.\u00a0<\/span><\/span><\/p>\n<p><span style=\"font-weight: 400; font-size: 12pt;\">We\u2019ve received many questions about this report from our customers. Answers to the most common questions are posted below. We will update this post as more information becomes available.<\/span><\/p>\n<h2><span style=\"font-size: 14pt;\">1. Why was the report released prior to all issues being addressed?<\/span><\/h2>\n<p><span style=\"font-size: 12pt;\"><span style=\"font-weight: 400;\">Disclosure of the report findings wasn\u2019t announced or broadly coordinated prior to its release. While the community was aware of the penetration test, full disclosure of results, including unfixed vulnerabilities, was a surprise. In the spirit of improving everyone\u2019s security, Ariel Zelivansky, who leads our research team, opened <\/span><a href=\"https:\/\/github.com\/kubernetes\/community\/issues\/3982\" rel=\"nofollow,noopener\" ><span style=\"font-weight: 400;\">#3982<\/span><\/a><span style=\"font-weight: 400;\"> in the community repo to discuss developing more coordinated processes for disclosure and proactive remediation going forward. Future vulnerability findings will ideally be coordinated among maintainers and vendors so there are already fixes available for vulnerabilities at the time of disclosure.<\/span><\/span><\/p>\n<p><span style=\"font-size: 14pt;\"><b>2. Does Twistlock detect these new vulnerabilities yet?<\/b><\/span><\/p>\n<p><span style=\"font-size: 12pt;\"><span style=\"font-weight: 400;\">Our <\/span><a href=\"https:\/\/www.twistlock.com\/2018\/10\/02\/building-ultimate-cloud-native-vulnerability-feed\/\" rel=\"nofollow,noopener\" ><span style=\"font-weight: 400;\">Intelligence Stream<\/span><\/a><span style=\"font-weight: 400;\"> consumes vulnerability data from dozens of vendors and upstream providers to build the data set used to identify vulnerabilities within each customer environment. Since this report was unexpectedly released, many of these vulnerabilities do not yet have CVEs assigned and thus many vendors have not yet had an opportunity to assess whether their distributions are vulnerable. As vendors conduct these assessments and publish their results, the Intelligence Stream will automatically pick up these vulnerabilities and enable Twistlock to detect them. Vendor analysis of newly disclosed vulnerabilities typically happens quickly, often within hours of disclosure. In this case, though, it may take a little longer for some of the lower severity findings to be evaluated and for CVE data to be published about them due to the sheer volume of the findings.\u00a0\u00a0<\/span><\/span><\/p>\n<p><span style=\"font-weight: 400; font-size: 12pt;\">No customer action is needed. As soon as vendors publish vulnerability data, it will immediately be picked up by the Intelligence Stream and used to detect vulnerabilities in your environment.<\/span><\/p>\n<p><span style=\"font-size: 14pt;\"><b>3. How does Twistlock protect me from these vulnerabilities?<\/b><\/span><\/p>\n<p><span style=\"font-weight: 400; font-size: 12pt;\">Twistlock includes a variety of different controls that mitigate findings in the report. Of the 37 findings, only five are rated as high impact. Of these five, one is a suggested enhancement, rather than a vulnerability. We examine those five findings below.<\/span><\/p>\n<h2><span style=\"font-size: 14pt;\"><strong>4. How does Twistlock protect against these issues?<\/strong><\/span><\/h2>\n<p><span style=\"font-size: 12pt;\"><b>hostPath PersistentVolumes enable PodSecurityPolicy bypass<\/b><\/span><\/p>\n<p><span style=\"font-weight: 400; font-size: 12pt;\">Twistlock provides two mitigations: First, our Kubernetes audit monitoring alerts on pods created with additional privileges (accessing host mounts). Second, we have a compliance rule that alerts \/ blocks in cases where pods are created with host mounts (this is compliance check #55 within Twistlock).<\/span><\/p>\n<p><span style=\"font-size: 12pt;\"><b>Kubernetes does not facilitate certificate revocation<\/b><\/span><\/p>\n<p><span style=\"font-weight: 400; font-size: 12pt;\">This is the suggested security enhancement. Re-keying certificate chains is overly burdensome. The finding recommends simplifying it. While we agree that this would be a useful security advancement, it\u2019s not a vulnerability and doesn\u2019t create any direct risk to users today. As this is a suggested platform improvement, it\u2019s outside the scope of our focus as a security product.<\/span><\/p>\n<p><span style=\"font-size: 12pt;\"><b>HTTPS connections are not authenticated<\/b><\/span><\/p>\n<p><span style=\"font-weight: 400; font-size: 12pt;\">The core risk here is the eventual access to etcd. Pods don\u2019t normally access etcd directly and as our Cloud Native Network Firewall automatically learns normal traffic patterns, we\u2019d see and block this anomalous connection. Further, the attack requires creating a malicious kubelet, which we help customers mitigate with our Trusted Images feature that only allows software to run from approved registries and repositories.<\/span><\/p>\n<p><span style=\"font-size: 12pt;\"><b>TOCTOU when moving PID to manager\u2019s cgroup via kubelet<\/b><\/span><\/p>\n<p><span style=\"font-weight: 400; font-size: 12pt;\">In this vulnerability, the process inside a container eventually writes to devices on the host. Our runtime defense feature already automatically detects and prevents these types of attacks. Twistlock will automatically learn that this is not normal file system access behavior and automatically prevent it.<\/span><\/p>\n<p><span style=\"font-size: 12pt;\"><b>Improperly patched directory traversal in kubectl cp<\/b><\/span><\/p>\n<p><span style=\"font-size: 12pt;\"><span style=\"font-weight: 400;\">This is <\/span><a href=\"https:\/\/discuss.kubernetes.io\/t\/announce-security-release-of-kubernetes-kubectl-potential-directory-traversal-releases-1-11-9-1-12-7-1-13-5-and-1-14-0-cve-2019-1002101\/5712\" rel=\"nofollow,noopener\" ><span style=\"font-weight: 400;\">CVE-2019-1002101<\/span><\/a><span style=\"font-weight: 400;\">, which Ariel found earlier this year. You can read about it in <\/span><a href=\"https:\/\/www.twistlock.com\/labs-blog\/disclosing-directory-traversal-vulnerability-kubernetes-copy-cve-2019-1002101\/\" rel=\"nofollow,noopener\" ><span style=\"font-weight: 400;\">our original blog post<\/span><\/a><span style=\"font-weight: 400;\">. The CVE involves a malicious image (which can be prevented with Trusted Images) and can be mitigated by requiring a read only rootfs, for which we provide a compliance check.<\/span><\/span><\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A recent audit identified previously undisclosed security vulnerabilities and design weaknesses in Kubernetes.<br \/>\nHave you got all the details? Our Twistlock team breaks down the report.<\/p>\n","protected":false},"author":689,"featured_media":97341,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[6768],"tags":[6731,6808],"coauthors":[6821,6875,7101],"class_list":["post-100749","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-secure-the-cloud","tag-kubernetes","tag-twistlock"],"jetpack_featured_media_url":"https:\/\/origin-researchcenter.paloaltonetworks.com\/blog\/wp-content\/uploads\/2019\/03\/Containers.png","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/origin-researchcenter.paloaltonetworks.com\/blog\/wp-json\/wp\/v2\/posts\/100749","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/origin-researchcenter.paloaltonetworks.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/origin-researchcenter.paloaltonetworks.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/origin-researchcenter.paloaltonetworks.com\/blog\/wp-json\/wp\/v2\/users\/689"}],"replies":[{"embeddable":true,"href":"https:\/\/origin-researchcenter.paloaltonetworks.com\/blog\/wp-json\/wp\/v2\/comments?post=100749"}],"version-history":[{"count":6,"href":"https:\/\/origin-researchcenter.paloaltonetworks.com\/blog\/wp-json\/wp\/v2\/posts\/100749\/revisions"}],"predecessor-version":[{"id":100810,"href":"https:\/\/origin-researchcenter.paloaltonetworks.com\/blog\/wp-json\/wp\/v2\/posts\/100749\/revisions\/100810"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/origin-researchcenter.paloaltonetworks.com\/blog\/wp-json\/wp\/v2\/media\/97341"}],"wp:attachment":[{"href":"https:\/\/origin-researchcenter.paloaltonetworks.com\/blog\/wp-json\/wp\/v2\/media?parent=100749"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/origin-researchcenter.paloaltonetworks.com\/blog\/wp-json\/wp\/v2\/categories?post=100749"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/origin-researchcenter.paloaltonetworks.com\/blog\/wp-json\/wp\/v2\/tags?post=100749"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/origin-researchcenter.paloaltonetworks.com\/blog\/wp-json\/wp\/v2\/coauthors?post=100749"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}