Since the term was first coined in 2017, GitOps has provided a new operating model for developing, delivering, testing, and deploying cloud-native infrastructure. GitOps and infrastructure as code (IaC) go hand-in-hand, enabling the use of version control, continuous integration (CI), continuous delivery (CD), and collaboration tools to deploy and manage infrastructure. With 208x more deployments and a 7x lower change failure rate, GitOps is a necessary component for teams to reach “the elite performer group”, as outlined by DevOps Research and Assessment (DORA).
One of the primary tenets of GitOps is that Git is the only interface that developers interact with, leaving the rest of the integration and deployment process to automation. This trust in automation enables complete version control and auditability of all infrastructure templates and, ultimately, increased developer velocity.
It also provides clean separations of concerns. Developers pushing code to the repository that ultimately gets deployed do not need access to the CI or CD platforms, nor the production environment. This separation helps enforce the security practice of least privilege, where fewer identities with access mean a smaller attack surface.
In addition to affording more control over how infrastructure is provisioned, GitOps places more responsibility on developers when it comes to security. The good news is, GitOps also makes it easy to embed security scanning and guardrails throughout the entire development and delivery processes.
Here are seven considerations for embedding cloud security controls into your GitOps flow.
1. Embed Security Scanning Early
One of GitOps’ main benefits is that much of the hands-on deployment process is automated in pipelines. Developers should write code, test it locally, submit a pull/merge request, commit, and move on. Because GitOps abstracts developers from the process after code is committed, it’s important to provide tooling for security scanning as early in the GitOps flow as possible.
Embedding security into integrated development environments (IDEs) or as a pre-commit hook is a great way to surface feedback to developers in the way they’re used to receiving feedback from other tests such as unit and integration tests. The earlier a test fails, the less friction.
Try to avoid reinventing the wheel when looking for points to add security guardrails. Your version control system (VCS) and continuous integration (CI) pipelines are both great places to embed scanning of your infrastructure codebase, which leads us to our next point.
2. Integrate Across the Development Cycle
While it may seem redundant, it’s absolutely crucial to enforce security and compliance policies at every part of the GitOps flow. The idea is to scan IaC templates before IaC gets committed to a repository, once it’s more fully compiled, and continuously once it has been deployed. This provides consistent, frequent feedback to developers to maximize the chance that misconfigurations get fixed quickly.
Additionally, new rulesets for enforcing security best practices are constantly being added to scanners as new threats emerge, and technologies evolve. For example, TLS 1.2 has recently become the minimum encryption standard. Templates with TLS 1.1 before this policy was enforced may have previously passed CI scans. Without making any changes, those templates would be out of compliance. By periodically scanning templates throughout the development lifecycle, you can catch new issues in existing code.
3. Provide Actionable Feedback
If you’re following GitOps practices, chances are you value speed and agility. To take full advantage of those benefits, however, you need to be fully equipped to take action quickly when security feedback gets surfaced.
Blocking builds or deployments when your teams are trying to push out updates as fast as possible without remediation or guidance creates friction. Within environments wherein security feedback is noisy and unintuitive, it’s not uncommon for developers to find ways to circumvent it altogether.
That’s why it’s important to keep the end-user — developers — in mind when embedding and enforcing guardrails. To ensure developers can easily comply, make sure that any security finding includes at the very least prioritization and remediation guidance with offending line items. You may take this a step further with an IDE extension or platform (such as those by Prisma Cloud) that provide the actual fixes.
4. Prevent Cloud Drift
All changes made to your cloud environment should be in IaC templates hosted and version-controlled in your repository. As the source of truth and a key component of GitOps, those templates should be enforced by reconciliation loops and immutable infrastructure.
In a perfect world, these loops continuously check the desired state specified by the IaC templates against the actual state of the resources in production. Updates to infrastructure, such as operating system updates, wipe out the old instances, and stand up new ones.
Of course, this is not a perfect world.
Most breaking changes can be rolled back with a revert command, but occasionally, fire fighting results in manual changes made to your running resources, creating drift. Going back to our second point of scanning across the development lifecycle, make sure you have the proper visibility and monitoring in place to keep your teams accountable for cloud drift. When identified, make sure that those changes get reverted ASAP, or countermeasures are added to IaC templates and pushed through pipelines to bring runtime environments in line with templates. Developers looking at their IaC templates should have the same picture of the deployed infrastructure, and even the slightest discrepancy can lead to risk down the road.
5. Track Progress Over Time
The point of checking infrastructure against policies and enforcing resulting feedback is to improve the security of your codebase and deployed infrastructure over time. Tracking progress comes down to the number of existing misconfigurations that are patched since being identified and the number of new misconfigurations introduced.
Both trends should decrease over time, and you should also have a benchmark (or better yet, an SLA) for how long it takes to patch identified misconfigurations. With enough exposure to security feedback as part of your GitOps flow, developers will build up the muscle memory to prevent, find, and fix errors.
Having baselines to monitor anomalies also builds that muscle memory. For example, suppose there’s a spike in misconfigurations. In that case, developers can track down whether it’s related to a new deployment or existing infrastructure is in violation of a new policy or security best practice.
6. Don’t Add Security in a Vacuum
In GitOps, collaboration is vital for success. Developers cannot blindly deploy infrastructure without passing the proper security guardrails, and security cannot assume those guardrails are perfect. Developers and security teams need to work together to iterate and improve on the process and tooling.
By nature, policies are binary and lack the full context of your business, needs, and environment. Because of that, guardrails can result in noisy and irrelevant findings. To minimize noise and tune feedback for the highest signal, work with security teams to determine what to let pass through. Additionally, security teams can provide feedback on what is and isn’t working to developer teams to continue reinforcing the positive feedback loop.
Embrace Automation to Adopt GitOps Best Practices
The paradigm of GitOps significantly increases release velocity and opens up new opportunities for improved security posture. To successfully embed IaC scanning and governance into your GitOps flows, both your development and security teams must lean wholly into automation. This process requires close collaboration and agreement between stakeholders to be done right. Enforcing strict automation through a pipeline is difficult, but the results greatly outweigh the costs when done intentionally and collaboratively.
Want to learn more cloud security automation best practices? Download the checklist and get six power rules for shifting your security left.