Minding the Machines: Security Challenges in CI/CD and Containerization

Innovations in computing tend to create new, bigger vulnerabilities. The excitement that greets the innovation also blinds people to expanded risk exposure. We’re seeing this now with the rising popularity of containerization, Continuous Integration and Continuous Delivery (CI/CD) of code and microservices. These innovations, which enable healthy increases in agility and software delivery speed, also result in deep, complex security weaknesses. New security solutions are coming on the market, though, to mitigate the risks.

Briefly, containerization and microservices are modes of software development and deployment that divide applications into small, self-contained units. In contrast to a conventional application, which comprises a single code base, a containerized app is made up of many specialized “containers,” each of which performs a specific function. The containers communicate with each other in order to execute the app’s functionality. This is known as a “machine-to-machine” interaction.  Systems like Kubernetes do the work of container orchestration. Micro services are comparable in concept, though different in execution. (But, not worth delving into too much here.)

There are many advantages to this approach: It makes hosting more efficient, with containers using a more proportional allocation of compute resources than a single code base for the app. The architecture also makes it possible to create new functions on a granular basis—increasing the speed, flexibility and efficiency of development teams.

CI/CD complements containerization and microservices. In contrast to the traditional de-install/re-install software update process, with CI/CD, IT operations teams can continuously place new code right into a production application while it’s running. It’s sort of like changing the tire while the car is moving. CI/CD accelerates software development and releasing.

Gartner blog author Christy Pettey highlights a problem with all of this, however. While projecting that containerization will jump from today’s 30% level of adoption to over 75% by 2022, she cites her colleague, Arun Chandrasekaran, Distinguished Vice President Analyst, Gartner, who raises security concerns. He said, “I&O teams [Infrastructure and operations] will need to ensure the security and isolation of containers in production environments while simultaneously mitigating operational concerns around availability, performance and integrity of container environments.”

To understand Gartner’s concerns about security, consider the problems inherent in securing machine-to-machine interactions: A container could easily contain malware, perhaps by way of a compromised open source library or an external actor penetrating the development environment. This puts the attacker right into the application, hiding in plain sight. A great deal of cybersecurity is focused on monitoring human activity and searching for code that doesn’t belong on the network. A tainted container evades both practices. Unlike a human hacker, the container is always inside. It doesn’t need network access to execute malicious acts like exfiltrating data. It’s nearly impossible to detect because it belongs right where it is.

A container could easily contain malware, perhaps by way of a compromised open source library or an external actor penetrating the development environment. This puts the attacker right into the application, hiding in plain sight.

The blog recommends that security be embedded in the DevOps (software development and releasing operations) process. The containerized environment should be secured across its entire life cycle, including build, development, deployment and runtime. Wise in intent, this “secure development practices” approach can be nearly impossible to realize without specialized tooling. Developers who are expert in security are rare. Developers generally don’t think security is their responsibility. And, they don’t have the time to check their containers for embedded malware.

Such machine-to-machine security weakness is the problem that firms like Portshift have set out to solve. Making Kubernetes its main focus, Portshift has developed a solution that establishes a digital signature for each container going into production. It can then monitor a containerized application to determine if any of its components do not belong, i.e. they’re unknown or malicious. The solution also monitors the machine-to-machine interactions to detect possible malicious activity.

“We’re giving the containers an identity, so we can track what’s going on inside a containerized application,” said Zohar Kaufman, Portshift’s VP of R&D.

“We’re giving the containers an identity, so we can track what’s going on inside a containerized application,” said Zohar Kaufman, Portshift’s VP of R&D. “The goal is to give organizations control over how containers and applications interact, where they run and what they can do. Communication will only occur after identity-based authentication.” No container can be deployed through CI/CD without authentication. The solution integrates into many popular DevOps and CI/CD environments.

He added, “The unfortunate reality is that today’s containerized apps, combined with the way they’re assembled, make it extremely difficult for a human being or traditional security monitoring tools to comprehend what’s happening when it goes into production. There are too many moving parts, too many machine-based interactions. We eliminate these blind spots.”

“The unfortunate reality is that today’s containerized apps, combined with the way they’re assembled, make it extremely difficult for a human being or traditional security monitoring tools to comprehend what’s happening when it goes into production.”

The challenges are only going to get more intense. As Kaufman noted, “A lot of companies are moving from DevOps to what we call ‘No-Ops.’ Developers will be pushing code into production on their own, with no IT ops support. This might be good for speed and efficiency, but it further widens the attack surface area. Without some sort of container identification, you’re going to have trouble.”

Photo by Kaique Rocha from Pexels