In a traditional system, with manually configured servers that are all different, there might be vulnerabilities on different systems, and attackers can see admins trying to track or block them. “They're exceptional measures going on in the static slow moving network, but with DevOps you have a very fast automated release pipeline, you're constantly redeploying,” says Guckenheimer. “If you are deploying everywhere on your net it doesn't look like a special action taken against the attackers.”
Don’t automate secrets
If DevOps ought to improve security, why doesn't that always happen? Sometimes the problem is that companies are using DevOps tools but not really understanding DevOps.
Upguard’s Sharp-Paul says “The primary error we hear about is the attempt to automate before you know what you’ve got. Someone got a budget for DevOps and went out and picked up Puppet Enterprise or Chef and wound up hitting roadblock after roadblock because they don’t understand the actual working nature of their infrastructure. They just bought into the sense of urgency to “do DevOps” and jumped into the deep end without looking.” The security of that approach should be fairly obvious by now.
Guckenheimer agrees. “I don’t think DevOps replaces any good security practices like key management, key rotation, changing defaults or what have you. I do think it changes the pace at which things move; I store more as code and I rebuild the infrastructure all the time, because I’m treating the infrastructure as code whether I'm doing PaaS or Chef or Puppet. That way, you can apply the good practices better. If you ignore them you're vulnerable. The security problems are there and they're exacerbated or alleviated; [with DevOps] I think you have an amplifier.”
There are some common security mistakes. If your build automation means putting your cloud credentials in your code repository or your deployment script, they’re vulnerable. Searching GitHub for AWS and Azure credentials reveals that many people are making the same mistake as Ashley Madison, Uber and D-Link.
Ashley Madison’s leaked code included hard-coded AWS tokens, database credentials, certificate private keys and other credentials. Uber had a database containing personal information about drivers compromised in 2014, after storing the key in a publicly available repo and D-Link recently published its private code signing keys in the open source code for a firmware update. Your cloud credentials are likely to end up subsidizing Bitcoin miners, who scan GitHub for keys and use them to run up hundreds or thousands of dollars of bills.
“What people are not necessarily doing in DevOps which they ought to be doing, is doing good secrets management, using Azure Key Vault and equivalents. What I need to do is keep all the secrets external and as I go from environment to environment though the release pipeline, I need to bind the secrets separately for each environment and make sure those secrets are all external, secure, encrypted and rotated continually,” Guckenheimer says.
Realistically, Azure CTO Mark Russinovich suggests a combination of education to teach best practices and static analysis tools to enforce them. “As much as possible, it needs to be built into the CI/CD [process] so developers don't have to opt in to it, it just happens,” says Russinovich. “You can tell them all day long ‘don't check secrets into code’ but somebody is going to do it, so what you need to do is let them know right then; don't catch it down stream, catch it right when they check it in so that they know they can't do that.”
But that enforcement has to be part an overall DevOps culture, Russinovich says. “What we believe is developers are actually mostly well intentioned. They might forget, there might be someone who hasn't taken training, hasn't taken it in a while, it slips their mind and the gates are just to catch that, because it's too much of a risk not to have them.”
The way the Visual Studio team handles credentials might be a good model. “We run Visual Studio cloud services; we have well over a thousand secrets in there, constantly rotated,” Guckenheimer says. “They’re well rotated and developers do not have access to any of the secrets for production, they're on a separate domain. We practice Just Enough Admin. So were you to somehow get access to the account of the director of engineering – through some back door or social engineering, or brute force it or steal his badge – it wouldn't do you a lot of good, in terms of hacking to get to the production service, because the secrets would be rotated. He would not have admin privileges, because those get enabled on a ‘just enough,’ time-limited expiration basis.”
At best, Russinovich claims, “you might get access to a little bit of source code he could see from his repository – and it would probably be pretty obvious to someone that his account was compromised, and we would shut that off.”
Beyond protecting credentials, DevOps gives you an opportunity to monitor not just the code your developers are writing but third-party code you’re relying on; Forrester’s Bittner suggests viewing that as part of your supply chain management and bill of materials. Monitoring the binary artifacts and components you use, with services like Sonatype and Artifactory, is a relatively novel idea but Guckenheimer predicts it will become common practice, in light of issues like Heartbleed, Poodle and Shellshock.
Shift security left
A more fundamental problem is the common disconnect between DevOps and security.
Getting that kind of visibility about problems and vulnerabilities is part of the way DevOps can help you “shift left” on security, by catching flaws earlier in the development cycle – as well as spotting vulnerabilities in configuration before you deploy to a live system. The potential for security problems and improvements applies as much to infrastructure and operations as to developers, points out Forrester analyst Amy DeMartine, noting that 47 percent of web applications servers have vulnerabilities because they’re misconfigured.
Done properly, she believes DevOps should always include security, but ironically, the reason they don’t is part of the problem DevOps sets out to solve.
[Related: Why are we racing to DevOps?]
“I think the reason many companies start DevOps and forget the security folks, is that it's yet another cultural divide,” says DeMartine. “Security people speak a totally different language – breaches, incidents, vulnerabilities and exposures – and everyone puts them at the very end of the development lifecycle, if they’re remembered at all.”