For three years in a row, our Global DevSecOps Survey found testing was the number one reason (by large margins) for release delays. A lack of automated software testing, combined with too many manual tests conducted too late in the process, was a story told time after time, and it certainly was one without any kind of happy ending.
Despite the undeniable progress DevOps has brought to software development, integrating automated software testing into the lifecycle has remained an elusive goal for many teams. Here’s a look at why testing is such a difficult step to get right, and how an integrated DevOps Platform can bring much-needed structure to the process.
The state of automated software testing
According to our 2021 Survey, it’s safe to say respondents are frustrated with software testing.
“Testing can be slow in both writing and running.”
“Testing delays everything.”
While there is forward momentum (almost 25% of teams say they’re fully automated - more than double the number from 2020), the same percentage reported zero automation or that they’re just beginning to think about it.
“Automated testing is ignored ‘due to time constraints.’”
But even teams that haven’t ignored automated software testing are hamstrung because the vast majority don’t give developers scan results within their IDEs. Fewer than 25% of teams enable SAST lite scanners in a web IDE and only 20% put results in a web pipeline report for developers. The situation is even worse when it comes to DAST, dependency and container scans: just 16% make DAST/dependency scan data available, and 14% do the same for container scans. While the percentage of teams reporting full automated software testing increased from 2020 to 2021, the percentage giving devs access to key test data barely changed in the same time frame.
Context switching makes everything hard
The fact that developers can’t easily get access to test results is a huge productivity blocker. “The best time to (fix bugs) is when I’m in "flow" - right when I’m writing the code and have a mental model of all of the things and how they are interconnected,” explained Brendan O’Leary, senior developer evangelist at GitLab, in a blog post last year talking about the developer-security divide. “So that’s basically the same day or same week as when I wrote it.”
Elevating your DevOps skills? Join us at Commit at KubeCon - Oct. 11!
So while not getting results “in the flow” is a huge stumbling block, developers are adamant about the importance of testing. When we asked developer respondents in our 2021 Survey what they wished they could do more of, testing was, by far, the number one response.
What’s the solution to this conundrum? More automation, more AI/ML and a DevOps platform to make everything seamlessly interconnected, visible and actionable.
Geo-sharing company Glympse offers an object lesson on the benefits of a DevOps platform. The company was using approximately 20 tools to get its software out the door, but after moving to GitLab’s DevOps Platform, the process was dramatically streamlined. Deployments have dropped from four hours to less than 30 minutes, and deployment fatigue, particularly around testing and code reviews, has vanished.
The struggle is real, but worth it
For teams who’ve tamed the automated software testing beast, and are humming along in their DevOps practices, the benefits are substantial. Here’s what they told us in our 2021 Survey:
“We are not relying on developers to have remembered to create and run tests for their code before deploying.”
“We automate everything possible, to be able to test our product ‘like in real life’ without any downside. This increases confidence and simplifies tests for everything.”
“Integration testing has been a big plus in how confident we are to release automatically and deliver a version. We are now able to deliver any day.”
“It helps that devs don't need to keep track of test running; they just need to push and pipeline will check their code before merge to master.”
