DevSecOps weekly, fourth issue

So far I have written a bunch about the tools of the trade when it comes to DevSecOps, but now I feel that it is time to tackle on the ideology itself and find out why only the tools are not enough in order for DevSecOps to work. After all, DevSecOps is not just about the tools, but rather a cultural movement. I’ll add my own thoughts from a viewpoint of an outsider, since I have no real work experience in software development, who has managed a team and understand why it is important to create a culture to a workplace. I also feel that this topic is important, since now that after almost a month of studies are behind me, we are starting our little projects at the training, but more on later posts.

DevSecOps overview

I’d like to start with my own interpretation of an image I received from one of the lecturers, which represents the whole DevSecOps procedure. Like the old saying goes, picture’s worth thousand words:

DevSecOps

But let’s not leave it there, but rather go through why DevSecOps is represent such a big part of this image rather than just simply being only about development and operations with extra security. It is an holistic ideology that is along the whole development pipeline, breaking barriers that have been holding development back before. During this post I will go through each step pictured and what is the role of DevSecOps in each part and how it helps mold the modern software development to the way it has been going. I will use couple of excellent sources I have found on the subject; DevOpsSec by Jim Ward and Building a Modern Security Program by Rebecca Huehls and Zane Lackey. I have also pieced together my own views and understanding of the subject thus far from my studies.

Like the image suggests before the process even begins, there must be a problem or an idea that needs to be solved. Service Design is created to tackle this more closely, thus I will not go in detail about that and keep focus on inner workings of DevSecOps in each of the parts. Before going on, I’d like to also point out why is security so important to emphasize on the already existing DevOps-ideology, that also takes security to account by default? Answer is rather simple; before modern agile development, with its continuous improvement (seen as word “development” on the picture), the role of security was handed to designed team at the end of development. When development and operations is cyclic, the need for identical security measures is obvious, when lacking security practices cancel even the most efficient DevOps-cycle. I like to think that even that something has a existing guidelines and processes available, it doesn’t really exist if there is lack of weight put on it. How to add weight some may ask? Well of course by giving it a name, Service Design is a good example of that, since it has been thing before but now that it has a name it is more recognizable. Thus, from now on DevOps must mean DevSecOps, so hop on and help me spread the message!

Design

First is something that I have been talking on two occasions in by my blog before; design, be it a Service Design mentioned above or UX-design. There is a role here for DevSecOps too. When something is designed, be it a software, or any other product, it is a good idea to reflect upon the security side of it. Jim Bird writes, that;

Secure design in DevOps begins by building on top of secure libraries and frameworks

So rather than thinking of the security of the software after thousands upon thousands of lines of code has been written and dozens of libraries, both in house developed or from external sources, have been introduced to project, DevSecOps aims to plan ahead. With effective risk management it is possible to determine which libraries are safe and what kind of programming methods are employed. Things to consider are, among others, does the developers understand how to even use the languages and frameworks safely? Are unsecure practices, such as addition of sensitive data elements to code, used? In short, when designing DevSecOps role is to work as a gatekeeper, whose job is to prevent bad habits and unsecure working methods. After all we are all humans, and humans are prone to err, but when ability to identify possible risks is capitalized, this risk can be minimized with proper guidance. Case in point; since development is iterative, security must be very strict. Even when there might be need to deviate from set of tools, it is important for DevSecOps to “stick by their guns” and not let team to give up on chosen tools, libraries and runtime stack, even though what is being developed is still up for a change.

In my mind I see DevSecOps as sort of a safety representative, that instead of managing health and safety at work, is managing health and safety of the software being developed. I highly recommend reading through part 4 and especially chapter called Secure Design in DevOps of Jim Bird’s book. He gives out excellent points on how PayPal does assessment when an new application starts. Bottom line from what I gathered from the PayPal way of doing is creating of certain guidelines on how to work and what is okay to do and what is not. It is very relatable to working on construction site, since if something is done without care, the whole building operation is at risk, along with the workers. When something goes wrong in software development, it doesn’t present any physical danger to humans involved, but I fully agree that creating culture of safety to any working environment makes people working on it feel less worried and more likely to do their best. Job security at its finest :wink:

Implement

This is the part where the development happens, meaning that lines of code are written, and the software is being build. This, along with testing, is the most visible part of DevSecOps, since there are a lot of visible tools and methods for these, which I have listed on previously here. But what other measures can be done to apply DevSecOps ideology to implementation phase? Since application must be secure by default, there must be shift in work culture as well. Strong cooperation amongst the security engineers and software engineers is a must. However, the most important thing is, that the security of the application should be in mind of everyone involved in the process, not just those whose job title describes them in charge of security. This can be achieved by making the security both as convenient and as available as possible. Think of this the same way that DevOps-tools, which provide convenience in other aspects of programming, such as continuous integration that tests the code automatically. For this, there is a method; security as a self-service. Rather than telling developers exactly what to do, it places trust in them to handle the security themselves. This can be achieved by letting the tools bring the security to the everyday work cycle. These can be anything from making sure that test environments provide understandable and on the point feedback on the code written, to placing automated security features to development environment used.

In my mind this exact phase shows how DevSecOps is all about both culture and tools. Rather than telling, even forcing, exactly what to do by holding each developer hand and yelling by their side when they do their work, tools can be put in place that allows programmers to carry out security themself. I think it is a really beautiful system since it places the trust on across all developer positions, ergo rendering input and contribution of even the lower ranked workers important. I strongly believe, that when everybody has a sense of trust placed upon them, their work ethics rise much higher and they value their own work much higher. When both of these are on a rise, so does the need to do your absolute best for the team, the project and workplace in general.

Happy worker is securely productive worker, so to speak. Bringing the absolute best also means that security measures are taking in to account and it doesn’t become “that irritating voice that nags me to do the right thing”. When security is all around, always available, but still “invisible”, it doesn’t become irritating routine but rather something that is natural, like breathing.

Test

I mentioned good testing feedback as a one way of influencing developers to work in more secure manner, but what else DevSecOps-culture brings to testing? One way is to gamification of the discovery of programming errors in the application, called Bug-Bounty Program. It is like bounty hunting, but instead catching dangerous criminals, a team of ethical researchers try to find out security vulnerabilities, or bug, of a program being developed. Each bug reported then nets them a reward for their finding. Usually these are closed for public and only selected security professional can try their best, but I also found out that it is even done for free, in an open way of teaching and learning security. Open or closed, I strongly believe that this is very positive thing in a cultural sense, since it creates a safe space for programmers to make mistakes, that can be corrected before anything major happens in an actual publicly released program. Quick fixing a released product can break the confidence of developers. Among other things these can be very stressful and motivation breaking situations. I said it once and I say it again, humans are prone to err, but it is much more productive to make errors in a safe environment where learning can happen rather than quickly fixing and not even fully realizing what and how something is being fixed on until later when there is actual time to study the issue, if there is time.

Since modern software development is cyclic, as seen in the picture with word “development”, and for it even possible, a Continuous Delivery pipeline needs to be built. This pipeline has acceptance, stress, and performance tests, as well as security tests and other checks, with pass/fail feedback to the team. All these things are done automatically. However, nothing just automatically transforms, to well automatic, and careful planning and cooperation between all teams that are involved in development. Again, a major cultural shift from highly closed teams that only concentrate on their own field of expertise to more open structure! So instead of everyone staying in their own “foxhole”, a good DevSecOps brings everyone together, and starts exposing details about how each team works, and especially how they want to work. In consequence lowering organizational structure and strengthening social exchange between teams. I personally do not like the idea where the left hand doesn’t know what the right hand is doing in a workplace, meaning that when decisions are done in one team, the other is kept completely oblivious about that decision. I strongly believe that openness and sharing of ideas is a key to shared victory and finding of best practices. To further demonstrate this, let’s flip the situation, how would non-cooperative teams work when they find out that there is a bug in a production version of the system? In a worst case; accusations start flying (“it is those lame back-end developers' fault”) and nobody takes responsibility. Because of the DevSecOps “forcing” teamplay to the software development, less of onus is put on single team, and fixing is a team effort that everybody shares. You lose as a team and win as a team, sounds much fairer than win as a team and lose as an individual.

Deploy and Analyze

I will bunch these two together since they go hand in hand, when the software is finally deployed to end users, the need for continuous analyzing for feedback, security issues and competition starts. This is the part where the tool of Continuous Delivery is also used, since its backbone of DevSecOps. Jim Bird puts it the best:

It provides an automated framework for making software and infrastructure changes, pushing out software upgrades, patches, and changes to configuration in a way that is repeatable, predictable, efficient, and fully audited.

For continuous deploy to be even possible, the source code of the program must be also written in a “good way”. “Good” being a very subjective term, but to put it most objective terms the code must be a) working and b) easy to change. Again, here is a chance to implement culture shift and embrace new way of thinking! Either by tooling, which is automating code analysis to catch common mistakes, or by doing manual code reviews. Code reviews are non-automated reviews of the code where the point is not to criticize one's ability to program. Rather the goal is to lay ground rules on how to write easy to maintain code that follows logical patterns that are set up. When code is being reviewed the findings are shared to the teams, so that the best practices are known by everyone, and pitfalls can be avoided. Learning new things while working, a very motivational situation! This method also provides transparency and encourages for critical code writing standards, so that the code is A-grade material from the very beginning. I think that it is great that this cultural movement encourages this kind of constant learning and constructive criticism, since in modern workplaces underline self-directedness and ability to set one’s own goals. Person who was clear path ahead tend to be more motivated, so there are many kinds of benefits to be found here that helps the team grow, not just make better programs.

Now that the ball has started rolling on the cultural movement of openly reviewing and talking about performance which DevSecOps provides, it is also important to have mindset for this kind of working style. I really liked how on Building a Modern Security Program Rebecca Huehls and Zane Lackey has a whole chapter called “Communicate with Empathy (aka Don’t Be a Jerk)”. The chapter even begins by stating: >Empathy needs to be a core part of your team’s culture.

This is exactly why soft skills are on the rise. A situation where legitimate request to security analysis of one’s programming ability is commented sarcastically only brings trust of whole team down by showing that there is not going to constructive criticism when it is asked. I strongly believe myself that reliance on only the so called “hard skills” are thing of the past and everybody should embrace this new sort professionalism that the soft skills bring to the table. Many think professionalism is cold and calculating, but I believe that professionalism means that you can give out and especially take constructive feedback, both of which also has empathy included. All this is important because DevSecOps as an ideology hinges on soft skills, like empathy, which cannot be reinforced by any tools, but by values. Therefore, a software development with just the DevSecOps-tools is only automation of core processes, not accepting the whole DevSecOps-mentality.