The Cybersecurity Canon: The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win

We modeled the Cybersecurity Canon after the Baseball or Rock & Roll Hall-of-Fame, except for cybersecurity books. We have more than 25 books on the initial candidate list, but we are soliciting help from the cybersecurity community to increase the number to be much more than that. Please write a review and nominate your favorite. 

The Cybersecurity Canon is a real thing for our community. We have designed it so that you can directly participate in the process. Please do so!

Book Review by Canon Committee Member, Rick Howard: The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win (2013) by Gene Kim, Kevin Behr, and George Spafford

Executive Summary

DevOps is perhaps the most important innovation that has happened to the IT sector since the invention of the personal computer back in the early 1980s. It is the idea that organizations would use the same Agile methodology they use today with their software development teams but expand it across all organizations in the deployment cycle: product managers, marketing professionals, developers, quality assurance practitioners, systems engineers, system administrators, operations staff, database administrators, network engineers and security professionals. The specific concept behind The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win, as well as the DevOps philosophy in general, is that the development, quality assurance, deployment, maintenance and end-of-life of IT systems, to include security updates, is very similar to maintaining a production line of any other product. The best practice that has emerged since WWII to manage production lines is the Toyota Production System. DevOps is the IT version of that system. In it, DevOps practitioners try to reduce technical debt by limiting work in progress in order to control the flow of the entire system. I predict that, in 10 years, we will all be immersed in the DevOps philosophy. Because of the way the authors of The Phoenix Project explain DevOps through the novel form, the ideas are much more accessible to non-IT people: CEOs, CFOs and, yes, CSOs. Because of that quality, it is a must-read book for all C-level executives, including security professionals, and you should have read it by now. 

Introduction

One of the constant problems I hear about in my travels is the tension between network managers, security personnel and the information technology staff. In the best cases, even when the teams get along, the passage of workflow from one team to the next is always cumbersome and inefficient. In the worst cases, one or more of the main groups acts as the Minister of “No” within the organization and hinders the other groups’ desire to upgrade their internal systems. To fix these issues, information technology and security professionals have proposed different organizational schemes to equalize the power dynamic:

• The Chief Information Security Officer (CISO) should report to the Chief Information Officer (CIO).
• The CISO should be a peer to the CIO.
• The CIO should report to the Chief Security Officer (CSO).
• The CSO and the CIO should report to the Chief Executive Officer (CEO) or at least the Chief Operating Officer (COO).

I have been known to partake in those debates myself. [1] But those discussions tend to argue for the idea that one group should be in charge of the other or that they should all be peers. In each case, at least one group is never going to be happy. It seems there should be another way.

The relatively new notion of DevOps (development and operations) emerged out of three converging ideas sometime in late 2009: The Agile Development Method [2]; the 2009 Velocity Conference talk; 10+ Deploys per Day, by John Allspaw and Paul Hammond [3]; and Eric Ries' book, Lean Startup [4] which influenced many Silicon Valley companies between 2007 and 2010. [5] DevOps is the idea that there needs to be a much tighter integration between software developers and information technology operations (IT Ops); that once the developers, the quality assurance teams, and the security analysts pass any new code or maintenance updates to IT Ops for deployment, their jobs are not done. Instead of creating artificial black boxes where updates come in, get worked on, and then are passed to the next black box, DevOps is the recognition that update creation, deployment and maintenance is one big system of systems that needs to be managed that way. It is the idea that organizations would use the same Agile methodology they use today with their software development teams but expand it across all organizations in the deployment cycle: product managers, marketing professionals, developers, quality assurance practitioners, systems engineers, system administrators, operations staff, database administrators, network engineers and security professionals. In other words, DevOps uses the Agile philosophy across the entire lifecycle of deployed systems from design, to development, to testing, to deployment, to maintenance, and finally to end-of-life. [7] This idea elevates IT Operations and Security Operations from being mere support organizations to core competencies within the business.

Some big and successful organizations have adopted DevOps: Google, Amazon, Netflix, Facebook, Microsoft, BNY Mellon, The Gap, Nordstrom, and Northrop Grumman just to name a few.  Experts in the field say that adoption is a key part of their achievement. [8]

DevOps is such a new idea though that defining it precisely is not easy and explaining the potential benefits of this change-of-perspective to executive management and other IT professionals is challenging. Enter The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win. [9] You read that right. The authors wrote a novel to explain the intricacies of DevOps. They got the idea from another iconic book written in the early 1980s, called The Goal: A Process of Ongoing Improvement, where the authors used the novel form to explain the Theory of Constraints. [10]

The Story

Parts Unlimited is an automotive parts manufacturer and online retailer that used to be the market leader. In recent times though, the competition has taken the lead in growth and profitability. The company leadership has long promised that a new online program, code-named “Phoenix,” will restore Parts Unlimited to its former glory, but the project is years behind. Investors are pushing for significant leadership changes to right the ship. The company’s board of directors has stripped the chairmanship from the CEO, Steve Masters, and given him six months to show dramatic improvement, or they will take other more strategic and drastic measures, such as splitting up the company. In response, the CEO fired the existing CIO and VP of IT Operations and promoted the hero of the story, Bill Palmer, to interim CIO. That’s the setup.

The story revolves around how Bill, the interim CIO, learns about DevOps from an Obi-Wan-like board member candidate, Erik Reid, and incrementally deploys the philosophy across the company to save it in the nick of time. Through the process, the VP of Application Development, the lead engineer, the Director of Distributed Technology Operations, the CISO, the Director of IT Service Support, the Senior Director of Retail Program Management, the interim CIO, the Chief Financal Officer (CFO) and, ultimately, the CEO, come to realize that DevOps is the most important function of the company. At the end, when Bill succeeds as interim CIO, the CEO decides to groom him to become the company’s COO. In other words, the COO’s main job in Parts Unlimited is to oversee the DevOps function within the organization.

Of course, this book could have been a non-fiction description of DevOps, but that would have been dry and uninteresting to most people. By putting the key elements into novel form, the authors make the technical concepts accessible to a larger audience, not just the technicians of the world but every kind of business leader. The entire C-suite has a part to play in the story as well as all the technical leaders. The authors are trying to teach a wide audience about some fundamental issues that DevOps addresses.

DevOps: Key Concepts

During the story, our Obi-Wan-like board member candidate, Erik Reid, keeps dragging our interim CIO down to the parts manufacturing plant to give him the lesson of the day. Obi-Wan is a parts manufacturing guru and, throughout the story, he keeps trying to explain to the interim CIO that the management of IT Operations should be very similar to streamlining plant manufacturing. Many of the concepts are similar and the problem-solving solutions used in plant manufacturing can and should be applied to IT Operations.

Toyota Kata

The Toyota Car Company has been a profit leader in the auto industry for many years. Several researchers believe that Toyota’s success can be attributed to the Toyota Production System (TPS) that Toyota leaders instituted immediately after World War II. The basic idea is to eliminate waste in every nook and cranny within the company. [11] That is easier said than done, but the TPS way boils down to two katas: one for improvement and one for coaching. The thing about katas is that they are not processes, or information flows, or checklists, or management frameworks. The word “kata” is a martial arts term that describes movement patterns. The martial artist practices them so much that they are second nature; so much so that, when they get into real situations, they do not have to think about what to do. Their muscle memory automatically kicks in. [12]

Researchers and business leaders have studied the TPS for over 40 years. Many books have been published discussing the philosophy, and yet, Toyota continues to innovate and stay ahead of its competitors. [11] One attribution to that consistency is that Toyota thinks about innovation differently from its western competitors. Many in the West think that innovation is a bold new idea that takes the industry in a different direction. Toyota believes that innovation occurs in the score of little improvements made every day by its employees. The company makes millions of them a year because their Improvement Kata and Coaching Kata are second nature to the employees and part of their culture. Not all the improvements work. How could they? But the idea is to try new things, fail fast, and move on to the next. The Phoenix Project borrows heavily from Mike Rother’s book, Toyota Kata, [13] and the idea of continuous improvement is a key concept that our Obi-Wan imparts to our interim CIO. The story begins with Parts Unlimited years behind on the  Project Phoenix: an automation upgrade for their service and support. At the end of the story, Parts Unlimited is rolling out code changes to Project Phoenix ten times a day.

Technical Debt

When you consider that IT Operations is similar to manufacturing plant operations, you start to look at where your employees are spending their time. For instance, you may wonder why the security team can never get all the patches installed. That task seems pretty straightforward. Why is it so hard? One contributing factor is the unintentional deployment of fragile artifacts or systems. Because these systems tend to break down, more and more of the technical staff have to spend time bringing them back online or keeping them online. This effort is called “technical debt.” For every system an organization deploys, the IT Ops staff gets some amount of technical debt that they have to plan for and accommodate. The Toyota Production System attacks technical debt at every opportunity by vigorously trying to reduce it; ruthlessly making their systems less fragile. This activity is the cornerstone of DevOps. The story begins with Parts Unlimited swimming in technical debt. IT Ops can’t get any “real” work done because the team is constantly reacting to emergencies around failing systems.

WIP

WIP stands for Work in Progress. In the Toyota Production System, everything from the start of the car production line to the end when a brand-new Toyota rolls out into the world is WIP. According to Stephen Franklin, WIP “refers to all materials and partly finished products that are at various stages of the process.” “Essentially, it’s an investment that has had zero return and depreciates in value over time.” [13]

Everything within WIP represents potential value that has not yet materialized for the customer. It is the same idea with IT Ops. The more things an organization has within WIP, the more things will happen to impede the workflow. It follows, then, that reducing WIP is essential to any organization. Limiting WIP, therefore, makes IT leaders consider the system as a whole. They need to understand how any new work started will affect the capacity of the system. In other words, if they introduce new work at the beginning of the product line, how might that cause bottlenecks in the capacity of the system to continue delivering constant flow?

This idea is a key piece to the DevOps mantra. Instead of applying the Agile software development methodology to the company’s programmers, DevOps applies similar ideas to the entire system: development, quality assurance, deployment, maintenance and end-of-life. This kind of thinking allows IT leadership to discover constraints in the system; things that, because of resource limits, cause bottlenecks in the system’s flow.

In the story, the key constraint that affected everything wasa  Parts Unlimited’s key engineer, Brent Geller. We all have had a “Brent” on our staff. This is the one person who is the go-to engineer for everything. When we need something new, give it to Brent. When something breaks, give it to Brent. When we need to install the latest security patch, give it to Brent. Even though Parts Unlimited had many engineers on staff, Brent always got things done faster. The IT team grew to rely on him for everything and the company’s leadership would cherry-pick him for their pet projects. The more work the company gave Brent, the more WIP they created because he was that one person and everything relied on him to push the workflow. He was the constraint that hindered workflow.

Kanban boards

A Kanban board is a visualization tool to help teams reduce WIP. [14] Toyota invented the tool back when it started developing the Toyota Production System. [15] In order to seek continuous improvement by limiting WIP, Toyota discovered that the task became easier if they could “see” the data and discover where the bottlenecks were. The generic Kanban has three columns: To Do, Doing and Done. Kanban practitioners place various kinds of work on the chart: user stories, defects, tasks, and features. Today, DevOps organizations expand the Doing column into plan, develop, test, and deploy in order to view the entire system. Reducing WIP means that leadership is controlling the amount of work flowing through the system so that it never exceeds capacity. The Kanban board allows leadership to quickly see what is flowing through the system. [15]

There is a subtle difference between this Kanban board idea and the Agile development method’s Scrum idea. A Scrum has regular fixed length sprints. A Kanban board shows continuous flow. A Scrum releases new code every two weeks while the Kanban board emphasized continuous delivery. In a Scrum, no changes occur during a sprint. In a Kanban board, changes happen at any time.

In the story, before the IT Ops leadership team adopted the DevOps strategy, they tried to manage all the work in a system by tracking deadlines. Years before, they paid for and deployed a very expensive ITIL (Information Technology Infrastructure Library) tracking system and relied on project managers to keep the system updated with detailed analysis of each of their projects. They never had the time to do that so the system failed miserably. By adopting the Kanban board system, the entire team could visualize the work moving through the system and make decisions about restricting or increasing the flow.

The Five Dysfunctions of a Team

The authors borrow heavily from Patrick Lencioni’s book, The Five Dysfunctions of a Team: A Leadership Fable. [16] In that book, Lencioni says one of the major reasons that a team fails is lack of trust between team members. He says there are five indicators of this team problem: unwillingness to be vulnerable within the group, cosmetic discussions versus passionate and constructive debate, no accountability, publicly supporting a decision while privately undermining it, and focusing on individual success versus team success. [16] In the story, when the team is days from utter failure, the CEO gathers the team together and walks his way through these indicators. He tells a very personal story about the status of the company; has a pointed discussion about what is really going on; explains to the group that he is accountable for the success or failure of Project Phoenix and will hold his subordinates responsible for their parts; and, finally, he imparts that the team’s success, not individual contributors, will be the thing that keeps the company solvent.

The Way

Every time the Obi-Wan-like board member drags the interim CIO down to the production plant, he imparts a new concept of The Way. The Way is a metaphor for continuous delivery. As the story opens, the Phoenix Project is years behind schedule. After the interim CEO adopts The Way, he is able to continuously deploy new code and updates as often as is needed, every day, multiple times a day. The Way consists of three ideas [17]:

1. The Flow: Understand how work moves through the system and know that changes will have random effects. Always try to increase the flow but never pass defects downstream.

2. Feedback: Understand needs from both internal and external customers. Shorten feedback loops whenever possible.

3. Continual Learning: Encourage experimentation and learn from failure. Recognize that the kata will build mastery.

Conclusion

I think that the concept of DevOps is perhaps the most important innovation that has happened to the IT sector since the invention of the personal computer back in the early 1980s. By forcing IT leadership and network defenders alike to consider the production system as a whole, to get us out of our stovepipe concerns for our individual projects, and to make collective decisions for the good of the organization that we work for, the community has invented The Way of continuous improvement for all parties involved. IT practitioners are most likely aware of the concept but my guess is that many security practitioners are not. The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win is an introduction to the subject. I predict that, in 10 years, we will all be immersed in the DevOps philosophy. Because of the way that The Phoenix Project authors explain DevOps through the novel form, the ideas are much more accessible to non-IT people: CEOs, CFOs and, yes, CSOs. Because of that quality, it is a must-read book for all security professionals, and you should have read it by now.

Sources

[1] "The Evolution of the Cybersecurity Executive Trifecta: The CSO/CIO/CISO," by Rick Howard, RSA Conference, 21 April 2015, Last Visited 24 September 2016,

[2] "To agility and beyond: The history—and legacy—of agile development," by Peter Varhol, TechBeacon, 26 August 2015, Last Visited 24 September 2016,

[3]"10+ Deploys Per Day: Dev and Ops Cooperation at Flickr," by John Allspaw and Paul Hammond, Velocity 09, 25 July 2009, Last Visited 23 September 2016,

[4]"The Lean Startup: How Today's Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses," by

Eric Ries, Published January 1st 2011 by Crown Business, Last Visited 23 September 2016,

[5] "The Convergence of DevOps," by John Willis, IT Revolution Press: Helping Spark the Cambrian Explosion, Last Visited 11 August 2016,

[6] Deleted

[7] "What is DevOps?" by Ernest Mueller, the agile admin, 2 August 2010,  Revised 16 January 2016, Last Visited 11 August 2016,

[8] "Keynote PuppetCon 2014: The Phoenix Project: Lessons Learned - Gene Kim, IT Revolution Press (Vimeo repost)" by Gene Kim, YouTube, 9 October 2014, Last Visited 10 August 2016,

[9] "The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win," by Gene Kim, Kevin Behr, and George Spafford, Published  by IT Revolution Press, 10 January 2013, Last Visited 28 July 2016,

[10] "The Goal: A Process of Ongoing Improvement," by by Eliyahu M. Goldratt, and Jeff Cox, Published 1982 by North River, Last Visited 23 September 2016,

[11] "THE OPEN SECRET OF SUCCESS: Toyota Production System," by James Surowiecki, The New Yorker, THE FINANCIAL PAGE, 12 MAY 2008, Last Visited 24 September 2016,

[12] "Toyota Kata: the 'how; of 'engaged leadership,' " by Mark Rosenthal, The Lean Thinker: Thoughts and insights from the shop floor, 28 June 2010, Last Visited 24 September 2016,

[13] "Toyota Kata: Managing People for Improvement, Adaptiveness and Superior Results," by Mike Rother, Published by McGraw-Hill Education, 1 September 2009 (first published 2009)

[14] "What is A Kanban Board?" by Leant, Last Visited 11 August 2016,

[15] "A brief introduction to kanban: What software makers can learn from Japanese manufacturing," by Atlassian, Last Visited 11 August 2016,

[16] "The Five Dysfunctions of a Team: A Leadership Fable," by Patrick Lencioni, Published by Jossey-Bass, 11 April 2002 (first published January 1st 2002), Last Visited 10 August 2016

[17] "Limited WIP Meeting presentation - The Phoenix Project book review," by Rudiger Wolf,  LinkedIn SlideShare, 26 November 2013, Last Visited 10 August 2016,

References

"Book review: the Phoenix Project." by skeptic, The IT Skeptic, 22 January 2013, Last Visited 10 August 2016,

"Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation (Martin Fowler Signature Book)," by Jez Humble and David Farley, Published by Addison-Wesley Professional, 27  July 2010, Last Visited 11 August 2016,

"Itil Service Support," by Central Computer, Published by Stationery Office Books (TSO), 1 March 2000, Last Visited 11 August 2016,

"Kanban 101: How to Use Kanban Boards to Manage Your Next Project," BY DANNY SCHREIBER, zapier, Last Visited 11 August 2016,

"Release It!: Design and Deploy Production-Ready Software (Pragmatic Programmers)," by Michael T. Nygard, Published by  Pragmatic Bookshelf, 30 March 2007, Last Visited 11 August 2016

"The Visible Ops Handbook: Implementing Itil in 4 Practical and Auditable Steps," by Kevin Behr, Gene Kim, and George Spafford, Published by Information Technology Process Institute, 2004, Last Visited 11 August 2016

"Visible Ops Security: Achieving Common Security and IT Operations Objectives in 4 Practical Steps," by Gene Kim , Paul Love, and George Spafford, Published by It Process Institute, 17 March 2008, Last Visited 11 August 2016,

"Where To Learn More About Concepts In “The Phoenix Project” (Part 1)," by Gene Kim, IT Revolution Press, Last Visited 10 August 2016

"WIP Limits: How to Journey Safely Into the Unknown (Part 1 of 3)," by  Stephen Franklin, leankit, 18 March 2014, Last Visited 25 September 2016,