Transitioning from Waterfall to Agile

Marianne Hang, Senior Project Manager

April 25, 2018

It was not too long ago that I published an article about the agile method of project management versus waterfall, and provided recommendations when one fits better than the other. In real-life project management, we don’t always have the luxury of making a clear decision between going agile or waterfall at the start of a project. Sometimes we consider the need for a more hybrid route that incorporates both waterfall and agile methodology. Or we may have to move off the waterfall route for the most part and onto agile because we need to leverage parts of agile that will move a project forward successfully. Going the hybrid route of melding the best of both worlds – waterfall and agile – is becoming more common. Transforming a waterfall based project into agile mid-flight is not as common but sometimes necessary for project rescues where the need to get unstuck is paramount. I must also say that this takes courage and character, while taking into account the risks and rewards. In this article, we discuss where and how to make some turns from waterfall to agile methodology, and recommendations to help ensure that even when these turns take you into foreign territory, you and your team will arrive successfully at the desired destination. And I guarantee all the stakeholders will be looking to you to rescue more projects.

Build a Business Case for the Change

Before we discuss the how and when of changing methodologies, we need to discuss the why. If we’re going to take a different route on the map to a final destination, we need to first understand the motivation for a detour. And it’s got to be a better reason than “we wanted to take the scenic route”. Unfortunately, strict waterfall methodology could be considered by some to be the scenic route of project management – where we see and do and experience absolutely everything on the project delivery roadmap. We get to the end eventually but we’ve taken the longest route to get there, which translates into the most expensive and time-consuming deployment. That’s a big price to pay, especially when there’s a better way. Expensive and time consuming are two adjectives that are definitely not going to resonate well with stakeholders. However, this doesn’t mean that we have to drop everything in our project toolkit for the sake of a shortcut. Besides, the shortest route may be the most dangerous. We may risk losing valuable members of our travel group along the way and arrive in a rush because we’ve become scattered and ill-prepared to execute an agile approach. In order to determine how and when we make a change from waterfall to agile methodology, including a hybrid approach, it’s critical to acknowledge the required preparation and consequences of such a drastic change. Just like any other project delivery change, we need to address a methodology change using a formal change management process with corporate best practices in mind. So rather than making a change of this magnitude on the fly – without communication, planning, collaboration, agreement, and approval – we must follow a rigid change management process and put together a formal change request (CR). Keep in mind that a methodology change will impact KPIs in a project, including scope, schedule, and budget. It will also impact resourcing and quality. Even when the impacts are good, we still need to call them out, and ensure that the Project Team and stakeholders are on the same page in terms of expectations. Building a CR also facilitates the driver behind the change. The change driver will be a key part of the go forward plan using the new methodology so it needs to be clear, communicated, and approved. Change drivers can take many forms. For example, in the case of a project rescue, the change driver could be as fundamental as inadequate requirements or project deliverables – i.e. an initiative may be bogged down in the development stage due to requirements that are too high level to build to. By implementing a more agile approach, we may be able to turn out quicker wins through a process that breaks down high-level requirements into more granular or digestible components, and then build to that mini-specification while we engage the business in an iterative manner where they can test and evaluate parts of a solution. This will eventually lead to the solution they need and want. Whatever the driver is, identify and document it clearly because the Project Team will need to refer to it frequently as the project moves out of the old methodology and into the new. As with any change, folks need to be able to adapt so providing a solid business case supported by an approved change request will support that adaptation, as well as the need to move forward and stay focused.

When and How to Make the Transition from Waterfall to Agile

Nothing can demonstrate this transition better than a real life example where I just happened to be the Project Manager for a technically challenging solution initiative. In short, we required an Operational Data Store (ODS) to broker real-time data between a Land Management System, deemed the source-of-truth for land data, and other consuming systems such as Production Accounting that could hardly do anything without real-time land data. And to make things more complicated, our ODS had to be PPDM compliant, where PPDM is an industry data reference model otherwise known as The Public Petroleum Data Model. You guessed it, this was project rescue for a petroleum company in town. Although I was a senior Project Manager capable of handling virtually any situation, I’m not shy to say that I couldn’t have done it without my Solution Architect. Although there was no process in place to make a change from waterfall to agile methodology, we adapted a hybrid approach to get us out of a completely stalled state, where work had ceased, and stakeholders and governance were seriously concerned. Reflecting back, when we stepped into the project, all KPIs were solidly red and had been for some time. It was dangerously delayed, the business was furious, the Project Team wasn’t working together and had lost momentum, and vendors were disengaged. And to top it off, we found out in the early stages of our rescue process that yet more software changes were in flight from one of the vendors and these changes had to be added to project scope due to their regulatory nature. We could go on about the challenges we walked into with this one but the important thing is how we turned this initiative around using a hybrid of waterfall and agile methodologies. Together with the Project Team, we were successful because we made the following decisions at the following times in the initiative. The following best practices that drove these decisions and timing can work for you as well – just remember to modify as needed for your own project circumstances.

Define, document, and obtain sign off on the change request supporting the move to a hybrid project methodology focused on an agile approach.

In our case, we had to build working relationships quickly with the stakeholders, especially the Business Lead.  We left egos behind and focused on building trust.  This took hard work, dedication, focus on excellence, lots of communication, and a sense of humor.  So don’t forget the people in this process – include them, train them, tell them why this is going to work, keep your word, manage expectations, build confidence, and focus less on being the heroes than on being the people who folks can rely on to make a win for everyone.  Remember it’s a project rescue and you’re instituting an enormous change in methodology.  It’s not going to be or supposed to be easy.

Train and engage the business lead in the hybrid approach.

We didn’t assume they knew what this approach would look like or that they were raring to get back on track. Remember, as the key stakeholder, the Business Lead can be the most challenging resource to get back on side and for good reason – they’ve got skin in the game and from what they’ve seen so far on the project, they’ve lost some skin so why should they trust you. But if you can get them on side, they will be one of the project’s greatest supporters and allies, as well as provide the invaluable business background, information, and testing that you’ll need in order to make a more agile approach work and really get the project out of the swamp.

Maintain project management work on a waterfall approach.

This is why we use the term hybrid for this initiative. Maintaining a waterfall focus on the project management role ensured the project could still be governed by the organization’s PMO that was not equipped to govern agile methodology. This built confidence at the governance level and ensured we had all necessary reporting, artifacts, sign offs, and reviews still in place while the more technical work of the project focused on an agile approach. This meant that especially in the early days of the project rescue, we needed to build new documentation and revise old documentation to make fixes, fill gaps, and demonstrate the new process. Flying blind is not an option here – agile doesn’t mean lack of documentation and reporting. Past project deliverables and information must be streamlined moving forward, otherwise the agile work products of your project are at risk because you need to demonstrate that you’ve got things under control. You can’t wait until the last minute to demonstrate this in a project rescue, and certainly not for any initiative as it’s getting to the go live and close out phases. Start immediately and you’ll quickly see how waterfall and agile methodology can not only work together, but support each other. We recommend that the first artifact you provide is the Project Plan showing the waterfall work stream and the agile work activities. This is what your stakeholders want to see and have explained – i.e. how it’s all going to work together and leave that swamp behind.

The Project Manager and Solution Architect must be aligned and work together as a team right from the start.

Again, we checked our egos at the door and aligned on the fact that it was all about making the initiative work. We’d been called upon to help and that’s what we were going to do. We both understood each methodology (waterfall and agile) and we worked to support each other in that understanding so we could manage both work streams within the one project and champion the need for both. From this point of alignment, we managed our respective roles and responsibilities in a collaborative method for the good of the Team, our stakeholders, and the project. The Project Manager was the overall manager providing timely resources and spearheading obstacles across the board, while the Architect managed daily activities with the development team and ensured that progress was made on a daily basis. In fact, the Project Manager worked with all the system owners that were going to interface with the ODS, while the Architect was focused on building the ODS that would serve the application owners. As a tightly coupled Project Manager and Architect front, we worked issues out privately and collaboratively, and we always presented a united front. We knew that by supporting one another, we supported ourselves, our people, and the initiative as a whole.

Identify, focus on, and communicate / market the key reason that drives the switch from waterfall to agile.

In this case (as it often is), it was to prove the technology and give the business a glimpse of what they were getting. After all, this was the first time that an ODS was going to be part of the application architecture landscape. I guess you could say that the ODS came to the rescue, and the purpose of the overall project rescue was to come up with quick wins to prove the technology concept and the vision of what the project was going to deliver for the business. Everyone needs to remember why they’re there and why the project is important enough to be rescued. Trust in the technology – the solution – is just as important as interpersonal trust. The Team was responsible for building, supporting, and marketing that technical vision continually during the project. The only way we could make that vision a reality was to transition to agile. We didn’t have the time, money, people, or support to go any other way. We knew we could do it but we needed to do it quickly and agile was the way to get there. And let’s not minimize the fact that we required technical team members to bring technical expertise to the table to really prove the proof-of-concept ODS. It was also key that the Solution Architect possessed strong people skills, especially when a few individuals weren’t on side or convinced the solution would work.

Demonstrate a proof-of-concept (POC) to support the new technology.

The project scope was to deliver new technology (in the form of an ODS) that the business didn’t really understand, nor did they understand the value they would get from it. It’s pretty hard for the business to support and champion a solution they don’t understand while you’re also changing methodology on them, so we came up with a POC to show them a partial solution as a demo. This not only brought the Project Team into a better space as they were engaged in building the demo and saw the value for themselves, it changed the entire playing field of the project because the business finally saw something concrete, built to their actual requirements, that really worked. We knew we couldn’t build the entire solution fast enough to show a quick win but the POC gave enough of that solution, quickly enough, to get the development ball rolling and the Team back on track.

Hire competent developers early on.

We got lucky and seconded a Developer with a strong database background (specifically Oracle) who was hands on and able to build what we asked him to in record time. I still think of him as our holy grail because he had strong technical skills and experience, as well as the people skills that made him a pleasure to work with (a definite bonus on any project rescue). He understood the agile process and the challenges of a project rescue, and he stepped up to all of this with our support. Agile work requires a skill set that can move accordingly, and not every developer can be successful in an agile environment. Our challenge is to find the ones who can and support them. I think the Solution Architect and Developer became best of friends due largely to their shared vision of the solution and their ability to work so well together in an agile manner. We also communicated the specific skill set requirements that make the agile process work to hiring managers and governance when we had to add to or change project resourcing. It is the job of the Project Manager and IT Architect to make this case to governance in a timely and convincing manner to ensure the right resources are in place when they’re needed.

Remove troublemakers, game players, points of inertia, and resources who refuse to move out of a pure waterfall work process.

Just as we needed to add some resources, we unfortunately had to remove some as well. Hard decisions must be made when support to move through a challenging initiative is refused. In the most luxurious waterfall approach – that scenic route we spoke of earlier – we would have focused on documenting every business scenario, algorithm, interface error-handling routine, etc. But in a hybrid approach where agile work defines the technical deliverables, and especially in the case of a project rescue, we needed to develop the solution in releases and in a Scrum fashion, where Scrum allowed us to talk through processing and solution requirements on a daily basis as we were developing one strong business scenario with all the key features needed to prove the concept. We had to become more pragmatic and solution savvy, and less detail oriented in order to demonstrate that our solution worked. It didn’t have to be perfect, it just had to work. Although the Solution Architect was responsible for the Scrums pertaining to the ODS and making sure that daily work targets were completed, there were times that I stepped up as Scrum Master and held daily Scrums with system owners to proof the interfaces to the ODS. We also implemented daily Project Team Scrums that included the Business Lead as a way of maintaining morale and focus overall. Although we maintained an agile focus, we made sure everyone understood that having some waterfall processes in the background guided all the agile or scrum activities, as well as other project activities supporting PMO governance. Despite the focus on agile and proof that the hybrid process was working, some project members continued to find safety and comfort in a one-size fits all approach that allowed them to take their time, philosophize endlessly, and never really have to come up with a definite solution. These people were either converted to the need and process of adding immediate value demanded by agile or removed from the initiative with governance support. Thus, resources needed to be aligned to agile thinking. In agile, there is no time to manage constant dissension and project politics focused on ego and personal demands. This is the time to come together for a common vision as a true team that puts individual philosophies aside in order to support the overall goal.

Leverage an industry reference model where possible to fast track data/process design.

We of course used the Public Petroleum Data Model (PPDM) to create the database structures that were compliant with oil & gas industry standards. This absolutely saved us the time, effort, and money that would otherwise have been astronomical in such a small window to deliver. Thus, this recommendation fast-tracked our delivery targets, in addition to supporting the company’s overall best practice of using what’s already out there and provided by experts. This strategy enabled us to immediately deliver a database based on an oil & gas industry reference model rather than design a custom data model that would have taken too long and never been in alignment with oil & gas industry standard. This strategy also allowed the company to build strong relationships with vendors – in our case, working with the PPDM experts was invaluable in terms of training, their background knowledge around what worked for an ODS solution and what didn’t, understanding of the industry, and support and motivation for our efforts in building something new that they too saw the value of. Engaging vendors to be a part of the agile process in a project can empower you with their knowledge and help navigate the waters of the iterative build process.

Three Key Points in Transitioning from Waterfall to Agile

There are three key considerations to keep in mind when making the transition from a waterfall to agile methodology, including going with a hybrid approach:

1.  Confirm, document, and sign off the business driver for the change in methodology. Treat this adjustment to methodology as a necessary and warranted change that is supported by existing best practices around change management.

2.  Remember the people. Don’t assume they know how to function within a new methodology or even what that methodology looks like, communicate and communicate some more, build trust, keep your promises, show them the documentation that proves out the new way of doing things, and work together as a united front as you market the new approach.

3.  Support and engage the right resources, while removing the roadblocks. Agile does not lend itself to a team that cannot align and work together, and by its very nature, agile is focused on the group rather than the individual. There is no room for grandstanding in this approach and very little time to tolerate such tactics. This should be the case for every project but agile definitely proves this out. Teamwork and a shared vision will manage the change to agile so that everyone wins.

And once the project is in the bag, we can go on vacation and take the scenic route where there are no timelines and deliverables to worry about.