Coach your Architects in Agile Architecture!

When companies transform towards an agile and DevOps way of working, they sometimes ask how to proceed with architects. Some companies ignore architects in their transformation, some will upskill their architects, and some will make the DevOps teams responsible for the architecture. A core problem we see is that those responsible for the transformation have little experience dealing with architecture in an agile way. The agile coaches are concerned with making the organization agile on a process level. The scrum masters are concerned with the agile process on a team level. And in some transformations, the team also has an agile technical coach who teaches the team new practices and techniques from Continuous Delivery, DevOps, and SRE. In contrast, we observe that architects tend only to get training, if any, and no coaching. This article will lay out why it is crucial to rethink how organizations deal with architects and to start explicitly coaching architects when going towards agile architecture.

How the role of architects is changing in agile and DevOps

The considerable debate in the industry is whether we still require architects when we can make the high-performing software teams responsible for the architecture. That is an ideal situation that is very hard to bring to reality, and irrespective of how you organize yourself, you still need to practice architecture outside the development teams. In this post, we will describe why:

• Software teams have different levels of work, focusing on their purpose within the system. They focus less on and are not incentivized to think about a higher level of work.
• Software teams are parts of your organization's system design; we still need to ensure they fit and align with your business strategy and value streams.
• All those parts should be made responsible for their own architecture decisions; It is the responsibility of the architects to facilitate it and that requires architects to learn new social capabilities, which are hard to upskill by training itself and involves coaching.

So, where does that urge come from wanting to remove architects? Well before, architects did their job in a waterfall organization outside the teams. So the feeling development teams get about architects is that they reside in their ivory tower, isolated from the teams, making decisions for the teams from outside their context and that were disconnected from what was possible in practice. Making decisions for the teams made (some) sense in a waterfall approach because you needed to design all the architecture before moving it to the development teams. And then, the organization goes through an agile transformation, primarily using the Scrum framework. Nothing is said about architecture; it says ‘(we have come to value) working software over comprehensive documentation over processes and tools.’ And that statement usually gave teams the idea that the old way of writing extensive architecture documents is no longer needed. But the problem we see is that the teams also do not know how to do software architecture correctly because they were never tasked and able to do so in the past. And as Douglas Martin said, architecture and design are inevitable; you can either have a good one or a bad one.

Questions about whether design is necessary or affordable are quite beside the point: design is inevitable. The alternative to good design is bad design, not no design at all.            — Douglas Martin

Along came the DevOps transformation focusing on merging Development and Operations practices so that the teams could take full responsibility for what they built. And as a bonus, it also brought in the architecture capabilities, which helped move the teams in the right direction. And it also removed much responsibility from the architects to the teams. Like what we observed in an agile transformation, architects are left out, which does not mean we do not need them.

We still require architects

Software teams are moving towards autonomous high-performing teams, owning the products or services they build for and getting closer to their customers or stakeholders. In addition, by continuously improving their team practices and capabilities, they get to know the business value they are delivering better—an excellent thing. And because of this, we see fewer and fewer “IT architects” in organizations. Something Edo Poll already wrote about in his article 'The value of Agile Architecture in a modern organization'. And organizations also need to take care that these high-performing autonomous teams stay aligned to the business strategy. We observe two major approaches:  

1. Organizations are implementing an agile scaling framework focused on implementing bureaucratic processes to manage the coupling between the teams to keep them aligned, lowering the team's autonomy. 
2. Or the teams get full autonomy and focus entirely on their own goal but do not know their impact on the organization's business value stream. 

And this is precisely why we still need architects, organizing business and technology teams for fast flow and either facilitating and making decisions with these teams or making decisions with them to make them aware of their impact.  Architects work on different architectural levels of scopes in an organization. In the whitepaper' Architecture as Business Competency,' Ruth Malan and Dana Bredemeyer describe the relationship of an architect with a team flawlessly. "It cannot be up to developers and business analysts to decide what parts of the architecture to apply and what parts to ignore. If you treat your architects as consultants rather than decision-makers … all you can expect to get is an assemblage of parts with unpredictable properties." The parts of the architecture on different levels are not abstract from the other levels; they are different parts of the system. Teams can still make decisions on their scope, but not on the scope of another level. So different levels of scope require other skills and a different way of thinking. That concept of architecture scopes also aligns with the work of Elliott Jaques about levels of work. Jaques makes a great analogy in that different levels of work have as different a nature in terms of modes of thinking as, by analogy, water is different to steam. Similar to how application scope is different from domain scope. Each requires other aspects and has other properties.

Different levels of work have as different a nature in terms of modes of thinking as, by analogy, water is different to steam – Elliott Jaques

Architects should still be dealing with all the different levels of scope as they did pre-agile, to quote Gregor Hope, ‘ride the architecture elevator.’ The difference in agile Architecture is that architects now need to take the lead in new opportunities and then continuously shift the responsibilities to teams and managers. We can do this through participatory design. We collaboratively model the solutions with stakeholders, users, and teams. However, this requires a new set of capabilities of an architect. These capabilities are more focused on the social domain than the technical.

Start coaching your architects

Instead of getting rid of your architects during the transformation, train your architects. Teach architects how to make collaborative architecture decision-making and do collaborative modeling workshops. Teach them how to facilitate these workshops between domain experts, stakeholders, and software engineering teams—designing a shared sense of reality with a shared language and understanding. Facilitating these sessions requires architects to invest in people and group skills, like holding space, dealing with resistance, seeing the power of ranking, dealing with cognitive biases, creating buy-in, and managing conflicts. And learning all these new things takes practice and, most of all, coaching. Coaching is vital because everyone will be insecure when dealing with group challenges for the first time. And understandably so, because we are entering a new domain of capabilities. Technical problems are relatively simple to solve, given enough time and knowledge. Dealing with social issues is a lot harder because they are complex. They require experience and reflection to solve. Every situation is different, and most problems architects are dealing with involve a person's inner emotional issues. For instance, if someone doesn’t feel welcome in a group, that is a challenge that someone needs to be aware of during these sessions. Is it just that person’s feeling in particular? they are dealing with? Or are they really not welcome in that session? No wonder architects then fall back on what they already know, making decisions in isolation and then getting stereotyped as ivory tower architects. When this happens, we get consultants in to solve the problem. Their job is to facilitate these sessions and consult the organization into transforming and shifting the responsibilities and changing the team topologies by leading collaborative modeling sessions. And sometimes they help the architect upskilling. But as soon as they leave and do the job, the underlying issues with the current architects are not solved. Of course, part of consultants' job is coaching architects during their engagement; however, it is never the prime focus. That is why we find it essential to make coaching architects more explicit in such a transformation. First, upskill architects via training and then define individual goals and growth paths to close gaps in skills and experience by applying the learned knowledge in practice. Next, let architects start co-facilitating these sessions to help them reflect on what they found hard and what issues they faced. Then coach them to facilitate sessions to face these issues and solve these for themselves.