Andon cord influenced DevOps 20/45 practice

Andon cord or call button influenced 20/45 rule — Problem Swarming

A few years ago, I coached a high-performing DevOps team. This was the company’s first foray into building a more collaborative environment between development (who built embedded software for slot machines) and operations (responsible for ensuring the slot machines worked flawlessly on the casino floor). Additionally, the technology stack was large and new to the company — they had wisely decided to move to a cloud based ecosystem for managing the slot machines and games on those machines remotely without having to send a technician to the casino every time there was a glitch or software needed to be updated. As such, yokotenkai (horizontal/lateral sharing of knowledge, ideas, best practice) was a very important secondary goal.

This group borrowed practices from Lean and adapted them to suit their purpose. One of the practices they made their own was the use of an “andon cord.” An andon cord is a way of signaling a problem (whether real or suspected) and alerting the team that they need help. This team implemented a “20/45” rule — if a team member was stuck for 20 minutes and didn’t have clarity on how to proceed, he was expected to pull the cord, i.e., ask for help. The team (and not just the team leader) would respond immediately and everyone would pitch-in to solve the problem. They had 45 minutes to determine a way forward before going back to whatever else they were working on.

This sounds like a very inefficient practice what with the constant interruptions. However, this turned out to have a number of beneficial side-effects:

• Whole team attention was brought to bear immediately on a pressing issue.
• By definition, in-process work is more important than work that hasn’t yet started; consequently, completing what’s in-process trumps new work. This ensured that team members were always focussed on moving existing work items forward.
• Quality was built-in (team caught problems before they were cemented in place).
• No one person knew the complete technology stack and experimenting and learning were critical to team success. Consequently, someone not pulling the cord for a week was a red flag. Either the person (1) wasn’t stretching and trying something unfamiliar and uncomfortable, (2) had a fear of being vulnerable and asking for help, or (3) had a tendency to work alone and figure out a solution by himself (wasting significant time, not utilizing the team’s existing skills and brainpower, and often ending up with a suboptimal solution).
• Everyone’s level of knowledge increased every time the cord was pulled and team brainstormed approaches. Over a two-month period people’s understanding grew by leaps-and-bounds and team members became comfortable with being uncomfortable.
• The agreed upon way forward was usually better as different perspectives and approaches were considered.
• Everyone on the team knew what was going on and what challenges they were facing.
• Team members felt completely empowered to determine the way forward.
• Team members demonstrated vulnerability by asking for help and loyalty by helping each other complete their work.

It’s now been 2 years since I coached at that company. I recently heard that all Agile teams there are taking this approach — Yokotenkai at it’s finest!

 

What is the problem you are trying to solve?

I had an interesting initial conversation with a couple of IT Directors this morning. One of their questions stood out for me: “Our teams aren’t meeting their commitments. What should we do to fix the teams?”

My first internal reaction was, “You don’t need to do a thing to ‘fix’ the teams.” My issue with questions of this ilk are:

1. Why do managers assume the problem is with the team or teams? Remember, half-a-century ago, Deming pointed out that 95% of the problems are systemic problems. And guess who is responsible for the system?
2. It’s easier to blame others than to take the time to reflect and introspect about how we ourselves are creating the conditions for these problems to come into being — problems we then keep complaining about.
3. Why do managers (and team members as well) not spend a few minutes thinking about what the problem really is — the root cause and not just the manifestation — before leaping to conclusions?

In this case, there are numerous things that could be keeping the teams from meeting their commitments — lack of skills, lack of appropriate training, over specialization, silos, handoffs, interference from without within the sprint, pressure to commit to more than appropriate, changing priorities, lack of alignment with other groups these teams may be dependent on, technical debt, dearth of testers, large batches with little flow — stories being delivered towards the end of the sprint, large stories, unclear stories, unnecessary steps, non-value add activities imposed on the teams, delays and waiting, and so on and so forth. Without understanding what is actually causing the problem, it’s very likely that you’ll end up solving the wrong thing; thereby, creating even bigger problems.

I use a simple diagram to clarify the issue and teach techniques like 5 Whys, Diagram of Effects, ToC Thinking Tools (especially the CRT), etc. to help teams and managers understand problems and fix root causes so the problem goes away once and for all.

Analyzing a Problem

How do you handle such situations? Do you have favorite techniques and tools you use? And how much time do you spend teaching problem definition and problem solving techniques?

 

Are you fixing the right problems?

Just because your team is faced with a number of problems doesn’t mean you should solve all those problems — that might be pretty wasteful, actually. Instead focus on and solve those few problems that are keeping your team from achieving the next milestone, or the desired measurable objectives; ignore the rest.

And when you do attempt to solve a problem, make sure you are tackling the root causes and not the symptoms. Poor inadequate solutions often lead to other bigger problems. Remember Sevareid’s Law, “The chief cause of problems is solutions.”