Andon cord influenced DevOps 20/45 practice

Andon cord or call button influenced 20/45 rule — Problem Swarming

A few years ago, I coached a high-performing DevOps team. This was the company’s first foray into building a more collaborative environment between development (who built embedded software for slot machines) and operations (responsible for ensuring the slot machines worked flawlessly on the casino floor). Additionally, the technology stack was large and new to the company — they had wisely decided to move to a cloud based ecosystem for managing the slot machines and games on those machines remotely without having to send a technician to the casino every time there was a glitch or software needed to be updated. As such, yokotenkai (horizontal/lateral sharing of knowledge, ideas, best practice) was a very important secondary goal.

This group borrowed practices from Lean and adapted them to suit their purpose. One of the practices they made their own was the use of an “andon cord.” An andon cord is a way of signaling a problem (whether real or suspected) and alerting the team that they need help. This team implemented a “20/45” rule — if a team member was stuck for 20 minutes and didn’t have clarity on how to proceed, he was expected to pull the cord, i.e., ask for help. The team (and not just the team leader) would respond immediately and everyone would pitch-in to solve the problem. They had 45 minutes to determine a way forward before going back to whatever else they were working on.

This sounds like a very inefficient practice what with the constant interruptions. However, this turned out to have a number of beneficial side-effects:

  • Whole team attention was brought to bear immediately on a pressing issue.
  • By definition, in-process work is more important than work that hasn’t yet started; consequently, completing what’s in-process trumps new work. This ensured that team members were always focussed on moving existing work items forward.
  • Quality was built-in (team caught problems before they were cemented in place).
  • No one person knew the complete technology stack and experimenting and learning were critical to team success. Consequently, someone not pulling the cord for a week was a red flag. Either the person (1) wasn’t stretching and trying something unfamiliar and uncomfortable, (2) had a fear of being vulnerable and asking for help, or (3) had a tendency to work alone and figure out a solution by himself (wasting significant time, not utilizing the team’s existing skills and brainpower, and often ending up with a suboptimal solution).
  • Everyone’s level of knowledge increased every time the cord was pulled and team brainstormed approaches. Over a two-month period people’s understanding grew by leaps-and-bounds and team members became comfortable with being uncomfortable.
  • The agreed upon way forward was usually better as different perspectives and approaches were considered.
  • Everyone on the team knew what was going on and what challenges they were facing.
  • Team members felt completely empowered to determine the way forward.
  • Team members demonstrated vulnerability by asking for help and loyalty by helping each other complete their work.

It’s now been 2 years since I coached at that company. I recently heard that all Agile teams there are taking this approach — Yokotenkai at it’s finest!

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s