LOTE #9: Gene Kim on Developer Productivity, the "Five Ideals", and Platforms

In the ninth episode of the Ambassador Livin’ on the Edge podcast, Gene Kim, author of the The Phoenix Project and founder of the DevOps Enterprise Summit, discusses the importance of developer productivity within the larger context of DevOps, and explores the “five ideals” as presented in his latest book, The Unicorn Project. He also outlines how the platform that engineers deploy onto should codify operational best practices that promote flow when testing, deploying, and releasing functionality.

The next (virtual) DevOps Enterprise Summit takes place 23rd-25th June.

Be sure to check out the additional episodes of the “ Livin' on the Edge ” podcast.

Key takeaways from the podcast included:

• Conferences, whether in-person or virtual, provide inspiration, knowledge sharing opportunities, and a sense of community: “great adventures demand great collaborators, or maybe it's even the other way around.”
• Cultivating an effective developer experience with safe and fast developer feedback loops are vitally important within the larger context of DevOps and the delivery of value to customers.
• Organisations like Amazon, Netflix, Google, and Microsoft recognise the value of developer experience and often have their most senior developers working on improving developer productivity. The next most senior engineers work on backend services and APIs, and junior engineers often work on features. In most other enterprises, the approach taken is the opposite way around.
• Using the read-eval-print-loop (REPL) within a language like Clojure allows for very fast developer iteration. Appropriately mocking or stubbing external dependencies within the inner developer loop reduces the build-test-iterate friction further.
• As stated by Matthew Skelton and Manuel Pais in their book “Team Topologies”, the platform that engineers deploy onto should codify operational best practices that promote flow when testing, deploying, and releasing: “if you run it on the platform, you inherit the best known understanding of how to solve certain problems safely, securely, abd reliably, without actually having to know it all.”
• "The Unicorn Project" introduces “five ideals”, or values and principles, that frame today's most important IT challenges that are impacting engineering and business: locality and simplicity; focus, flow, and joy; improvement of daily work; psychological safety; and customer focus.
• Maxine, the protagonist of The Unicorn Project, is an amazing engineer but after she is exiled to the Phoenix Project she can get no work done because of the ineffective developer experience: “she can't write tests, she can't write run tests, she can't even get the help she needs. The mythical 10x developer is now Tom Hanks in the movie Castaway where they can get nothing done.”
• Locality and simplicity can often be simplified down to the “lunch factor”: when you want to get something done, how many people do you need to take out the lunch? Partitioning of domains, creating loosely coupled architectures, and enabling build and test isolation allow small teams to independently build, test, and deliver value to customers.
• Focus, flow, and joy is all about the psychology of optimal experience, being in “the zone”, and being able to get stuff done. Professor Mihaly Csikszentmihalyi's work in this space is highly recommend for listeners wanting to know more.
• Continuous improvement is driven by culture. “Learning to Lead” by Dr Steven Spear is a highly influential paper in this space. "Transforming Nokia" by Risto Siilasmaa is a candid book that talks about group dynamics that is relevant to any leader.
• Large enterprise organizations have the potential to win markets when they focus appropriately on engineering productivity: “it's not small beats big. It's fast beats slow, and so big and fast can decimate the competition”. The book “Project to Product” Dr Mik Kersten outlines how enterprises can thrive in the age of digital disruption.
• Psychological safety and customer focus are vitally important when it comes to collaborating to sustainably deliver value to a business.

Subscribe: Apple Podcasts | Spotify | Google Podcasts | Overcast.fm | SoundCloud

This week's guest

Gene Kim is a Wall Street Journal bestselling author, researcher, and multiple award-winning CTO. He has been studying high-performing technology organizations since 1999 and was the founder and CTO of Tripwire for 13 years. He is the author of six books, The Unicorn Project (2019), and co-author of the Shingo Publication Award winning Accelerate (2018), The DevOps Handbook (2016), and The Phoenix Project (2013). Since 2014, he has been the founder and organizer of DevOps Enterprise Summit, studying the technology transformations of large, complex organizations.

In 2007, ComputerWorld added Gene to the “40 Innovative IT People to Watch Under the Age of 40” list, and he was named a Computer Science Outstanding Alumnus by Purdue University for achievement and leadership in the profession.

Full transcript

Daniel (00:01):

Hello everyone. I'm Daniel Bryant. And I'd like to welcome you to the Ambassador Livin' on the Edge podcast. The show that focuses on all things related to cloud-native platforms, creating effective developer workflows, and building modern API's. Today, I'm joined by Gene Kim, author, researcher and all around DevOps expert.

Daniel (00:18):

Now, I'm going to assume that Gene really needs no introduction to the listeners of this podcast. But just in case you haven't bumped into his work, he is a multiple award-winning CTO and researcher and author of six books, which include personal favorites of mine, The Unicorn Project and The Phoenix Project. Gene is also the founder of IT Revolution and hosts the very successful DevOps Enterprise Summit, which I've attended both in London and San Francisco. In fact, the next summit will be held as a virtual event, 23rd-25th of June. I thoroughly encourage you to attend.

Daniel (00:45):

If you like what you hear today on the podcast, I would definitely encourage you to pull over to our website. That's www.getambassador.io where we have a range of articles, white papers, and videos that provide more information for engineers working in the Kubernetes in cloud space. You can also find links there to our latest releases, such as the Ambassador Edge Stack, and also our open source Ambassador API Gateway, and our CNCF hosted Telepresence tool too. Hello, Gene, and welcome to the podcast.

Gene (01:09):

Daniel, I'm so happy to be here and we have so many friends in common and I never quite connected the dots that you've been at Datawire for three years. This is so exciting.

Daniel (01:17):

Super, super. So just to set a bit of context for our listeners, Gene, could you share a recent career highlight? Now I know you're involved in books, conferences, advising. I'm guessing you've got a lot of career highlights to choose from, but just something to give a bit of context for where you're coming from.

Gene (01:31):

Yeah, for sure. I have been studying high performing technology organizations for 20 years and I wrote the book called The Phoenix Project with the coauthors and so forth. But yeah, recent career highlight was just last week when I wrote a blog post called "Love Letter to Conferences." And as part of that was really trying to crystallize my own thinking in terms of what made the most memorable and meaningful conference experiences great. And which parts are universal, whether physical or virtual, and which ones could actually be modified in a virtual format to make them better. And as part of that, I actually gathered 10 years of conference photos and there was over 700 photos, and posted a big JPEG image, it's 11,000 pixels tall. And I got a little tearful just looking at some of those photos. Just about how great it was to meet fellow travelers and kindred spirits on a common journey. And that just reminds me of how much, now that we're going to two months of sheltering in place, just how much we miss in-person, physical conferences. That was a kind of a highlight that looks back over 10 years.

Daniel (02:31):

Yeah. That blog post was fantastic. I'll be sure to link it in the show notes for listeners. I think if I look back in my career, going to conferences has been really influential in the direction I've taken. I'm guessing it's the same for you?

Gene (02:41):

Yeah. In fact, I'm counting two learnings out of that, that were profound for me, was a conference is so much more than just a bunch of talks. And I think it crystallized a notion that great adventures demand great collaborators. Or maybe it's even the other way around? Great collaborates lead the great adventures. So it's all about the company we keep.

Daniel (02:59):

Oh, I completely agree. So, traditional first question on podcast, Gene, is about developer experience. The coding, testing, delivery, observation, that kind of thing. I always ask, protect the guilty and the innocent here, but can you describe what you've seen as your worst developer experience?

Gene (03:15):

Oh my gosh. Where do I start? Three come to mind. Can I do more than one...

Daniel (03:17):

Yeah, rock on.

Gene (03:19):

For 20 plus years, I've self-identified as an Ops-person. And this is just by getting my graduate degree in Computer Design and high-speed networking. It was my observation that was Ops where the saves were made. It was Ops who saved us from terrible developers, Ops that protected data, because it wasn't the security people, for sure. And yet, for now four years, I really self-identify, not as an Ops person, but as a developer. And it's because I learned Clojure, this amazing functional program language that runs on the JVM or in JavaScript. And so, what has been amazing is I've become very fussy as a result.

Gene (03:53):

There're things I used to do 10 years ago that I used to get enjoyment out of that I now despise. So, three things come to mind. One was, and this is probably about six years ago, we were on vacation. I won a budget, 30 minutes to make a two line change to a Ruby on Rails app. And it took me, I think seven hours. Because they couldn't push into Heroku because of some Gem file local, something, something. 30 minutes turned to seven hours and that's happened to... I think that's the common thread, where I just want to do one thing, and you get sort of tangled up into something in the environment or dependencies. And it's so far removed from what I actually want to do.

Gene (04:32):

And I think that the second story that comes to mind, one time I wrote an app to do Trello card management. And it was just so glorious writing it in Clojure. I could just get leverage all of the Java libraries. And then you use OAuth, but I built so much functionality in two weeks. I felt super heroic, just the amount of stuff I could get done. And then it was time to run it in the cloud, right? And then I was trying to push to Heroku and just trying to get OAuth running, but I couldn't even get there. It took me a week. Just mired in YAML files and I can't even remember. "How to get keys in the right place," and just all this stuff that... Oh, gosh.

Daniel (05:15):

It's callbacks, isn't it? Things that are getting the callbacks all wrong, that's...

Gene (05:17):

Oh, my gosh. Yeah, right. And each time it would take seven minutes to test out. It's just the horrible, horrible feeling. OAuth was the fun part. No, in fact, Daniel, getting OAuth running was the fun part. The six days prior to that...

Daniel (05:29):

Not so much, right?

Gene (05:32):

Not so much fun. And OAuth is never fun, just to show you how terrible it was. Let's say the third one is.... You know that first story about Ruby on Rails, that's where I got Docker, really. I just never wanted to do that ever again in my life. But then, it became my first experience running in Kubernetes. I mean, that was...

Daniel (05:48):

Ooh, interesting.

Gene (05:49):

Yeah, Kubernetes was the toughest thing I've ever learned. I watched John Willis do stuff, and it was actually our mutual buddy Flynn at Datawire.

Daniel (05:55):

Oh, yeah. Yeah.

Gene (05:57):

He would tell me how to get a shell running inside of a Kubernetes cluster. And I would sit down and try to do it myself, and I wouldn't even know what to Google for. That's just like, "Keep control, then what?"

Daniel (06:10):

Yeah, yeah.

Gene (06:12):

And that makes Heroku look fun. I would add that to my list of things I just never want to do again. I never wake up in the morning aspiring to write Kubernetes deployment files or trying to connect things with side cars and so forth. It's an engineering miracle. It is great, but holy cow. It's a lot to take in. And on a scale of one to 10 for fun, it's a negative 12. Do any of these resonate with you or...? I don't think I'm a...

Daniel (06:48):

They do, totally. Oh, I was actually curious, one of my notes here, Gene is... Because I'm a Java programmer by trade. That's my native language. I've done a whole bunch of stuff on the JVM, but I've not done much Clojure. And I was thinking with Clojure that using, say a REPL, the Read-eval-print loop, that must be super good for getting a fast feedback loop going.

Gene (07:03):

Oh, my God. Yeah. In fact, I did something that the other day that I have actually not done in 10 years. You're talking about the OAuth callback problem, right? It's like, "Oh, I got the URL point in the wrong place. How do I change it? Do I need to change it in Google Cloud or whatever?" Yeah, it was seven minutes cycles to build tests on Heroku and then seven more minutes that one, three minutes to build test, and then three minutes to actually push it into Kubernetes. How can you... If you don't know what you're doing and all you have is trial and error, seven minutes per iteration is soul killing.

Gene (07:36):

I spun up a raw VM, in my case, it was a Google Cloud, and I was actually running the web server with a REPL in it, just on the raw VM. And it was just amazing because then with a REPL you could change it within seconds.

Daniel (07:52):

Yeah, yeah.

Gene (07:52):

And you wouldn't have to restart the... Yeah, we wouldn't even have to deal with the 11 second startup time. So, that was a magical experience. Yeah, Clojure is amazing because it really is a dynamic language with all the things that you expect in a REPL where you can just pick one command, one key press, and basically reload your entire state or everything. Or relay your code and the state remains the same. It's just awesome.

Daniel (08:18):

Every developer listening would recognize, where do you get that fast feedback loop? We'll talk about more about this later on, Gene, back to the five ideals of The Unicorn Project. But that flow, that speed. Once you get into the zone, it is a magical moment as a developer, right?

Gene (08:31):

No, absolutely. That's... In the JavaScript world, when you can actually get the dev tooling to work and get the hot code reloading, it is magical. That was a stunning moment. And with the JVM you can get the same thing. And with the REPL you are literally talking about millisecond feedback times per iteration. And it's just a magical experience. And I think the people who came from the Smalltalk background, they've been saying, "We've known this for decades." Your VP of Engineering told me the same thing. I believe once you experienced that, it is really tough to go back and even deal with compile times and rerunning executable. It's just a...I have no interest in that.

Daniel (09:14):

Yeah, totally makes sense. Yeah. But, before we dive into some of The Unicorn Project content, how important do you think the developer experience is in the grander scheme of things such as DevOps and delivering value?

Gene (09:25):

Oh yeah.

Daniel (09:25):

Because I don't see many people talking about the first mile, if you like, of developer experience.

Gene (09:30):

Right. Catching some of the Microsoft Build conference, there was a... I tweeted out two things that just struck me. The first one was a core comp, "These shouldn't have to be being able to run two Python versions on your laptop."

Daniel (09:43):

Yes. That'd be great, yeah.

Gene (09:45):

It's so hard right? And then the second one was like, "It's hard to call yourself a world-class engineering shop if it takes three days to onboard developer." I think dev productivity is so important. In fact, so much of The Unicorn Project was really to try to capture those sensibilities of one, how much joy and flow and focus you can get when you have the right dev productivity tools. The second thing is, that I tried to convey in The Unicorn Project, was this weird inversion of values, right? And if you take a look at the Facebook, Amazon, Netflix, Google's, Microsoft's, they put their best people on dev productivity.

Gene (10:18):

Google famously puts 1500 developers, of the most senior developers often with PhDs, on dev productivity. That's a billion dollars to spend a year. And then they put the next best developers on the back-end APIs. And then the most junior developers on features. Whereas, in most enterprise shops it's the opposite. They put their best developers in the features or the app, because you can see them. The next best developers on the backend and then their summer interns on dev productivity and build tools.

Daniel (10:45):

Yeah, sadly true.

Gene (10:49):

Yeah. When I was at Tripwire, I was there for 13 years as a CTO and founder. I did the same thing. In 2006, our Cruise Control server broke that ran our CI systems. And we didn't replace that... We left that position unfilled for 18 months because we didn't put value in it. And our code integration times went from a week at the end of the project to a six weeks.

Daniel (11:13):

Wow.

Gene (11:13):

And so we couldn't ship releases as frequently as we wanted to. But that link between cause and effect wasn't obvious at the time. So yeah, I think dev productivity is so important. I think it's what makes the difference. It really is what differentiates compounding technical debt versus using compounding interest in your favor where you're just making developers more productive every day.

Daniel (11:37):

Yeah. Well said, Gene. I think that's a perfect segue actually, into talking more about your latest book, The Unicorn Project. Our Datawire team and I have very much enjoyed reading this. In fact, it's now one of our onboarding essential reads for all new hires, alongside Lencioni's Five Dysfunctions of a Team and Basecamp's Shape Up too. I'm guessing some listeners haven't had a chance to read The Unicorn Project, yet. Could you provide a quick executive summary for us, please?

Gene (11:57):

Yeah. Yeah. So, it's set in the same timeline as the Phoenix Project. But instead of being told from the Ops leadership perspective, it's really told through the eyes of Maxine, an amazing developer who's exiled because the payroll outage is blamed on her unjustly and is exiled to the Phoenix project. And it was really so much of the personal horror stories of my worst developer experience is like Maxine's life.

Daniel (12:20):

Yes.

Gene (12:21):

It's amazing to me that you can take the best developer and you put them in a situation where there's no dev tooling. She can't do builds, she can't write tests, she can't run tests, she can't deploy, and she can't even get the help she needs. That mythical 10x developer is now like Robinson Crusoe or Tom Hanks in the movie Castaway where you can get nothing done.

Gene (12:46):

And so the organization is hiring... They have scores of dev contractors who are just idling away because they can't do the most basic thing that developers should be able to do on their first day. So, the mechanism to frame that way out is really the five ideals, which is really framed from my experiences. Showing the best and worst of being a developer. And the first is locality and simplicity, the second is focusing on joy, the third is improvement of daily work, the fourth is psychological safety, and the fifth is customer focus. And it's just really meant to evoke the emotions of how good, good is when all of those tools are there to help support developers and the culture and architecture and how bad, bad is.

Daniel (13:32):

I think any developer reading the book team will totally empathize with it. As in I've worked in startups, I've worked in larger organizations, I worked for the UK government for a while, and I can recognize dysfunctions in all of those organizations. But generally the bigger the organization, the more I could identify with that book.

Gene (13:49):

Yeah. And that's a shame, right? Because one of the lessons for me is, it's not small beats big, it's fast beats slow. And so big and fast can decimate the competition. It's the greatness that large complex organizations who have the market, the customers, the channel relationships, those are the ones that have absolutely the potential to a win in the marketplace. And so hopefully the The Unicorn Project can highlight what's missing.

Daniel (14:20):

Yeah. I think one of the key things for me in The Phoenix Project and many other things, is that awareness. I've done consulting over the years and folks often didn't know what good looked like. And The Phoenix Project gave them an insight. Back when I was consulting, I used to hand out The Phoenix Project and they were like, "Wow, is this really possible?" And I'm like, "No, it's seriously is, but you have to invest in it." But I think folks just don't know what good looks like sometimes.

Gene (14:44):

Yeah, for sure. And I think that's what the narrative fiction format or the business fable format is so effective at. Because you can tell the story in very broad brush strokes on a very large canvas in The Phoenix Project and The Unicorn Project. It follows a similar format of the hero's journey, where you spend a third of the book really painting the desolation of bad bad is. And to explore every nook and cranny of it to really evokes that feeling of, "Holy cow, this is me." Regardless of whether you're a developer, or a QA or Ops, or information security, or the product owner, or the business leader or whatever.

Gene (15:23):

I think that's kind of a luxury you have that you can't really do in a nonfiction format. Or you can, but you can't fire the mirror neurons to the extent that you can with it with a story. The Five Dysfunctions of a Team, when I read that book, I remember exactly when it was, I was on a plane. I remember having to close the book a couple of times, and I could even smell that nervousness you feel.

Daniel (15:46):

Yeah.

Gene (15:48):

I want to throw the book across the airplane cabin. It was just so traumatizing to read.

Daniel (15:55):

Yeah. The best book, same like best TV series I find, is the ones that do drive emotions. Even as techies, we still are emotional creatures. Right?

Gene (16:03):

Yeah. In fact even for writing The Unicorn Project, I was actually reading a lot of screenwriting books just to have... Because here's a craft of really telling the stories and there's a trope that I used. It was called "All hope is lost." A man, the hero's journey where you go through the depth of despair, and then the climb to ascendancy, the "All hope is lost" is what happens at the very end where you think you've won. But then the second death star shows up. It's like, "Oh no." And then all your buddies die.

Daniel (16:36):

Oh, no.

Gene (16:36):

So, that does that. That's a vocation and a craft that has spent centuries honing the tools to evoke emotions.

Daniel (16:41):

Yeah, for sure. Now let me... The book, The Unicorn Project is fantastic. I wouldn't mind to dive into a couple of the ideals, Gene.

Gene (16:47):

Sure.

Daniel (16:48):

But that were really, really useful. And then the first one that jumped out to me, and you and I were talking a little bit off mic about this, the first ideal being locality and simplicity. I think it's been a lot of good efforts in the community over the last few years. We have microservices decomposing your code base. We have Kubernetes specializing in femoral hardware, but actually I find sometimes it feels like it's got harder and there's more dependent things. Not less.

Gene (17:13):

Yeah. Yeah. For sure. So, that's what I was really trying to speak to in the first two ideals. The first one is locality and simplicity. And this is really what I learned from Rich Hickey, who is the creator of the Clojure programming language. And so the locality and simplicity, I'd love to simplify it down to the lunch factor. When you want to get something done, how many people do you need to take out the lunch?

Daniel (17:34):

Oh, interesting.

Gene (17:36):

Is it the Amazonian ideal of the two pizza team? Where they can independently do what the customer needs without any external dependencies. Because of the partitioning between domains, right. And architectures that allow small teams to independently build test and deploy value to customers. So that's a lunch factor of two pizzas versus in most architectures it's like 40 teams, right?

Daniel (17:58):

Yeah.

Gene (17:59):

You got to buy pizzas for the entire building to deploy. And so that is really says to what degree can we build and test and deploy our components in isolation? Can we write the features we need without having to use integrated test environments and so forth? So for sure, and my own personal experience with that is even these kind of events, sourcing architectures, using Pub/Sub, dramatically increases the simplicity and the composability of components that I think is really kind of at the core of a domain driven design. Where you can actually change small pizza systems independently from everyone else. I've-

Daniel (18:40):

That decoupling.

Gene (18:41):

Exactly. And I think those are the conditions when you can have small lunch factors, build, test things in isolation. That's how you can actually, as a developer, have that feeling of focus, flow and just joy in your work.

Gene (18:56):

And so much of that is inspired by Dr. Csikszentmihalyi who wrote the book Flow: The Psychology of Optimal Experience. And just the best description is, imagine yourself in a time when you're having so much fulfillment and joy out of your work that you lose sense of time and maybe even sense of self. That transcendental experience.

Gene (19:14):

So you can't do that if you have a horrible architecture that you can't do anything without calling 40 different meetings and getting all of them to say yes. So, yes, all of that... Architecture and microservices are a part of that, but Holy cow. If it means that you now have, in the case of Netflix, 1800 different services.

Daniel (19:34):

A lot.

Gene (19:35):

It is a lot. And that without the right greatness and architecture and tooling around all that, that's a pretty horrible experience if everyone has to know Kubernetes.

Daniel (19:43):

Agreed. Yes. Yes. Totally makes sense. What's your thoughts, Gene... Over the years we've had DevOps, we've had a lot of other things popping up, like site reliability engineering, coined by the Google folks. A couple of fantastic books there. We're interested in GitOps from the Weaveworks team and we've been following them for many years. Loving what they're doing around declarative config and this control loop and so forth. What's your thoughts on all these new practices that have popped up like SRE and all these new implementations of get ops. How do they marry into DevOps, do you think?

Gene (20:16):

Yeah, I can't speak for DevOps in general, but I think they're all very exciting. And to me what they have in common... Well, one of the things they have in common is this notion of how do you take functional expertise, like a SRE and infrastructure and operations and security and QA, and get it out of people's heads and put them into a platform, where anyone using the platform can leverage all the greatness? If you just run it on the platform you inherit the best known understanding of how to solve certain problems safely, securely, reliably without actually having to know it all.

Daniel (20:48):

Yeah. Yeah.

Gene (20:49):

Without having to write a Kubernetes deployment file. By the way, it sounds like I'm trashing Kubernetes. It's amazing. It's an engineering miracle. I use it for my own production code, but wow.

Daniel (20:58):

Yeah.

Gene (20:58):

Sometimes it's a reminder of how bad it can be. I have screenshots of my search history on Google trying to find, how do I make this?

Daniel (21:12):

Kubectl this, kubectl that, right?

Gene (21:12):

Yeah, exactly. How to make an error message go away and be with my YAML configuration file malformed. And the book Team Topologies by Matthew Skeleton and Manuel Pais, just does such a great job in painting how we really need infrastructure and operations, security, how they should be structured so that they can elevate the state of the practice for the entire company. With knowledge, not in their heads, but in the tools they build and enable for others.

Daniel (21:48):

Yeah. It makes a lot of sense. Matthew and Manuel, I'm lucky enough to call buddies of mine. We know each other from the London tech scene and the conference scene. And when that book was published, I got the preview as a sneak peek. It's just fantastic. And I read that they pitched sort of stream aligned teams, I think enabling teams, a complicated subsystem teams, and this platform team. And I really like your pitch there that the platform team is kind of the code advocation of all the good practices for running your apps. Is that right?

Gene (22:13):

For sure. And I think it was also important to write is that the mode of interaction is not tickets. It is about self service. And one of the things that I've done since the beginning of the year is been having weekly calls with a mentor of mine, Dr. Steven Spear, who wrote the most widely downloaded Harvard business review of all time called Decoding the DNA of the Toyota Production System. Which was based on his PhD doctoral dissertation at the Harvard Business School. Which is based on his six months working on the Toyota assembly line for tier one Toyota supplier. And it's just amazing. His thinking has influenced my work for going on seven years now. And one of the things that I'm really trying to understand is this notion that he has is that there's really, when you talk about organizational dynamics, there's this very parsimonious constructs, he uses you have basically structure and dynamics.

Gene (23:07):

So structure is really kind of how you organize teams. It is the architecture you work within. And then there's dynamics, which is everything else. It's a culture. It's signals. Is it fast feedback or no feedback? Is it a culture of fear where everyone is afraid to tell bad news? Or is it amplified? Like safety culture at Alcoa. And it says, as leaders, so much of the dominant knobs of which we really control are really about structure. And I think Team Topologies really does is explain what are the teams? What are their interfaces? What are the domains of responsibility that allow for these amazing ways of working that lead to better performance, better employee engagement, more productivity, happiness and so forth. So I'm actually restudying the work they did because I think they did such a splendid job in really defining kind of the structure of the organization and how it does result in vastly different dynamics than the old world where to deploy, I need to talk to 40 different teams who may not even know who I am.

Daniel (24:13):

Definitely one thing as my career has progressed, it's almost become a bit fractal in some of the things I see. But I remember talking about company and cohesion in terms of software architecture. I can see that now in organizations too. And something you just said there, Gene, I was just thinking about. I've chatted to Manuel and Matthew quite a bit about APIs teams. As engineers... I've worked in Java, spent lots of time around API design, interface design. As a manager, I didn't really think about that so much with my team.

Gene (24:41):

Yeah.

Daniel (24:41):

How my team is exposed to the rest of the organization. So I'm guessing there is knowledge that is transparentable from an engineer to organizations.

Gene (24:50):

Very much so. In fact, this is kind of my biggest aha moment. So here's... It blows my mind. So Dr. Steven Spear, in the late nineties, he's working on his doctoral dissertation at the Harvard Business School about the Toyota production system. And it turns out that one of the most formative influences on him is Dr. Carlos Baldwin, who wrote a lot of similar works on code architecture, modularity.

Daniel (25:14):

Oh, interesting.

Gene (25:14):

Who was also one of the biggest influences on someone I'm a huge fan of, Dr. Mik Kersten, who wrote the Project to Product book.

Daniel (25:20):

Yes.

Gene (25:23):

So that's a genuine, "What the F" moment, right? It's two of the people who influenced me the most, Dr. Steven Spear, Dr. Mik Kersten were both influenced by Dr. Carlos Baldwin from two totally separate domains.

Daniel (25:38):

Yeah, interesting. Yeah.

Gene (25:38):

And so, he was talking about how teams work together in Toyota. And he was using the language of interfaces and the sequence and timing, which is actually very, very close to how we talk about APIs. In terms of their agreed upon protocols, I give you this, you give me that. So I haven't all unpacked and processed it. In fact, it's something I'm doing in the next couple of podcasts that will be released that I'm doing called The Ideal Cast, interviewing Mike Nygard on his views on architecture.

Daniel (26:07):

Oh, lovely.

Gene (26:08):

Steve Spear and Elisabeth Hendrickson. It is that same insight that you had, is something that I've been thinking a lot about, which is organizational design actually has a lot in common with API design. And the Team Topologies book really put their finger on it.

Daniel (26:26):

Agreed.

Gene (26:26):

It's exciting. It's exciting that architecture is so much more than designing APIs. It is really the architecture of which the entire organization to work within to achieve goals. And does architecture drive us towards a lunch factor of 300? Or does it drive us towards the Amazonian ideal of two pizza teams that can all be advancing towards a business goal?

Daniel (26:48):

Yeah. Something I just want to pick up there, Gene, that you said. I've definitely seen it from your work and from John Willis's work as well, is this cross pollination of ideas. So actually my post grad, I did post grad and my professor, was we're in Computer Science, but he was a professor of Biology. He encouraged me to read around biology and I learned a lot of biological principles I could apply to computing. And I think I see that in your work. And I remember listening to a podcast by yourself and John Willis. This is looking outside of computing to bring interesting ideas to the field.

Gene (27:17):

Yeah. I love that. And that's the fear... You mentioned biology, that's a paragon of modularity.

Daniel (27:23):

There you go.

Gene (27:24):

And yeah, totally. And I think to your point, I love this phrase that, "There are no new problems under the sun. It's just a old solution applied to new domains." And it's typically the areas between domains that actually lead to the biggest breakthroughs. And so that would tend to explain where we're seeing the greatest excitement and breakthroughs happening in technology. It's infrastructure, it's code, it's get ops being taken all of these principles from software engineering and using them for infrastructure and more.

Daniel (27:54):

That's it.

Gene (27:55):

Which affects developer productivity. I think I would agree with you. It is very exciting to be able to have fellow travelers and friends and experts in different domains. And to be able to learn from it. And for me, it's incredibly satisfying and intellectually stimulating and exciting to feel there's something important there that we can bring to our respective professions.

Daniel (28:19):

Yeah. Well said, Gene, well said. I want to just pick up on the third ideal for a moment. You mentioned about the third ideal being the improvement of daily work. And what I've seen from my career, like the FAANG companies you mentioned, they're really good at this. Paying down the debt, invest in developer experience. I work with some more traditional organizations, big market caps doing very cool stuff. But I had a hard time convincing the senior leadership to invest in this thing. And I think it was because they weren't really tech savvy or not to the level you would probably see in the leadership of Facebook, Google, take your pick. What's your experience been around that? Have you got any advice for folks if you're in, say my position, you're trying to convince perhaps your leadership, you have to invest more in this?

Gene (29:01):

Yeah. I think that's the toughest. I think my most recent thinking on this is that it's because they're using a bad way of thinking. And the great book Project to Product shows that this is what you get when IT technology is part of a cost center and they're using project management as a primary way to make decisions. And so technical debt is very difficult to talk about when you don't have a project code to assign it against.

Daniel (29:26):

Yeah. There you go.

Gene (29:27):

And you have these large planning cycles where it's always too late and it's very difficult to make longterm rational decisions when the project management approval process budgeting just sort of drives you to a certain way of thinking and doing. And so, the part of the goal of The Unicorn Project was to highlight how good, good is. Not just from a dev productivity tooling, but just as impact on developers doing daily work.

Gene (29:51):

And for me, the dazzling book that I stumbled into, which I'd rank as one of the two best books I've read in the last decade, is a Transforming Nokia by Risto Siilasmaa.

Daniel (30:02):

Ooh, I've not heard about that one

Gene (30:04):

Lovely, amazing book. And I'll be honest, my first reaction is, "What are we going to learn from the story of how this person contributed to the tanking of Nokia?" But it is a stunning book.

Daniel (30:17):

Oh, interesting.

Gene (30:18):

He joined the board of Nokia in 2007, as he has this unflinching analysis of his own shortcomings as he couldn't overcome the domineering characteristics of the board chair. But the shocker to me... It was just this marvelous, candid book about group dynamics that is relevant to any leader. But then in terms of technical, he describes how in 2009, he learned from the VP of Strategy that the build times for the Symbian OS was taking them two days.

Daniel (30:52):

Oh, wow. Yeah.

Gene (30:54):

And he said... He was the founder of F-Secure. He's a technical person. He said, "It felt like being hit in the head with a sledgehammer, because if it takes two days for any developer to know whether the code worked or would have to be redone, this thing, that all their dreams, hopes, and aspirations are hinging upon, it's an illusion. It doesn't exist."

Gene (31:11):

And that led them to go to Windows mobile, which didn't treat them so well is that they dragged them into the gray, but that was actually a better bet than staying on Symbian OS. So I found it dazzling because he saw that at the board level, about dev productivity. And so that we'll be in a just world, right? That will be candid conversations. That every large market cap organization talking about digital disruption should be talking about.

Daniel (31:36):

Yes.

Gene (31:36):

So yeah, I think that book does so much to talk about how important it is for any organization that's doing anything in technology. And I hope some... That book is referenced and I just would recommend that book to anybody. And by the way, then he tells the inside story of selling the company to Microsoft for $7 billion, even though they were probably 60 days away from declaring bankruptcy.

Daniel (32:01):

Wow.

Gene (32:03):

A story on culture. As, he then led Nokia into the switching space and is actually one of the few companies that are gaining market share against Huawei. So it's just a neat, neat book.

Daniel (32:16):

Yeah. Very nice. I don't want to ignore the fourth and fifth ideals. It was interesting. So the psychological safety, I know that's been super important in my career and customer focus. We hear a lot of folks talk about that. How come you put psychological safety and customer focus as four and five, rather than perhaps one or two? Which some... Amazon, for example, I would think would put customer focus as number one.

Gene (32:37):

Yeah. Yeah. I love the notion of the concentric circles of areas of control, areas of influence, and area of concern. And that was the way I thought about it and what outcomes results from what.... and that's what's led the order. But I think there's also...

Daniel (32:52):

Okay.

Gene (32:52):

Another person asked, "Why is customer focused last? Shouldn't that really come first?" As you mentioned. And the conclusion that we came up with that we both thought was delightful, was that it's really from my mind. Until you can get your crap in order, why talk about the customer?

Daniel (33:08):

Interesting.

Gene (33:11):

It really is. Or to paraphrase my friend, Peter Moore, who is famous for many things, but he's also the brother of Dr. Joffrey Moore, author of Zone to Win and Crossing the Chasm.

Daniel (33:19):

Oh yeah, great books.

Gene (33:19):

He said, "Until you tackle internal architecture and improvement daily work, you don't belong at the table." Get your crap in order first, and then you can talk about being at the table and contributing to the grandest goals within the organization. So I just like that. This just shows what it takes to be an effective technology leader.

Daniel (33:41):

Yeah. And I remember reading a Harvard Business Review, sometime I go back saying, "Companies, you need execution, the ability to execute, as well as a good strategy. And in fact, if you have a good strategy with no execution, you're going to go nowhere."

Gene (33:53):

Right. Exactly. And in my mind, it was just very important to put the things that we can control the most first. And then from a narrative perspective, the whole notion of customer focus leads to core context. Core are those core competencies that create lasting, durable business advantage that customers are willing to pay for versus context, which is everything else. It could be mission critical, but customers really don't care. So world-class payroll services, world-class expense reporting, important maybe, but maybe not something that customers actually are willing to pay a premium for if that's not what they're buying. So just the idea of saying we built all this great dev tooling, and now we have a shaky CI system, servers that are falling down and they're crashing because of a hardware fans going out or something. That does beg questions like, "Okay, is this something that is a core competency? Or is this something that we really should be relying on a vendor on?" Because it's their core competency. So from a narrative perspective, it's just felt like the right place.

Daniel (34:54):

No, that makes complete sense. Thanks Gene, that's really good context. And it makes complete sense with a target audience in mind. And you've entered in fantastic references. I'll put these all in the show notes so listeners can dive in a bit more detail. I've definitely took in a couple of references that I haven't heard before. So I'll be following up on them, which is awesome. But if people want to follow your work online, Gene, what's the best way?

Gene (35:10):

Yeah. Probably the best way to reach me is on Twitter. I'm real Gene Kim and my DMS are open. Just Twitter is probably the best place.

Daniel (35:18):

Super. Thanks for your time today, Gene.

Gene (35:20):

Oh, thank you so much. And look forward to seeing you soon and hopefully at the DevOps enterprise virtual summit coming up at the end of June.

Daniel (35:27):

I've been to a couple in the real life and they've been fantastic. I've learned so much from them. So I'm going to give them a shout out too. Please do go along to Gene's conference, the DevOps Enterprise Summit. It's fantastic.

Gene (35:37):

Brilliant. Thank you so much.