« swipe left for tags/categories
swipe right to go back »
I learned a very profound thing from my partner Dave Jilk at Feld Technologies 25 years ago. I have been practicing, and getting better at it, ever since. It’s a core part of the way I work with people and I have Dave to thank for it.
First, some context. Feld Technologies was my first company. Dave and I started it in 1987. We hired, then fired, a bunch of part time people and then just worked together – the two of us – for the next 18 months until we hired our first employee (Shawn Broderick). We were cash flow positive every month because we never raised any outside money. We both did everything, working very closely together. As the company grew, we partitioned a lot of things – I became the sales guy – generating much of our new business. Dave became the software guy, managing the team and getting the work done. But we continued to work closely together – he sold plenty of business and I did plenty of work, including doing all the network integration work for our clients, and occasionally managed something.
We were both young and very inexperienced so we learned a lot together, mostly by screwing things up and then fixing them. Sometimes we had a lot of fun, sometimes we were under tremendous stress, and every now and then one of us was miserable. We were (and continue to be) best friends so when one of us was very unhappy, the other could pick up on the vibe quickly and we talked about it.
I remember a stretch of time where I could tell that Dave was really aggravated with me. This wasn’t uncommon – our love and respect included plenty of “moments” as we were both developing into real adults. But this aggravation seemed deeper and didn’t surface in an obvious way.
I remember taking Dave out to dinner at a sushi place called Nara around the corner from our office at 260 Franklin Street in Boston. I can picture how the night felt – dark and empty with plenty of downtown Boston ambient noise. We went to Nara a lot – this was way before sushi became trendy and it was one of the few places in Boston, located a few blocks away from our office. They had excellent huge bottles of cold beer and amazing fish. And it was always quiet and there was always a booth open.
We sat down, got our beers, and I started with the issue, as I often do.
I asked, “Dave, what’s bugging you so much right now.”
“Why? What am I doing that’s bugging you.”
“Working with you is like reading the last page of a novel first.”
I sat nursing my beer for a quiet, long minute pondering this. I mentally read the last page of a novel and thought I knew what Dave meant. Eventually Dave broke the silence.
“When I bring an issue to you, you immediately tell me the answer. 99% of the time you are correct. So I then go spend all of my time looking for a solution that is better that yours. But I only find it 1% of the time. This is incredibly unsatisfying to me.”
I think he may have added something like “fucking demotivating” but by this point I totally groked it. We had an awesome dinner discussing what over the last 25 years we have regularly referred to as “the last page in the book problem.”
Today, I try hard not to start by telling the answer immediately. The CEOs and entrepreneurs I work with need to learn how to get to the answer. And their answer, in many cases, will be better than mine since I don’t have enough context or information to be right 99% of the time like I did when I was the president of Feld Technologies. But even more importantly, a great CEO knows this also. His team doesn’t want to always hear the answer first. Sometimes they do, or need to, but often they want to be able to talk openly, collect data, and come to it over time.
This brief moment has had a profound impact on how I work. While I despise Mr. Socrates (the guy who just asks question after question after question and never expresses a point of view) and don’t emulate him, I definitely ask more “guided questions” when presented with a problem. I tell more stories to try to give examples of how others have solved the problem. And occasionally, when I realize the CEO is asking for the answer (e.g. when Bart Lorang, in the middle of a board meeting, says “Brad, just tell me the fucking answer – I know you know it.”) I tell the answer. But in the back of my mind I always remember that part of learning the answer is figuring out how to find it.
I was fired from my first two jobs. Here’s the story of one of them, which first appeared as part of LinkedIn’s My First Job content package.
“You’re fired.” Those were the last two words I heard from my boss after working for six months at Potatoes, Etc., my first real job. I smirked, immaturely threw my apron at her (I was 15 years old after all), and slammed the door on my way out.
My final three words, preceding hers, were “you’re a bitch.” In hindsight, her response was predictable.
I remember riding my bike home the three miles from Prestonwood Mall where I worked. I had no idea what I was going to tell my parents, but I decided I’d just tell them what happened and see where the chips landed. I felt ashamed of myself for being so disrespectful to my boss, even though she had constantly demeaned me, and all the other people that worked at Potatoes, Etc. I didn’t have any respect for her, but my parents had taught me better and I was proud of my ability to suck it up and not lose my temper.
Potatoes, Etc. was one of those local fast food restaurants in a giant shopping mall from the 1980s. Remember Fast Times at Ridgemont High? Yup – that was us, except Potatoes (as we liked to call it) was staffed by the “honors kids.” I think the Greek souvlaki place was staffed by the jocks and the Corn Dog place was staffed by the stoners, but it all blurs together 30 years later.
In hindsight, the Potatoes, Etc. supply chain was pretty cool. Idaho spuds appeared magically in 50 pounds boxes and ended up in a dank, gross storeroom. Each shift, one person was responsible for getting them, cleaning them, putting them on trays, covering them with industrial grade salad dressing, and racking the trays. Another person was responsible for putting them in the convection oven and making sure there were enough potatoes cooking at all times to handle the spikes in demand. Another person manned “the bar” – cutting open the potatoes and filling them with whatever goop and toppings the customer ordered. And the last person worked the cash register. After we closed, we were all responsible for cleaning up.
Since we were honors kids, we had a lot of fun with the supply chain. We did a good job of load optimization. We figured out process improvements to cut, fill, and serve the potatoes. We ran a parallel process on cleaning and closing up, so we could be done in ten minutes. We were never, ever short on cash.
Our boss was a young woman – probably in her early 20s. I remember the smell of smoke and alcohol on her breath. I remember Saturday morning shifts where she would come in at 1pm, clearly hung over. She liked to yell at us. Her favorite form of managerial shame was to call someone into the back “room” (there was no door) and dress them down randomly so everyone in the food court could hear.
We were good kids. It took a lot to get a rise out of us. Sure – we’d complain to each other about her, but we bonded together and did a good job regardless of her antics. Every now and then she’d do something that she thought was motivating, like bring a case of beer into the back of the store and offer up cans to us (we always declined – remember, we were good kids). But I can’t remember a single time she praised us – or at least me – for anything.
I had been racking potatoes all day on the day I got fired. I was cranky – I wanted to work up front but today wasn’t my day. I was tired – lifting 50 pounds of potatoes and washing them one by one is a drag. And I was bored out of my mind.
My boss probably noticed I was in a bad mood. A kind word from her would have made all the difference in the world. Instead, she came over to the full rack of potatoes, started pulling them off the racks, and without even looking at me dumped them one by one in the sink.
“You suck at washing potatoes.”
“You’re a bitch.”
My parents were gentle with me. They made sure I understood the lessons from the experience, which included the power of respect and not losing your temper with a superior.
But most importantly this was a key moment that I think back to whenever I consider motivation. My boss never did anything to create a context in which we were motivated. It wouldn’t have taken much. And, if she had, respect – and motivation – would have followed. At 15, I learned what it was like to be on the receiving end of a boss who had no idea how to create an environment in which the people that worked for her were motivated. I’ve carried that experience, and the resulting insight, to every subsequent thing I’ve been involved in.
As Amy and I get to the end of Season 2 of Battlestar Galactica, I’m noticing more and more management and leadership lessons. Oh – and it’s awesome SciFi.
In my experience, it’s a challenge for CEOs and management teams to get focused on a small set of numbers that drive behavior. I talked about this in my post Three Magic Numbers. I regularly suggest that you should only have three numbers that you focus on daily – that reflect “what is going on right now in the business.”
You should be able to discern what is going on from the daily trend of these numbers. Sure – you’ll look at plenty of other numbers, but these are the three you focus on every day. You don’t need fancy tech for this – just a white board.
If you are a BSG fan, you’ll recall the white board behind President Roslin’s desk. It has one number on it. The number of survivors alive at that moment. This number started showing up in the opening credits some time in Season 2, and after a few episodes I noticed it changing each time in the credits, often based on what had happened in the previous episode.
This is BSG’s KPI. The number of humans alive. Right now.
When I reflect on this KPI, I realize it drives all the behavior on BSG. The easy behavior to focus on is keeping the number from decreasing. But as Gauis eloquently states late in Season 2, if the trend line continues, based on a complex regression analysis he’s done, the human species will be extinct in 18 years. Soon after, Admiral Adama reminds Roslin that the number generally just goes down, and that Roslin had said early on that if the human species is to survive, the colony needs to start “making babies.”
This is an obvious set up for a much more complex social issue – that of pro-life vs. pro-choice. But obvious set up aside, Adama is focusing on the KPI and reminding Roslin that the goal is for it go up, as well as not go down. It turns out there is a lot of richness in the number.
In my world, as companies grow, I notice a proliferation of KPIs being tracked. On a periodic basis, I encourage CEOs to keep paying attention to all the numbers, but surface – on a daily basis – the three magic numbers that drive their business.
Do you know your three magic numbers?
I’m doing a one hour CEO roundtable on an “about weekly basis” with each of the Techstars classes. Yesterday I did a face to face with the Techstars Boulder CEOs (they are across the hall from my office) and then I did my meetings with the Techstars Chicago CEOs and the Kaplan EdTech Accelerator CEOs by video conference.
This is a new experiment for me. I’m trying a different approach to mentoring the Techstars teams this year. I’m still a lead mentor for two of the Boulder teams (Kato and SnowShoe) but for all the other programs, including Boulder, I’m trying a weekly one hour CEO only session.
One of my big goals is to generate more peer interaction between the CEOs of the various companies. We do this aggressively within the Foundry Group portfolio and it’s one of the really powerful things about Techstars. But historically it’s been adhoc and random, rather than in an organized way. This is an effort to get the CEOs to really bond with every one of the other CEOs during the program.
So far the experiment is working great from my perspective. I’m stunned by the depth of the conversation and I can see the relationship dynamics being very broad as well as intellectually and emotionally intense.
Each of the three meetings yesterday were totally different, as Techstars Boulder is in week 8, Techstars Chicago is in week 4, and Kaplan EdTech is in week 2. As I was taking a shower this morning, I kept thinking about the rant I went on during the last 10 minutes of the meeting with the Techstars Chicago CEOs.
By week 4, a team is deep in things. The stress is showing. Everyone is tired and working at their max capacity. They’ve been exposed to a wide range of mentors and lots of conflicting data. Stuff is breaking all the time. Everything is uncomfortable and – in some cases – distressing.
In reaction to a particular conversation, I strung together quotes from three of my favorite books about entrepreneurship. The rant went as follows:
- “It’s not that I don’t suffer, it’s that I know the unimportance of suffering.” – John Galt in Atlas Shrugged
- “Fear is the mind-killer.” - the Bene Gesserit is Dune
- “Anxiety, the next gumption trap, is sort of the opposite of ego. You’re so sure you’ll do everything wrong you’re afraid to do anything at all.” – Robert Pirsig in Zen and the Art of Motorcycle Maintenance
I used the quotes as the anchors on a longer rant, but I did it extemporaneously. I hadn’t realized how nicely these quotes fit together until this particular moment, prompted by the particular situation. In hindsight, the only quote I forgot was my favorite of all time – “Do or do not, there is no try.” – Yoda.
And – it reminded me that three books should be on every Startup CEO’s reading list along with Matt Blumberg’s new book, Startup CEO.
An increasing number of companies we are investors in are focused on DevOps. A year or so ago I read an early draft of a new book titled The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win. I really enjoyed it and asked Gene Kim, one of the authors to write a guest post on DevOps. He wrote it a while ago and it has sat in my draft queue waiting for the perfect moment to emerge. That moment is now. Following is a guest post on DevOps by Gene Kim, Multiple Award-Winning CTO, Researcher, Visible Ops Co-Author, Entrepreneur & Founder of Tripwire.
Since 1999, my passion has been studying high performing IT organizations. On this journey, we benchmarked 1,500 IT organization to understand what differentiated the highest performing organizations and allowed them to do what the others only dreamed of. Our findings went into a book that we published in 2004 called The Visible Ops Handbook, which described how these organizations made their “good to great” transformation.
Since then, this journey has taken me straight into the heart of the DevOps movement. Although I initially dismissed DevOps as just another marketing fad, my friend John Willis corrected me, in the way that only true friends can do, saying, “Don’t be dense. DevOps finally proves how IT can be a strategic advantage that allows a business to beat the pants off the competition. This is the moment we’ve all been waiting for.”
In that moment, I saw the light. Over the years, I’ve come to believe with moral certainty that everyone needs DevOps now, especially software startups where the successful execution of Development and IT Operations preordain success or failure.
Today, we can see how DevOps patterns enable organizations like Etsy, Netflix, Facebook, Amazon, Twitter and Google to achieve levels of performance that were unthinkable even five years ago. They are doing tens, hundreds or even thousands of code deploys per day, while delivering world-class stability, reliability and security.
DevOps refers to the emerging professional movement that advocates a collaborative working relationship between Development and IT Operations, resulting in the fast flow of planned work (i.e., high deploy rates), while simultaneously increasing the reliability, stability, resilience of the production environment.
The culture and practices that enable DevOps to happen cannot be delegated away. In a growing startup where teams start to specialize and multiply, the chaos of daily work often starts to slow down the smooth flow of work between Development and IT Operations, sometimes even resulting in outright tribal warfare.
In this blog post, I’ll describe what this downward spiral looks like, and what everyone in the company must do to break this destructive pattern and ensure that Development and IT Operations work together in a way that creates such a competitive advantage that it may almost seem unfair.
Why Everyone Needs DevOps
There is currently a core, chronic conflict that exists in almost every IT organization. It is so powerful that it practically pre-ordains horrible outcomes, if not abject failure. It happens in both large and small organizations, for-profit and non-profit, and across every type of industry.
In fact, this destructive pattern is the root cause of one of the biggest problems we face as an industry. But, if we can beat it, we’ll have the potential to generate more economic value than anything we’ve seen in the previous 30 years.
I’m going to share with you what this destructive pattern is in Three Acts, that will surely be familiar to you. (You can get the whole story in my book, The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win).
Act I begins with IT Operations, where we’re supporting a large, complex revenue generating application. The problem is that everyone knows that the application and supporting infrastructure is… fragile.
How do we know? Because every time anyone touches it, it breaks horrifically, causing an epic amount of unplanned work for everyone.
The shameful part is how we find out about the outage: Instead of through an internal monitoring tool, it’s a salesperson calling, saying, “Hey, Gene, something strange is happening. Our revenue pipeline stopped for two hours.” Or, “the banner ads in my market are being served upside down and in Spanish.”
There are so many moving parts that it takes way too long to figure out what caused the problem du jour, which means we’re spending more and more time on unplanned work and increasingly unable to get our planned work done.
Eventually, our ability to support the most important applications and business initiatives goes down. When this happens, the organization suddenly finds itself unable to achieve the promises and commitments made to the outside world, whether it’s investors, customers, analysts or Wall Street.
Promised features aren’t delivered on time, market share isn’t going up, average order sizes are going down, specific revenue goals are being missed…And that’s when something really terrible happens.
In Act 2, everyone’s lives gets worse when the business starts making even bigger promises to the people we let down, to compensate for the promises we previously broke. Often, the entire organization starts dreaming up bigger, bolder features that are sure to dazzle the marketplace, but without the best grasp on what technology can and can’t do, or fully realizing what caused us to miss our commitments in the first place.
Enter the Developers. They start seeing more and more urgent date-driven projects put in the queue, often requiring things that the organization has never done before. Because the date can’t be moved (because of all those external promises made), everyone has to start cutting corners.
Development must focus on getting the features done, so the corners that get cut are all the non-functional requirements (e.g., manageability, scalability, reliability, security, and so forth). This means that technical debt starts to increase. And that means increasingly fragile infrastructure in production.
It is called “technical debt” for a reason—because technical debt, like financial debt, compounds.
When technical debt begins to accumulate, something very insidious starts happening. Our deployments start taking longer. What used to take an hour now takes three hours, then a day, then two days—which is okay, because it can still get it done in a weekend. But then it takes three days, and then a week, then two weeks!
Our deployments become so expensive and so difficult that the business says that we have to lengthen the deployment intervals, which goes against all our instincts and training. We know that we need to shrink the batch sizes, not make them bigger, because large changes make for larger failures.
The flow of features slows to a trickle, the deployments take even longer, more things go wrong, and because of all the moving pieces, issues take even longer to diagnose. Our best Dev and Ops people are spending all their time firefighting, and blaming each other when things go wrong.
I’m guessing that most of you can relate to at least some portions of this story? As I said, this happens both in large enterprises and growing startups alike. In my fifteen years of research in this area, I’ve found almost all IT professionals have experienced this cycle.
Act 3: How DevOps Breaks Us Out Of Our Downward Spiral
We know that there must be better way, right? DevOps is the proof that it’s possible to break the core, chronic conflict, so we can deliver a fast flow of features without causing chaos and disruption to the production environment.
When John Allspaw and Paul Hammond gave their seminal “10+ Deploys Per Day: Dev and Ops Cooperation at Flickr” presentation at the 2009 Velocity Conference, people were shocked and amazed, if not outright fainting in the aisles at the audaciousness of their achievement.
It wasn’t a fluke. Other organizations such as Facebook, Amazon, Netflix and the ever-growing DevOps community have replicated their performance, doing hundreds, and even thousands, of deployments per day. DevOps is not only for large, established companies. It’s for any company where the achievement of business goals rely upon both Development and IT Operations. These days, that means almost every company.
We all need to be putting DevOps-like practices into place. This is why Kevin Behr, George Spafford, and I wrote The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win.
A novel, you might ask? How is a novel going to solve my problems?
As a friend once told me, “Before you can solve a complex problem, you must first have empathy for the other stakeholders. And story-telling is most effective means of creating a shared understanding of the problem.”
Dr. Eliyahu Goldratt demonstrated the power of a novel as a teaching tool through his book, The Goal: A Process of Ongoing Improvement. It’s a novel written in the 1980s about a plant manager who has 90 days to fix his cost and due date issues or his plant will be shut down. When I read this book nearly 15 years ago, I knew that this story was important, and that there was much I needed to learn, even though I never managed or worked in a manufacturing plant.
It isn’t an overstatement to say that The Goal and Dr. Goldratt’s Theory of Constraints changed my life—in fact, it probably was one of the biggest influences on my professional thinking. For eight years, my co-authors and I wanted to write The Phoenix Project, because we all believed that IT is not a department, but a strategic capability that every business must have.
As you can imagine, I was incredibly honored and thrilled when Jez Humble, author of the award-winning book Continuous Delivery recently told me, “This book is a gripping tale that captures brilliantly the dilemmas that face companies which depend on IT. The Phoenix Project will have a profound effect on IT, just as The Goal did for manufacturing.”
For those of you are looking for some places to start your DevOps journey, here are my three favorite DevOps patterns:
- Make sure we have environments available early in the Development process. Enforce a policy that the code and environment are tested together, even at the earliest stages of the project.In the ideal, IT Operations is able to create an environment (that is, everything except for the application code: databases, operating system, networking, virtualization layer, etc.) with one step. That can be as simple as copying a virtual machine, or as complex as an automated build system that generates the environment from scratch (e.g., puppet, chef, etc.)Furthermore, use the same build mechanism to build the Production, Test and Dev environments at the same time. If we modify the Agile sprint policy so that instead of merely having shippable code, we have shippable code and the environment that it runs within, we’ll have done code deployments many, many times when it’s time for the real-life production deployment.
- “Wake up developers up at 2 a.m. when they break things.” Yep, you heard me. This quote came from Patrick Lightbody, the CEO and founder of BrowserMob. He continued, “When we woke up developers, we found that defects got fixed faster than ever.”The goal is to shorten and amplify feedback loops, and to bring Development closer to the customer experience. In DevOps work streams, developers often deploy their own code, and fixes forward when things go wrong. By doing this, developers can see the consequences of their decisions and actions.(Note the symmetry here: the previous pattern #1 about making environments available early is all about embedding IT Operations into Development, while this pattern is about putting Development into IT Operations.)
- Create reusable deployment procedures: When every deployment is done differently, every production environment can become different, like snowflakes. When this occurs, no mastery is ever built in the organization in procedures or configurations. As Luke Kanies said, “If your infrastructure is special, you’re doing it wrong.”To make this reality, create reusable user story for IT Operations, such as “Deploy app into high availability environment,” which then goes on to define exactly the steps to build the environment, as well as how long it takes, what resources are required, etc.By doing this, we codify the deployment and engineering procedures requires to build reliable, resilient and properly configured environments, and can then factor that into our planning processes, such as in the PMO.
If you enjoyed this taste of DevOps and believe it can help achieve your goals, “The Phoenix Project” is available now, or you can download a free 170 page excerpt of the book. And of course, you can always find the latest writings on DevOps at the IT Revolution blog, where you can get our free whitepaper “The Top 11 Things You Need To Know About DevOps.”
Long live DevOps!