On Configuration Management and Company Culture: an Interview with Chef’s Adam Jacob
We recently interviewed Adam Jacob (@adamhjk). Adam is the co-founder and CTO of Chef, which helps organizations manage their IT infrastructure. Alongside his role at Chef, Adam is an active advisor to our venture fund, Amplify Partners. Disclaimer: Chef is an Amplify Partners portfolio company.
Tell us a bit about yourself and your background.
Sure. I’m a systems administrator by trade. I started running bulletin boards when I was eight. The coolest thing about my computer, in my opinion at the time, was the modem. By 16, I officially became a system administrator and started working for an ISP because it was the early ’90s–if you knew how to turn on a modem–you were hired.
Then, for quite awhile, I worked for a number of companies that basically acquired a lot of other companies. They’d find something in a target market, snap it up and make it more efficient. This rapidly changing set of experiences gave me the chance to see a lot of different infrastructures. Overall, during these ten years, I got to see a lot of different code, a lot of different applications and a lot of architectures.
About seven years ago, I got tired of that cycle and decided to start a consulting company. So I reached out to one of my buddies, Nathan Haneysmith, who was one of the Linux practice leads for IBM Global Services West. We joined up and sold fully automated infrastructure to startups. You could pay us a flat fee, between $25,000 and $50,000, and we would fully automate your infrastructure, soup to nuts, from provisioning, application deployment, monitoring, trending, identity management, backups–all of it. Practically speaking, we’d be on retainer and help you maintain it.
The idea was to use the same automation across all the customers, and make it easy for us to maintain lots of them. Unfortunately, it didn’t work out that way. For a particular customer, the automation worked just fine. But for five customers? Not so much. It became a mess.
So to solve our own problems, we kept building tools, and eventually I wrote Chef, showed it to Jesse Robbins, and the rest is history.
Can you expand a bit on what Chef does?
What Chef does, at the most fundamental level, is let you describe how you want your servers to be configured by declaring a state. It then looks at your system and checks if that state is correct. If it is, it does nothing. If it’s wrong, it figures out how to fix it; and it fixes it for you. That’s the quick summary. The big tool in the space when we got going was Puppet (though CFEngine 2 and 3 were both out at the time) – but neither were a good fit for solving our issue (which was having so many different infrastructures to manage at once).
For context, Amazon’s EC2 had landed about a year before we started the company. All of a sudden, you could run infrastructure in minutes. We aimed to build a platform that’s full of useful primitives so people could solve what they needed to in their own context. More specifically, you define policies programmatically via the API, rather than manipulating a bunch of opaque things on disk all over the place.
How did you come to the conclusion that Chef had the potential to be a real business?
Well, it felt like we truly had built a better mousetrap. My prototype worked better than I thought it would, and people who used it loved it. We also believed we could build a really high scalability, high density hosting service. In fact, our original plan was to run a hosted version of Chef, in addition to the open source implementation of the server you could run on your own. We were about six years too early, it turned out, for the hosted strategy.
By contrast, things are quite different today. The Netflix internal IT group recently decided they aren’t going to run any internal servers anymore. If that had been true six years ago, I think the hosted approach would have been a fabulous business.
Six years in, where does Chef sit in a rapidly evolving infrastructure landscape?
At a really high level, everything is increasingly being consumed digitally. Everybody is carrying a smartphone in their pocket. Everybody who uses Uber would rather use Uber than hail a cab. And Uber is just a small slice of this mega transformation.
Think about what happens when you buy, say, a tractor. What if your new combine came with a Wi-Fi hotspot and a 4G uplink. On top of that, you had an app for your iPad that would give you the full maintenance breakdown for the tractor–and it could auto-order parts. That sort of stuff has become possible and awesome, and as more and more of it impacts our lives, we start to demand the same across every domain. This, in a nutshell, is “software eating the world.”
What does this mean for infrastructure? All the things we used to do in building our operations, the infrastructure, the IT–what Jeff Bezos called the undifferentiated “muck”– all that moves from being relegated as a seemingly unimportant back-office affair to a critical part at the front of the front office.
It’s obvious this transition won’t happen seamlessly. Inertia is a powerful thing. The core problem is that the massive businesses of today have to fundamentally “re-tool” to survive in this new world. And it’s not a matter of simply re-doing the UI. They have to rethink their internal operations, which in turn really shapes how the whole company functions from the top to the bottom.
So where does Chef fit into this? We think of ourselves as a programmatic layer for describing how all the new and old components (all the muck, so to speak) should come together and run.
I’ve heard you say that tooling is culture institutionalized. What do you mean by that?
Your culture, whether it’s a good or bad one, starts with some fundamental belief. There’s some set of beliefs, whether you write them down or not, that defines you. As a company, you look to build systems and structure that match your beliefs.
Suppose, hypothetically, we run an online bank. If you, as a customer, can’t access your account, you’re going to be (most likely) be upset. So, as an organization, we set as a core goal to have 100% uptime and never make our customers upset in this way.
There are a number of approaches to achieving quality of service. You could slow everything down and make it go through an incredibly rigorous QA process where you deploy a change to the website only once a year. In this instance, you control everything, from the power in the data center, all the way up to the application code being deployed with an iron fist. And yes, this might get you to 100% uptime if you execute well. In this particular approach, you have implicitly or explicitly institutionalized a culture that decrees failure as unacceptable and that the path to 100% uptime is through rigorous human verification and process.
Then, suppose a consultant came to this bank and encouraged them to change their beliefs through a new set of practices and workflows. They’ll likely rebuff such efforts because the tooling has institutionalized the belief and behaviors that characterize their organization.
They might say,
“I can’t do that.”
“I couldn’t possibly deploy the online bank code 100 times a day.”
“How would that even work with my ticketing system?”
Tooling is, in this way, culture–institutionalized. And if the root of the current and very existential problem for many organizations is as profound as a fundamental economic shift towards software as a delivery model for (increasingly) everything, then cosmetic “cultural” adaptations will likely flounder. The legacy organization will fail because it can’t recognize that systems and tooling are the primary cause of their stagnation and failure to change, despite saying the right things.
Ultimately, the meme that claims, “You shouldn’t care about the tools; you should focus on the culture,” is not particularly helpful or true.
Bringing it back to your experience with Chef, how have you gone about codifying and shaping the company culture?
The default mode for us at Chef is distributed. As an example, when running an all hands meeting, the speaker addressing the company sits in a mostly empty conference room and broadcasts using GoToMeeting. During this, in the background, people can ask questions through HipChat.
I should note that we didn’t start this way for these meetings. In the past, we’d center our all hands meetings in Seattle. Everyone would pile into a big room. But while there was a ton of really great energy, the remote folks had a much tougher time feeling engaged (or even hearing what was going on).
We then made the decision to center everything around the remote experience. This vastly improved things. Now, the meetings are all recorded, posted and available if you missed one. Deciding that, organizationally, we’re going to design ourselves around the experience of people being remote is a key piece of our own culture. I have to thank Kevin Smith, Christopher Brown and others who taught me about thinking in terms of remote work.
Another part of our work at Chef has been getting everyone comfortable with intense collaboration. If you think about organizational structure, you really need to get to a place where what you’re building is a network and the people in that network should be comfortable with getting context from other nodes directly. If you are remote, and you’re in a different time zone, and what you need in order to make a good decision is a piece of customer data or a piece of our financials or some other detail, do you wait for me to wake up on the West Coast to contact someone to get financial data, or can you go directly to them and just ask for the information? You need to be able to go directly to the relevant people or information immediately.
This broad philosophy requires a pretty high degree of trust, but the flip side is the more of that data you expose and the more direct communication you can have with different people, the more people will start solving their own problems in creative ways. Rather than instituting a central authority, which is really hard when you’re distributed, you start to see more empowered individuals making decisions within their own context and then across other contexts in order to move the business forward.
There’s something distinctly honest about a company that helps organizations wrangle distributed servers building those products in a distributed way.
Thanks. I should say that of course there are parts of our company that are not what we want them to be. It’s not like we’re some magical unicorn; I wish we did more internal continuous delivery, for example. We’ve helped big companies get really complex continuous delivery projects and be very, very successful. Yet, we ourselves are still struggling to get there. We can get so focused on what’s happening for our customers that we lose a little bit of internal focus. But we’re working on it every day.
Thanks a ton for your time Adam!