Skip to main content

An extremely long and convoluted argument for open sourcing internal tools

One of my favourite ideas is that of a Local optimum, a mathematical concept that can be applied more broadly. If you've never heard of the term, it's that starting somewhere you may incrementally improve but never discover that a better solution exists.

For instance, imagine you're in the 10th century, trying to develop the fastest means of transportation. The current state of the art is the horse; you therefore decide to spend time breeding better horses. You make a small improvement, but with this approach you have no chance of discovering any of the many technologies that have now replaced the horse. (1)

Pseudo-mathematically, this might look something like:





The graph above - despite its paucity of dimensions - captures not only the idea that as one reaches peak horse there is limited (and difficult) improvement left, but that to explore another avenue won't deliver immediate success. And indeed, if our 10th century entrepreneur had decided that horses were a dead end, no amount of experimentation in his lifetime with other forms of power would have been likely to improve matters. Even in the 18th century, when technology had advanced sufficiently to make motorised vehicles possible, it took many years of effort to make them useful.
See the source image

If you're trying to improve 'something', the choice that will most reliably reward is incremental improvement. Measure exactly where you are, decide what you want to improve (horse speed? horse endurance?), and put effort towards that. After all, the chance that you will discover some entirely new and successful way of looking at a well understood problem is small, and even if you are that talented and lucky, the chance of executing on it is even smaller. Our imaginary 10th century horse breeder could have alighted on the idea of an internal combustion engine, but without an incredible amount of time and capital would have been unable to put that idea into practice.

But that sounds a bit terrible, right? Surely not everything is 'well understood', and many things are undoubtedly 'well understood' wrongly. However, this is all a lead in to my (under-informed by knowledge or practice) ideas about human endeavour (business?), where one should be careful about where and how it makes sense to chase an optimum.

Chasing the right optima

If you have a well-functioning team (department? company? group?), people will be attempting to improve productivity, and the natural way to do this is to climb the hill immediately in front of them. When you have a small company, this is almost always closely related to the reason for its existence. So if you're a horse breeder, you're trying to optimise horses. You're not likely to worry so much about accountancy, or fertiliser for your track, or construction of stables: it's enough to hire someone to do a job that reflects the current state of the art, since there's really a limited prospect of competitive advantage and your resources are better spent on what you intimately understand.

However, as your business expands - perhaps you have a multi-country horse breeding empire - such things will inevitably increase in importance, and as it becomes worthwhile to do things like hire a full time accountant, so people and teams spring up with missions tangentially related to your core reason for existence. Our hypothetical accountant, now employed full time with an assistant, now has a lot of time to think about the very best and most efficient way she could do the accounts.

That's great, right? Maybe she could fire the assistant, or at least set up the structures such that the business could grow without needing to employ more accountants. However, if they're really good, they may start to invent processes that are well outside the norm for accountants, either tied intimately to the business (i.e. hill-climbing the local optimum), or attempting to advance the state of the art in accountancy generally (attempting to discover another optimum).

Attempting to discover another optimum in an area that is not closely connected your primary business seems the most futile of the two. For instance, imagine we're in the late 1970s, and our accountant has an idea for something like a spreadsheet. She convinces people that she should hire a computer programmer, and - this is a fantasy, remember - after 6 months a great piece of software is produced. Sure, it has a few quirks, but it really does make doing the accounts faster and it's really useful for ad-hoc financial projections. Yet even in this unlikely scenario, where it pays off in the near term, unless the 'spreadsheet' is spun out into a wider business, in a few years it will be surpassed by VisiCalc and Lotus 1-2-3. And it will be easy to hire people who can already use them. What was innovative software is now merely a source of annoying bugs, requirements to hire programmers to port it to new systems, and specialised training for anyone to use it.

What's happened in both of these cases - the local and other optima - is that they're chasing the accountancy optimum as they should, but they've been so enthusastic about it that they're over fitting. They've left the business in a state that makes it harder to replace them, to employ new people at all, or to adopt the inevitable innovations of the 'rest of the world' which should naturally come with staff turnover. Adding or replacing people on the team, which ought to be an advantage (fresh ideas and enthusiasm!), now becomes a hindrance (many months to train! previous experience useless! new hires frustrated by inability to contribute!). And what were obvious measurable wins become entrenched 'how we do things here', with the sunk cost fallacy and bureaucratic inertia making it hard to unwind.

Software

Of course, what I really care about, or indeed know about, is software and programmers. As a software product based company grows, the percentage of developers allocated to customer facing development inevitably declines. It makes sense to allocate people to specific tasks (SRE, CI, testing, developer tools...), and since they're programmers they'll inevitably want to make things more efficient. And if you hire programmers, their tendency is to solve problems with more programs, just as if you hire managers their tendency is to solve problems with more process. Yet the former can be just a harmful as the latter to organisational flexibility.

Consider the first item on the famous Joel Test to evaluate the quality of a software team: "Do you use source control?". I'm hoping that, for most programmers these days, this is a 'WHAT?'. After all, everyone uses source control, right? But RCS has been around since 1982, CVS since 1990, but even by 2000 NOT using source control was so prevalent in the industry that it was the most important question to ask. What was going on?

Sure, some of it was simple inertia and reluctance to break what was already working. And I'm sure many of the hold-outs were small teams that simply weren't connected enough to recent software development to realise that it was a clear advantage. But for larger organisations, my suspicion is that they'd overfitted the local optimum: they'd developed complex bespoke processes around how to manage changes that they were deeply invested in, both in man hours and psychologically, and so every time some new hire said "hey, why don't we use source control?" it was a hard sell. They made a small win for a few years with additional process/programs, but when the industry moved on struggled to escape their local optimum.

Still, at some point the organisation does become large enough to do... interesting things. Google apparently has enough stuff that ramp up time is measured in months: "You are working in one of the most convoluted engineering environments out there. There's a tool for every problem, and a quirk for every tool, and probably another quirk for the first one." When you're that big, it makes sense to try and push the state-of-the-art forward internally: after all, there are enough people working on it to make it likely that 'better than the world' solutions will be found to common problems AND there are a critical mass of people worth training. Nevertheless, even at Google's size, there's a significant ongoing cost to maintaining all of that software and process, and in the long term even the most innovative organisations tend to stagnate and become rigid (IBM?).

However, for those of us not in Google (Amazon, Microsoft, Apple, Facebook, ...) size organisations, any internal project not related to the primary objective is a cause of concern. It's a way to drift away from the rest of the world, and if not subject to public scrutiny, most likely in a unproductive way. It's easy to make an argument to a few people that your project will produce a win, even if it's a bit hacky or adapted to current quirks; however, if it is truly solving a generic problem, the chance that it really will be better than the world's state of the art in a few years is limited. It's both statistically unlikely, and even if it were immediately successful you won't have the manpower to keep it ahead of the world.

So, what should you do? If it's not something you want to sell (or declare a separate objective in itself), then just make it public. After all, you're making the claim that it's better than anything else out there by developing it. Let the public help you validate its usefulness, and make sure it's not overfitted to your current use case - the rest of the world is a proxy for your own organisation in a few years time.

What you're doing then is backing the world against the largest companies. Perhaps I'm just urging you to believe in some kind of software Darwinism: make the 'rest of the world' ecosystem as dynamic as possible, and help it converge on good solutions as quickly as possible. Every bit of purely internal code is deadweight, encouraging sunk cost fallacies, frustrating new hires, and demanding man-hours for its continued existence.


Conclusions

  1. Choose your optima carefully. The world is a gigantic simulated annealing machine, and will throw up better approaches. If you want to beat the world, only try and beat it where you're prepared to commit to it.
  2. Never perfect a local optimum outside your core area. You're just buying yourself inflexibility.
  3. Only consider investing in non-local optima outside your reason for existence (i.e. "we can do this better than the world") if you're:
    • the size of Google.
    • prepared to spin it out as a separate goal/product - i.e. sell it or give it to the world (research publications, open source). The chance that you discover one is low, it won't be a competitive advantage, putting it out there in the world will help validate and improve it, and if you keep it to yourself it's useless in the long term.
  4. People turnover outside your core area is valuable. You don't want to be tied to the practices of your crazy accountant, and your main aim is to keep up with the world, not out-do it.

Comments

Post a Comment