Make boring technology choices!

In the last few years, there’s been a deluge of new technologies that have appeared as possible choices for CTOs and Engineers. There’s a trendy new technology in every category from NoSQL, new programming paradigms, container/infrastructure orchestration systems like Kubernetes. Drummed up by big companies’ blog posts, they promise to make things faster, cheaper, and more reliable. Making the right, or at least “not the wrong,” technological choices is critical as it will impact team velocity, your software reliability, but also your ability to attract the best talent.

Great tools and technologies have a way to disappear and can handle pretty much anything you want to throw at it. On the contrary, bad tools require you to pay close attention to it. They become a limiting factor in your technical designs. A typical example of this difference is the choice of databases between PostgreSQL and MongoDB. Regardless of what use-case fits your application better, the administration of these databases is entirely different:

• PostgreSQL defaults are safe and reliable: you will not be surprised by its behavior, and you can reliably build an application without fearing a loss of data.
• MongoDB, on the other hand, forces you to think about how to configure and deploy it. If you choose MongoDB, as you build your software that relies on it, you’ll have to make sure you use MongoDB the way it was intended to be used. Otherwise, you’re at risk of losing data. In other words, the tool is in the way of the maker.

The story of a failed project

In one of my first jobs (late 2000s), I was hired to work on a promising new version of the platform that the company had been relying on for more than ten years. It was a successful company by a lot of metrics, but it was relying on some arguably shoddy infrastructure and antiquated technological choices.

To their credit, the leadership team decided to invest in a lot of new technologies for the new project. They committed a large team to that project: more than 50 engineers were brought on board. They picked some of the most exciting technologies at the time, such as Single Page App (SPA with Knockout.js), a new promising database (MongoDB), a brand new Continuous Integration and Deployment pipeline.

Initially, the project started well: the code was committed and deployed quickly. Unlike SQL with its schemas, the document DB store would not complain about changing data being inserted and would not require painful migrations. Unfortunately, the reliability of these tools was significantly overestimated, and they turned out to fail relatively often. Even worse, chained together, the failure rates (already relatively high) of each of these technologies were multiplicative… For example, MongoDB was the first technology to show signs of failing for some complex queries. A layer of code that I pushed fixed the issue on MongoDB, but following my changes, the Continuous Integration and Deployment pipeline would then took hours to run the suite of tests per commit, leading to a backed-up release process. From there, problems of evolving the data models and the frontend only got worse, and the pace of release came to a crawl… Eventually, consultants from multiple companies were called in to “fix the problem”, but… they were just adding their own layer of complex solutions that ended up with even more issues. The project was killed, and the teams returned to the old codebase…

So, what happened exactly? Is there something to learn from that failed project?

Making boring technical choices

Whenever friends or people I mentor come to me asking about technology choices, I recommend them to go with whatever is considered “boring.” I need to take a moment to distinguish boring from: not scalable, not efficient, not interesting, etc.

Good “boring technologies” share a few common traits:

• They are reliable and have been shown to work in a variety of different scenarios for long periods of time. For example, a lot of Facebook runs on a modified version of MySQL, at a scale that no startup can match.
• They are unsurprising. Surprise is what will make you waste weeks trying to figure what happened and why suddenly your systems went down. Think about a plane that would go down as soon as a knob would be in the wrong position: I guess you wouldn’t want to step in it.
• They have a good level of adoption. A community of users that can help figure out issues when things don’t work can make or break a project. If you can get advice when you need it, that will go a long way in making sure your project won’t come to a halt. Having personally wasted hours on a never-before-seen bug and nobody to help, don’t underestimate the impact of low adoption on the success of a project.

The aerospace industry has known for decades that when building planes, making the deliberate choice of using tried and tested technologies. For example, the Curiosity Rover was launched with “outdated” camera sensors at the time, in order to be able to establish their reliability within the system. It’s not a flaw; it’s a design choice. Your startup may not be building a plane or a rover, but you want to consider your work as a reliable platform for future innovations and make choices that will be reliable if you succeed and complexity grows.

Where to use non-boring technologies?

It’s difficult to resist the urge to go with the newest technologies. Not only are they inherently exciting, but there is this fact that non-boring technologies will help attract the best talent. Even Paul Graham, in one of his old essays talks about it. Having worked at big companies showcasing how they use this or that new language or framework, I can tell you that from the inside, things are not as rosy as they claim. They often have multiple teams and dozens of SWEs, Site Reliability Engineers actively maintaining the deployment of these technologies. This is something that, as a startup, you can’t afford to do. Even if you could, these resources are likely better to be spent elsewhere.

I think there’s still room for experimentation and taking a calculated risk. Not every choice of technology needs to be “boring”. If you want to be daring, I recommend using a simple principle: the less critical a system is, the more you can afford to take a risk on the choice of technology. For example, a database is an absolutely critical piece of your infrastructure. Any incident with data loss or security could be a startup-ending event. You should definitely go with “boring technology.” On the other hand, for the choice of your frontend JavaScript framework, which is subject to constant changes anyway, you can be more adventurous and choose one of the most recent technology. The ROI could be positive in the end.

–

Startups are risky: they fail for multiple reasons: lack of funding, poor traction, bad hires, poor product-market fit, the list goes on forever… Each department’s role is to limit the risk of them becoming a danger to the business: the chain is as strong as its weakest link. When it comes down to choosing technologies, don’t underestimate the lasting impact of your decision and go with the safe, reliable, and maybe a bit boring choice that will be a stable foundation for your startup to thrive.

New comment