How we choose technologies at Corso

Choosing a technology can be a difficult, stressful, time-consuming process. It's easy to find yourself in complete analysis paralysis, spending tons of time and money ensuring that your decision is the correct one.

Let's say you need to choose a database for your new system. Just about every application ever built uses a database, so no matter what you are building this is hardly untrod territory. That means it should be easy to find somebody who has built an application like yours, right? You just need to read their blog about why they picked the tech, mirror their choice, and voila!

The first thing you do is to read up on the different types of databases out there, just as a quick refresher. After all, you don't have to pick a new database every week, so it's important to be fresh on what's out there. Relational vs. non-relational, centralized vs. distributed, OLAP vs. OLTP. Documents, graphs, objects, keys/values. Wow - there are a lot of ways of storing data! You remember reading about how the CAP theorem is important, so you put that on your refresher list. Oh, and isn't PCALC considered to be a successor to CAP since it addresses distributed systems? ACID - gotta brush up on the details there. Or is it going to matter? Maybe we can skip figuring out whether each database being researched is ACID, in the interest of time. Plus, we just started using a new cloud provider. Better make sure we take our new environment into consideration...

... and suddenly, this easy choice isn't looking very easy. Since you still have to figure out the rest of your tech stack, this process will have to be repeated several times for development languages, frameworks, API styles, and more.

Unfortunately, having experts to assist doesn't help as much as you would think. Going back to our database example, the veteran DBA you have on staff is going to have one opinion, your VP of engineering is going to have another, and your systems guys will think something else. Who is the right expert to listen to?

Making decisions around technology

In our opinion, the impact around your technology choices isn't as significant as you think it is at the time you are making the decision. The decision is ALWAYS made with incomplete information, and any time a company looks back regretting a technology choice it is almost always because more (and better) information is now available.

Picking something you can quickly move forward with is almost always more valuable than spending an inordinate amount of time being absolutely sure you make exactly the right pick.

You could spend weeks or months putting together requirements documents, defining test cases, building proofs-of-concept, running tests, and hiring consultants to validate your choices. Or, you could gather up a few key individuals who have the capability, responsibility, and authority for the success of the project. Collaboratively lay out the basic needs of the tech you are selecting, and give everyone some time to do some research and talk to some team members and stakeholders. Then, make the decision together after a few hours of discussion.

Overdoing the process of selecting the perfect technology can be both a form of future-proofing and a form of premature optimization. Selecting a technology as if you know everything it will ever need to do is delusional, as is trying to optimize for some eventual use case that may or may not ever come.

You've probably heard technologists say things like "I sure wish we had picked the right components from the beginning", with a wistful look in their eyes. Or "Boy, if we had just made decisions knowing we were going to scale 100x over the next 3 months, it would have been totally different". Just remember to recognize the survivorship bias - those companies may have picked the "wrong" technology, but it wasn't wrong enough to kill the company. The companies that never actually got something deployed because they were too busy making sure they picked the right stuff? You won't hear them complaining, because they never got to the point that it mattered.

Realistically, could those wistful tech organizations have anticipated the future well enough to pick the "right" technologies? Ir is it wishful, revisionist thinking to look backward and pine for the different choices they would have made?

When we talk about how unrealistic it is to think we can future-proof software, we suggested working towards flexibility instead. The same philosophies apply here; instead of pretending that we know what the future holds, we should do everything we can to architect for flexibility.

So how should you arrive at a technology choice? How do you avoid the analysis paralysis?

Specific recommendations about how to pick software based on your specific needs are impossible to do in a general post like this of course. The right choice must be made by capable, responsible people who have the authority to make the decisions. When people understand the problem space well and are familiar with the available options, choosing correctly is usually straightforward.

Here are a few points to think through when choosing what technology to use:

How well does the tech fit into your existing stack?

It's unusual for a technology to live on an island, and a greenfield approach to a brand new set of technologies happens pretty infrequently. The more integrated a new solution is with your existing software and the technology you expect to have in your stack, the faster it can start to provide value.

How mature is the solution?

All else being equal, a mature solution will generally function with fewer problems, should change less over time, and should have more people who are familiar with it. All of these are valuable attributes for components in your tech stack. At the other end are the technologies that are constantly changing, crash all the time, and are hard to hire experts for.

Is the solution secure?

Security needs to be a consideration, at the level that it matters for your company and for the solution needed. Security and usability are often inversely correlated, and if you introduce a solution with more layers of security than you need you will often be compromising your ability to produce value.

If you store PII or are subject to standards like SOC, HIPAA, PCI, etc., then please make sure that any solutions you consider support your needs for security. If you need an anonymous polling solution though, is it really necessary to double-encrypt all your data at rest and make sure that everybody uses a hardware-based multi-factor authentication system to submit a poll?

How "industry-standard" is the technology?

If everybody in your industry operates with SOAP, first of all, sorry - secondly, does the solution you are considering support SOAP? Standards are important, and an effective way of keeping your stack flexible. Incorporate standards whenever you can, and use software that also incorporates standards.

Is there good documentation in place?

Good documentation hints at maturity, stability, and good support patterns. Plus, you will need documentation to use the software, so ensure that it is as high-quality as possible.

How will you get support?

Some software is only sold by and supported by a single company, other software (especially open source) might have support options from various places. Is the company that builds the software going to be around a while? For an open-source solution, will companies providing support have a reason to keep providing it?

What will the solution cost?

There are a LOT of factors that go into how much a particular technology will cost. It's very common to have your internal costs for personnel to implement, support, and extend the tech outpace the raw licensing costs. The Total Cost of Ownership (TCO) should always be considered, although that can be difficult to calculate (especially in advance).

How does the technology scale?

Some tech, especially tech provided by one of the big cloud companies, effectively scales infinitely. Most modern tech available today has a solution for scaling to the levels needed for almost any company, but there might be costs involved in the future when you need that kind of scale. What will that process look like if/when it comes around? Based on the best predictions available, what are the upper bounds for the type of scale you expect to see?

What does "flexibility" mean for the different technologies available?

Sometimes one technology choice might be a specialized, fine-tuned solution that targets a very specific need, where a different choice might be more akin to a swiss army knife - it can do almost anything, but might not be able to solve a very specific problem as well as the more specialized solution.

How quickly can you hit the ground running?

Is the technology ready out-of-the-box, or are you going to need to do a lot of configuration and dev work just to get it going? If you want to move quickly then this factor will need to be part of your decision-making process.

Committing to your decision

Once you get to the point where you have made a decision, what's next?

Well, you probably just have to pull the trigger. Unless you make a decision without even thinking or being aware of the options available to you, it's very unlikely that your choice will be completely wrong in a way that can damage the company. If your software has been architected for flexibility, it may be a project to switch to something better in the future, but it won't break you and you can be much more confident in the new solution.

There can be a tendency to choose flashy, new, trendy technologies. Many of those trendy technologies will fall off your list if you have gone through a selection process such as is outlined above, so make sure to be methodical in picking something that actually meets the needs of the system. For the greatest flexibility, sticking with boring tech will almost always pay off.

Involve the right people in the decision-making process; make sure you understand what problem the technology needs to solve and what the possibilities are; optimize for flexibility; be ready to commit to something knowing that it may need to eventually be replaced - and implement with that in mind.

You won't make the perfect decision every time, but you can certainly expect to be successful with your initiatives overall.