Machine learning teams can use proprietary platforms like Knime or SageMaker, or build their own using open source tools. Companies often sell proprietary platforms as more powerful, more efficient, and easier to use. But in reality, they’re often more complicated and less powerful than open source alternatives.
We build our own MLOps architecture from purely open source foundations. Here’s why ownership of your platform is important.
Open source is higher quality
People are sometimes suspicious of free tools but open source software is often higher quality than paid alternatives. Megacorps like Google and Microsoft rely heavily on open source software and frequently have their best engineers working on these projects.
Open source projects also benefit from having many more eyes on them than closed-source alternatives. If a developer finds a bug, they can contribute a fix that benefits everyone.
Finally, open source software is usually not marketed, meaning incentives are better because quality is the key factor determining success. By contrast, the success of commercial projects can depend far more on marketing, partnerships, and branding. This means companies are often incentivized to spend more time and effort on these areas than engineering.
“You get what you pay for” is therefore a real misnomer when it comes to open source software.
Open source leads to transferable skills, which makes hiring easier
Engineers want to learn open source tools because they can use this knowledge at other companies, too. No one wants to be trapped in their current job because they spent ten years learning the ins-and-outs of some internal platform.
Training your team to use tools valuable to other companies, like Tensorflow and Kubernetes, might make some people nervous, but really it’s the only sustainable way to attract top engineering talent.
Your team can benefit from these transferable skills in other ways, too: it’s far easier to find expert consultants or even simple community-sourced help if you use the same tools as everyone else.
Open source solutions are more modular
Being “all-encompassing” is a key way enterprise platforms try to differentiate themselves. Instead of building only a training platform, or only a deployment tool, they sell themselves as “all-in-one” solutions or “the last tool your team will need”.
But being the best at everything is unrealistic. And the monolithic nature of these solutions makes it harder to swap out individual components. If your team wants to use the best model registry, often they just can’t plug it into their existing platform.
By contrast, open source software is usually more granular and has a strong focus on integrating with other platforms. This means an open source solution is more like playing with lego: if part of it’s giving you trouble, you can simply detach it and switch in an alternative.
There are hidden costs in proprietary platforms
Proprietary platforms are incentivized to constantly sell upgrades and new features to existing customers. In many cases, this means they’re not upfront about their limitations.
During sales, they might convince you that a low or mid-level plan suits your needs perfectly. Only after buying in do you realise they’ve strategically removed key features, or added specific limits designed to cripple your workflow until you pay for the next upgrade.
These platforms can also raise their pricing with next-to-no notice, so your team can either scramble to rewrite everything or pay more than you budgeted.
When does it make sense to use proprietary platforms?
If your team doesn’t have much engineering expertise, especially when it comes to DevOps and creating and managing infrastructure, then setting up and maintaining open source solutions can be challenging. However, we built Open MLOps - a free open source repository that automates the set up of a production-ready suite of MLOps tools.
Open source tools tend to be built “by developers for developers”, so non-technical teams like marketing might also prefer proprietary platforms in some cases.
Finally, if you’re working on small internal projects or proof-of-concept applications, there’s often little need for infrastructure. If people aren’t using your solution as a product, an off-the-shelf tool for something like a quick visualisation is often good enough.
Does your team need help adopting open source MLOps?
We’re a machine learning agency that loves helping research teams set up open source MLOps infrastructure. Contact us any time if you’d like to discuss MLOps tools.
.png){kind=small}
