We caught up with Douglas Mason, data scientist and CEO of Koyote Science, where he is building machine learning models to predict COVID-19 outbreaks. He freely shared his wisdom and lessons he has learned from over a decade of data science work, ranging from his PhD at Harvard to his time at large companies such as Pinterest, Twitter, and AWS (Amazon).
Douglas’s background
Douglas took a unique path to becoming a data scientist. Although computers were always part of his household growing up, he thought they were boring, and he told his family he would “never study computer science.”
Instead, Douglas went to USC to study filmmaking, thinking he’d follow his dream of becoming a film director. Soon, he realized that filmmaking school wasn’t all he’d imagined it would be, and he took up classical guitar instead. From there, he accidentally discovered a passion for theoretical physics, which he found fascinating (and which paid more than guitar playing).
Not long after, Douglas discovered his interest in data science and climbed the ranks until he was heading engineering and data science teams at Twitter, Pinterest, and AWS. He describes working in the field as feeling like he’s living in a science fiction film.
“When I work on that kind of stuff, I feel like Doctor Strange — as if I’m in the Multiverse. You actually get to live this Rick and Morty parallel-universe life.”
But Douglas wasn’t content with using his expertise to improve revenue at large corporations, so he went on to found his own business, Koyote Science. He’s currently focused on building COVID-19 models to predict outbreaks.
Lessons learned from shipping machine learning projects
Douglas’s success hasn’t come without some hard-earned lessons. He told us about some of the challenges he’s seen across many of the teams and projects he’s worked on.
Lesson 1: Use machine learning to work with users instead of taking over
At Twitter, Douglas worked on a feature called “who to follow.” This gives Twitter users personalized recommendations about which accounts might be interesting for them. As a data scientist, Douglas discovered that people used this feature a lot. At first, it seemed great — people were following nearly everyone it recommended. But in the longer term, people who used this feature visited Twitter less.
Their feeds were filled with tweets chosen by an algorithm, rather than tweets from people they chose themselves — and there were just too many of them.
By reducing the number of “who to follow” recommendations, Douglas improved long-term engagement.
It’s common knowledge that long-term and short-term goals often conflict, but Douglas discovered a deeper lesson here. As machine learning solutions become more capable, it’s often tempting to use them to do too much. This is almost always a mistake. Douglas says:
“I aim to build products that work with the user rather than trying to take over from the user.”
AI as depicted in science fiction — with human-level intelligence — is probably to blame for the fact that many people try to do too much with machine learning. In many cases, it’s best used to augment human actions rather than replace them.
Lesson 2: Data pipelines and good engineering are more important than math and algorithms
People get very excited about new machine learning algorithms. First we had neural networks (NNs), then convolutional neural networks (CNNs), then generative adversarial networks (GANs), transformers, and more. Algorithms are fun and exciting to talk about and explore.
But Douglas, a self-confessed math nerd, has learned that the math and algorithms tend to get far too much attention, while real success comes from good data, good engineering, focusing on the customer’s problem, and not getting trapped in the math. He says:
“It's very, very rare for the algorithm to make the difference. It's almost always the data pipeline. In my work, I have been able to reduce errors by 90% with a data pipeline, compared to 75% with a better algorithm. And yet everyone wants to talk to me about the algorithm, but no one wants to talk about the data pipeline.”
We use metaphors that associate machine learning algorithms with neuroscientists and data pipelines with plumbing, so it’s not surprising which one grabs popular attention. Douglas found success by focusing on the less glamorous aspects of machine learning. In most cases, deciding what data to use and how to present it to the algorithm is more important than the algorithm itself.
Focus on the customer’s goals
Many “AI startups” today talk far more about the solutions they provide than the problems they solve, and Douglas has learned to maintain a laser focus on customer goals. Sometimes this means pulling himself away from the more enticing theoretical aspects of machine learning. He says:
“As a mathematician, I love all the nuances of the math, and easily get lost in it. But the reality is that there's an infinite amount of math out there to learn. It's not feasible to lock myself in my room and learn all the math before I focus on customer goals.”
The truth that many data scientists don’t want to hear is that successful machine learning solutions are not usually about creating something new, powerful, and exciting. More often, seeing problems from the correct angle and using tried and tested approaches is what you need.
Work with and learn from experienced engineers
Douglas has personally engineered many successful machine learning solutions and led teams of software engineers, but he remains modest about his engineering ability and emphasizes the importance of solid engineering.
“At Amazon, I let the engineers do as much as possible, because they're better than me at engineering. I would love to give you another answer, but they're efficient, they're thoughtful, they've seen these structures before, so they know about implementation details.”
It’s not all smooth sailing though. Douglas acknowledges the difficulties of getting different experts to work with each other, especially when highly technical people tend to have very strong opinions about tiny decisions.
The best way he’s found to get everyone on the same page is by constantly releasing Minimal Viable Products (MVPs), which takes us to our next lesson.
Lesson 3: Always build Minimum Viable Products (MVPs)
Douglas swears by MVPs, which demonstrate core pieces of a solution, even if many of the features are missing. When developing a machine learning solution, he’ll aim to deliver a new MVP every week.
He uses these to:
- Avoid traps: If a project is taking too long, the difficulty of building even an MVP can be used to argue that the project should be cut early, before years of effort are wasted. Douglas says:
“If something ends up being way harder and I keep doing MVPs and never reach the goals, then that gives us information about the difficulty of what we're attempting to do.”
- Communicate: Both technical and non-technical people tend to better understand things they can see and use, rather than abstract ideas.
“People's response to an abstract concept of something is often completely different to their response when they see something real. That's why I'm always putting out MVPs. People who are looking from a higher-level perspective can gain the required intuition to give me feedback.”
It’s better to have to trash two weeks of work than two months, and MVPs can help with this.
MVPs have other benefits too. By releasing stripped-down versions of a solution, Douglas often discovers that less is more.
“What you end up delivering is often much simpler than the thing you originally intended to do, but it's refined.”
Of course, customers are sometimes unhappy when it turns out that the best solution was the simplest one. Douglas compares building machine learning solutions to creating art: it’s about the time that went into development, not the effort required for the final product.
“There’s a classic Zen story about a king who hires an artist. The artist works for a year, but then paints the final painting in only three seconds. When the king complains, the artist says, ‘Oh, I spent a year trying to paint much harder things.’”
MVPs keep you open to finding a better, simpler solution, even late in the development process, and it’s important to stay agile so you can pivot to these better solutions if necessary.
People often think something has to be complicated in order to be powerful, but in fact the opposite is often true.
Lesson 4: Control and precision are more important than size and power
Large machine learning models, such as GPT-3, are exciting and often make their way into headline news. But Douglas compares large models to early (failed) attempts to build planes. These planes competed against the famous, successful plane built by the Wright Brothers. What made them different? The Wright Brothers focused on control, while their competitors were going for size and power.
“What the Wright brothers did that was so ingenious was that they didn’t go for bigger engines. They were bicycle mechanics. They didn't even use powerful engines. And instead, what they focused on was control.”
This is similar to machine learning models. As Douglas says:
“We made the biggest model that does all this stuff. But then people ask, ‘How do I interpret this stuff?’ ‘How do I control it?’ ‘How do I make sure that my models don't go off the rails?’”
Large machine learning models might often be more powerful, but unless they solve real problems, they’re not useful. If a model produces amazing results unpredictably and only some of the time, that’s not useful. If a model produces accurate results but we don’t understand why and can’t be sure the results will always be accurate, then that’s also not useful.
Instead, smaller, simpler, and arguably less powerful models that offer more interpretability and consistency are more valuable in nearly every case. Just like with flying, we need to be able to steer and to land, not just to go fast.
Shipping machine learning projects successfully
Douglas shared many lessons in our chat with him, but these were his most important rules for successfully shipping machine learning solutions:
- Use machine learning to work with users: don’t overstep and try to take over from them.
- Focus more on the problem, data, and engineering than on math and novel algorithms.
- Focus on being in control of the solution you produce, rather than making it as ambitious and as large as possible.
- Build MVPs and ship smaller pieces regularly. Even if they don’t have all the features, tight feedback loops are even more important here than they are in software engineering.
We (Data Revenue) have shipped dozens of successful projects. If you need help with yours, don’t hesitate to get in touch.
