Building a Lean AI Startup - Part 1
Last April, I had the chance to present at the Data Council conference in San Francisco about how we developed an AI/ML SaaS product while staying lean. In this series of post, I’d like to expand on some of the content I discussed in that talk.
If you’re more of a slides person, you can find them here. If you’d rather have it narrated to you in a lovely French accent, the video of the talk is available here.
Note: in the rest of this series I will use Data Science / Machine Learning / AI interchangeably. There are important differences between them, but for our purpose here, I figured it would make the story easier.
Why being lean?
It’s 2019 and I haven’t started a company since 2014, so I’m not sure if the Lean Startup is still a popular thing. It seems that the word MVP has gotten as tantamount to early-stage startup lingo as “Angel Investor” or “Culture Fit”.
In short, In the early 2010, Steve Blank, Eric Ries et al. popularized this concept. It was then adopted heavily (or developed separately) by startup accelerators like Y Combinator (“Make something people want”). The overall idea is to iterate as fast as you can to product-market fit by going quickly through the Build / Measure / Learn cycle. The underlying message was that most startups would waste too much time building something people didn’t want.
There are certainly other ways to build a startup, but at MadKudu, we've always enjoyed the idea of creating more customer value with less. We also started the company with no funding initially, so proving our ideas quickly was important.
Why is being lean harder in data?
Going through the lean cycle is already pretty hard in normal circumstances. (Reasons vary, but for me personally, the main difficulties were a combination of engineering perfectionism and general fear of confronting my ideas to the outside world).
It turns out it is even harder with data (although probably not as hard as with hardware). Why? You have more steps to go through to test your idea, making the “Minimum” part of MVP harder. Each extra step is a possibility, if not of outright failure, at least of losing precious time to market.
When building an AI product, you need to prove 3 things:
- Your problem is well-suited for AI
a. You can collect the right data
b. You can predict with “minimum accuracy” - There is a market for your models (“model market fit”)
- You’re solving the problem correctly over time.
And you cannot skip any of those steps. For example, I’m sure there is a big potential market for an algorithm predicting the result of horse races (or NFL games), but making it a reality is complicated. Conversely, you can probably predict with correct accuracy what time your cat will need food daily, but it is unclear that you will create a large business around that idea.
The third step might be the less less obvious and the one that is most specific to AI. Early in MadKudu’s life, we looked at predicting churn for SaaS companies. There is a potential market for that problem. It is possible to collect the data. It is also technically feasible to get a sufficiently robust model predicting churn. In that case, what wasn’t promising in our early experiments was the ability for the model results to be useful over time to a Customer Success team. Therefore the problem wasn’t correctly “solved” over time.
Additionally, the first phase has challenges that are inherent to data products. Collecting data, in particular, is a notable Catch-22. You need to have data to start building your algorithm. But to get data, you need to convince someone that you will do something useful with it.
What can I do about it?
As in the originalLean Startup, the idea, is to go as quickly as possible through a shallow but full test of your hypothesis, rather than spend too much time on any single feature.
Last year, the folks at Zetta Venture Partners summarized these steps very concisely in a graph we wished we'd had when we were getting started:
The steps they highlighted match very well what we've experienced first-hand and are as follows:
- Capture Training Data
- Train Model to Minimum Algorithmic Performance (MAP)
- Test Model in Real Life
- Capture Feedback Data
- Measure Customer ROI
- Appropriate packaging
Depending on your background, your natural inclination will be to spend too much time at certain step. If you're background is Data Science, I'll bet that you'll be inclined to spend more time than necessary at the "Train Model" step. If you're on the Data Engineering side of the force like me, maybe your downfall will be spending too much time optimizing your input pipelines. Know which step you're most likely to get stuck on and keep yourself honest.
If you're starting a new project, I would recommend that you map your learnings to this pyramid. If you know where you are at any given progress and report your progress based on what hypothesis you've tested in your narrow slice. (If you are not reporting your process to anyone, I'd recommend you start there (even if it's a weekly meeting to your cat), but that is a different story). Ask yourself the following questions: How long have we spent at that step? What is missing for us to get to the next level of pyramid? Can we get there faster?
In the next entries of this series, we’ll go through some of the specific challenges we encountered at each step and some of the pitfalls we avoided (some by craft, but most by luck). My goal is to share some of the things that worked and didn’t work for us. I hope that this can be helpful to some of you getting started, either with your own startup projects, but also for internal projects within an established company.
Further reading