Data Science Careers: What Is the Best Option in Data Science?
Our fearless leader and CEO Sam Levan recently spoke at Galvanize in San Francisco about data science careers. A common student question is, “is a job at a startup or a big company better for data scientists”?At MadKudu we're in a good position to answer it - the 5 of us have more than 25 years of experience in data science. We’ve built everything from the world's largest fraud detection system to quick hacks in Google sheets. So what is the best career option for an aspiring data scientist? Google or HotNewStartupWithCuteBlueDoggyMascot.com?Before answering I’m going to share a secret about being a data scientist. Whether you work at a 2-person startup or CapitalOne, there is one attribute that best predicts your probability of success. The super-duper-secret to being a great data scientist is ...... wait for it ......... wait for it ............... WAIT FOR IT!!! .........
Data science is a SOCIAL skill
That’s it. That’s the big secret. Your career as a data scientist will be defined by how well you can communicate, write, listen, organize, lead, and empathize. Do you need hard skills? Of course. You can’t do the job if you don’t know how to use R, Python, or MatLab. You have to know how to measure the statistical significance of your results. But unless you fancy yourself the next Will Hunting, being a brilliant Python coder won’t make you any more effective than a good one if you can’t work well with others.
Data scientists don’t work alone
Data scientists try to solve business problems with data - an iterative activity that requires working with cross-functional teams. Suppose you're a knowledge engineer working at a bank. On any given day you will need to:
- Talk to regulators about the riskiest type of criminal activity.
- Help analysts understand customer data and what behavior the bank can track.
- Ask (beg?) the operations to pull new data sources for you.
- Testify in court.
The most effective data scientists are team players who make everyone else more effective. If you want to be a lone hero data science isn’t for you.
Communication beats code
Every morning I read Nate Silver’s analysis on FiveThirtyEight. Is Nate the world’s greatest statistician? Of course not. Nate Silver's brilliance is his ability to help us understand how data answers important questions in politics, sports, and life. We try to deliver the same value in our work. This is - by far - our most popular blog post. Why was it featured on Growth Hackers, Hacker News, and Growth Hacking digest? It wasn't a great study - just 9 sample companies. We didn't build any amazing models - everything was done in Google Sheets. As an example of data science, it is ... meh. Our customers loved it because we helped them understand what the data means and how they can use it to solve business problems. Any idiot can have an opinion. Lots of smart people can compile numbers. Few people can help others understand why data matters and what they should do - be one of them and you've got the world at your feet.
Data science careers: Startups vs Bigco
Back to the students’ questions: “What is the difference between being a data scientist at a startup vs a Bigco?”
Specialist vs generalist
The biggest difference between being a data scientist at bigco vs a startup is your degree of specialization. At a bigco, you have the opportunity to work for months and years on the same problem. Are you excited about spending 2 years creating the world’s greatest recommendation engine for Facebook? Do you like doing primary research? Becoming an expert in building models to solve 1 problem? Becoming a master in R, Python, or MatLab?That’s life in a bigco. I know Knowledge Engineers who spend a career detecting violations of stock market wash sale rules. Startups? Hahaha! You won’t know what you’re working on next week, much less next year. You’ll be spending your time helping marketing, sales, and product teams answer basic questions. Since you’re constrained by date and time you’ll do much of your work in spreadsheets or SQL. It isn’t uncommon for a data scientist at a startup to be juggling 5 different problems at the same time. Your expertise will be your ability to quickly acquire and apply new skills - fortunately, this is a great skill to have.
Support system
Unless you work at MadKudu, you may be the only data scientist at your startup. Your colleagues may not understand what you do or how you can help them. You may have to define your objectives. On your first day, you might be told to “go help the sales team find the best leads”. Does this terrify or excite you? At bigco, you will have a support system. Your boss will tell you which project you’re working on. More experienced data scientists can answer your questions. Have a problem? Ask your boss - that’s what she’s for.
Getting dirty
Data science textbook examples are fairy tales. In 20 years I’ve never encountered such simple problems. In the real world:
- Simply getting the data is HARD.
- People don’t agree on what columns mean.
- Everything changes while you’re doing analysis.
Bigcos have teams of people to help solve these problems: server-side developers to populate the data warehouse and business analysts who write data dictionaries. At startups … well … it is probably up to you. The developers are all too busy finishing the next release and supporting customers to run SQL queries. You have to look in the code to see how the product generates the account_activated event in Mixpanel.
Risk
At bigco, you’ll have a nice salary, 401(k) plan, and benefits. You’ll work a little harder the 3 months before bonus time so you can get that new car. It feels safe - but is it? Life at a startup is the opposite. Part of your compensation will potential, unknown upside from stock options. Will you have a job next quarter? It depends on whether the CEO can close the B round. It feels risky - but is it? I’ve worked for the world's biggest, most stable employer and at any-day-we’re-dead-lets-start-stealing-office-supplies startups. I’ve had friends lose $500K starting a company and others struggle for years to find a job after being laid off. Here is how I think about risk. Working for startups is very risky in the short run but incredibly stable in the long run. The stability of bigcos comes at a price - you develop fewer skills, build fewer relationships, and don’t get regular experience marketing yourself. This risk is particularly true for a data scientist who can get stuck working on the same problem ... with the same tools… and the same people … for years. A major industry downturn can be economically devastating when all companies in a sector are laying off employees. Both bigco and startup careers have risks - you just need to understand the risks you’re taking and be smart about managing them.
What’s best for you - bigco or a startup?
After reading this post you’re probably more confused than ever - because there is no one answer. My #1 piece of advice is to go out an interview with big and large companies. Meet the teams and ask lots of questions. What is the #1 problem you would be solving? Why? Who else is on your team? What do they say about the problem? What tools would you be using? It’s the only way you’re going to see what is best for you. Best of all it will allow you to work on those social and communication skills - the most critical ones for your career in data science. Photo credit: goingfar.org