Data Science & Online Retail - At Warby Parker and Beyond: Carl Anderson Interview
We recently caught up with Carl Anderson, Director of Data Science at Warby Parker (and previously at One Kings Lane). We were keen to learn more about his background, his perspective on how data science is shaping the online retail landscape and what he is working on now at Warby Parker...
Hi Carl, firstly thank you for the interview. Let's start with your background and how you became interested in working with data...
Q - What is your 30 second bio?
A - I have a fairly diverse background. I grew up in the UK. I have a B.Sc. in Biology, M.Sc. in Biological Computation, and a Ph.D. in Mathematical Biology from a Probability and Statistics department. I spent a few years doing postdocs at Duke, Georgia Tech, and in Europe before becoming a faculty member at Georgia Tech. I left academia to do consulting in complex systems and agent-based modeling. After that there were some short stints doing lossless data compression algorithms and working for the Sunlight Foundation. I spent 4 years as a senior scientist building analytical systems for our large-scale models at Archimedes in San Francisco, before a stint as a data scientist at One Kings Lane. I'm currently the Director of Data Science at Warby Parker in New York.
Q - How did you get interested in working with data?
A - Like many data scientists, I have a classical pure science background and data are the lifeblood of the empirical scientific method. If you want to be a scientist, you have to care about data.
Q - Was there a specific "aha" moment when you realized the power of data?
A - As a biological sciences undergrad, I remember being fascinated by some classes and labs in paleoclimatology. I loved the idea that using a simple non-specialized microscope one could count and measure tree ring growth over hundreds, and with Bristlecone Pines, thousands, of years. With similar evidence from ice cores (which spans longer time periods) and shells of tiny Foraminifera animals, a person can combine these data proxies to provide a compelling reconstruction of ancient climates. In a sense, all of this evidence is just sitting around us waiting to be used (just like the current big data hype today). I loved that idea.
The other key moment for me was in my Master's degree in Biological Computation. This unique masters degree took biologists and taught them to do three things: computer programming, mathematical modeling, and statistics. This gave me a whole new suite of tools in my toolbox and helped me realize that I could come up with hypotheses and run computer simulations and generate data de novo and not be limited by sample size in the way that the biologist empiricists were. This led me to do a lot of agent-based models on badgers (Masters) and with ants, bees, wasps (Ph.D.), later with humans, information packets and so on (consulting).
Very interesting background and insights - thanks for sharing! Let's change gears and talk more about Data Science in Online Retail…
Q - What attracted you to the intersection of data/data science and online retail?
A - When I left scientific consulting to join One Kings Lane (a successful home décor flash sales site), I was interested in working somewhere that was consumer facing. I had always been interested in style and design and the fact that the company dealt with 4000 new products, 60% of which had never been sold on the site, each and every morning was a real draw from a data perspective. Likewise, Warby Parker designs and makes beautiful products that customers love. Both of these companies represented interesting challenges.
To be honest, I am fairly agnostic about the context and generator of data (which in my background has been as diverse as ants, robots, and battleships) and am drawn to interesting intellectual problems. How do we deal with the cold start problem of 4000 new products? How do we generate a holistic view of our customers as they interact across our channels of mobile, web, school buses, and physical retail stores? These are all fun challenges.
Q - Which online retail companies do you consider pace-setters in terms of their use of data science? What in particular are they doing to distinguish themselves from peers?
A - To state the obvious, Amazon has been a leader for many years now. Their site is littered with different recommenders and they are sitting on a mountain of data. Their UI and experience can be kind of clunky though. That is, the site can expose weird recommendations that can be kind of jarring: people who bought X happened to buy something completely random and seemingly unrelated Y. The best data science, though, will be invisible. It will just work (more on that below).
Birchbox (with Liz Crawford as CTO) is doing some great work. Nordstrom's data lab has done an impressive job putting together a new team, selling the power of data science internally, and getting proven better-than-human recommenders in front of customers.
Q - Which companies/industries do you think online retail should emulate with respect to their use of data science? Why?
A - Netflix drove the field forward with the Netflix prize. However, they continue to innovate with A/B testing and measuring everything, (see http://www.slideshare.net/justinbasilico/learning-a-personalized-homepage). Interestingly, they have a very strong focus on driving the re-subscription rate (i.e. getting people to keep paying a monthly fee), akin to maximizing the lifetime value of the customer. This is a very downstream metric that rolls up all of the user's experiences but it also means that they know how much they can spend to acquire and keep customers, (see https://blog.kissmetrics.com/how-netflix-measures-you/).
I'm also a really big fan of LinkedIn's data science team. One thing that can happen is that data scientists can try to be too clever and develop an unnecessarily complex algorithm to infer customer's intent, taste etc. They forget that one can simply ask the customer directly. Does Joe Doe know about project management? We don't know for sure so let's ask their contacts. While the system can be gamed (same as Yelp reviews and any other user generated content), it provides a simple --- and in LinkedIn's case, a very large --- direct source of data. Online retailers are a little too reticent to ask customers what they want. What would you like to see in our next flash sale? If you could choose the color of the next widget, what would you choose? Get the data and let product managers and data scientists sift through it.
Q - Where can data science create most value in online retail?
A - There is the old cliché about the right thing at the right place at the right time....One Kings Lane was always trying to find the right balance between getting products that the data science team and OKL buying team believed individual customers would love versus serendipity, letting them explore and chancing upon something new, different and unexpected. This is a really delicate balance and the oft-cited example of Target knowing a teenage girl was pregnant before the girl's father, fell to the wrong side of that line. Great data science will help companies understand their customers better, not just from a historical context but grasp the current context --- what do they really want or need right now? --- so that the data products will be viewed as a boon, a digital assistant that will aid the customer.
For instance, Google Now is viewed by many as helpful and not creepy as it provides you the train or bus times or weather at your current location when you need them. This, however, is not true of most online retail experiences where the cross sell and upsell is crude and in your face. Data science needs to disappear from the customer's perspective. Jack Dorsey said it better:
"I think the best technologies -- and Twitter is included in this – disappear... They fade into the background, and they're relevant when you want to use them, and they get out of the way when you don't.".
Data science in online retail is not restricted to customer facing websites and services though. My team works with just about all teams in the company and has potential to improve operations and decision making across the board from supply chain, finance, merchandising, and marketing. I reviewed the range of these contributions in a recent blog post and in the past ten days have been working with business owners in all of these different areas. Particular examples we are working on right now include performing unsupervised clustering and decision-tree based classifiers of customers to identify signals early on in the relationship that might identify high lifetime value customers. We are also iterating on our tools and models for predictive sales forecasting at the sku level.
Really interesting ... fascinating to see how data science can have an impact across retail functions; not just the consumer facing parts of the business model. On that note, let's talk more about your work at Warby Parker...
Q - Firstly, what are the major similarities and/or differences between how data science is being applied at Warby Parker relative to One Kings Lane?
A - When I arrived at One Kings Lane, the data engineering, warehousing, and analytics were pretty much all in place. The data were, for the most part, there ready to use; of course, it is never as clean as you might think or want. At Warby Parker, in contrast, all of that was missing and it has been my team's responsibility to create that, to get the data in place before we start to make use of it. We have just turned that corner and with a core set of data in place, we are now shifting our focus to building models and data products, as well as provide a robust set of reporting and business intelligence tools to support our analysts..
Q - What are the biggest areas of opportunity/questions you want to tackle at Warby Parker?
A - Much of the last year has been spent putting our data infrastructure in place. Getting data into databases, creating a single source of truth, and creating an accurate catalog. Now that we have that, this year is going to be very different. We are going to focus a lot more on understanding the customer. Who are they and what makes them tick? What's the relationship between online and offline (we plan to open more physical retail stores in the coming years) experiences? For this, we will marry sources such as clickstream, transactional history, in-store analytics and social media.
Q - What projects have you been working on this year, and why/how are they interesting to you?
A - Two weeks ago, we sent our Home Try-On recommender out for A/B testing (our Home Try-On program allows a customer to order five frames, have them shipped home to try-on and can then send the box back to us, all free of charge.) Unlike most e-Commerce sites our basket size is very small. Customers wouldn't normally purchase a pair of glasses frequently, as they would groceries. However, our HTO program ships boxes of five frames to customers. This is a really great dataset because it is reasonably large and you can look at the covariances among the five frames plus what they subsequently purchased and build a recommender based on basket analysis. We hope that the tool will help customers better choose frames that they'll want to purchase.
Q - What has been the most surprising insight/development you have found?
A - Warby Parker sells a monocle and it has an extremely high conversion rate. Most people who order this in their Home Try-On boxes end up purchasing it. Conversion is so high that we had to tweak our basket analysis algorithm specifically to account for it.
Q - That's very surprising! :) ... Last thing on Warby Parker … About a year ago you wrote a very interesting blog piece on "How to create a data-driven organization" and how you planned to do so at Warby Parker … a year on, what do you think is working well?
A - Great question. I have been working on a follow up "one year on" post that will appear in Warby Parker's tech blog very soon. (As such, I'll keep these responses short.) ... We have got much better at evaluating what the tech team should be working on. Different business owners essentially have to compete for software developer (agile team) time and have to quantify the return on investment. The underlying assumptions for both the return and investment have to be clear, justifiable, and the metrics must be comparable across proposals. That way, all managers (who vote on all the competing initiatives) are able to view what different teams are interested in and what will drive the business forward.
Q - Has it proven harder than you imagined?
A - Getting analysts across the company to knuckle down and learn statistics. With a Ph.D. from a statistics department, I am very biased, but a sound basis in statistics seems to be an essential tool of any analyst. Like many skills, statistics may not feel useful and relevant until a specific project comes along and the analyst needs it --- for instance, when we need to A/B a change in the website or optimize email subject lines and send times.
Q - Anything you wouldn't repeat if you could start over?
A - There is nothing that we've done that I've completely regretted and wouldn't do. There are many things, however, that I would do better the second time around. These range from getting business intelligence tools in place, running our analyst guild, and productionizing our systems (according to my boss, things don't exist until they are in production).
Carl, thanks so much for all the insights - really interesting to get such a detailed feel for the work and culture you are developing at Warby Parker! Finally, it is time to look to the future and share some advice...
Q - What excites you most about recent developments in Data Science?
A - Without doubt, deep learning. With my biologist's background, I have always had an interest in cognition and Artificial Intelligence. However, it has never delivered. When I read Jeff Hawkins' On Intelligence, a few years back, it blew my mind. Here was a biologically plausible hierarchical model of the neo-cortex and which represented a generic learning model. Numenta's (Hawkins' company around this) model at the time was in its infancy. It has since improved significantly but I think has been leapfrogged by the exciting deep-learning work by Hinton et al. Here we have a relatively simple hierarchical model (stacked auto-encoders, which if you squint, arguably look similar to Hawkins' model) that is proven to work: it really can classify cats autonomously, it can win kaggle competitions without any domain knowledge etc.
Q - What does the future of Data Science look like?
A - I think that it is very bright indeed. Computational power is only going to get bigger and faster, algorithms (such as deep learning) will become easier to train and use, and hopefully tooling will be easier for the average non-technical user to harness machine learning.
I do wonder about the term data science though. It may well disappear over time as sub disciplines become more established. If you look at the history of science, rich gentlemen were not scientists but "natural philosophers" and members of the Royal Society would attend readings of papers that spanned the gamut of what is today known as biology, physics, chemistry, and mathematics. Differentiation into those disciplines only happened relatively recently (1800s) and as those fields grew and became more established, we defined sub-disciplines such as oncology and later pediatric oncology. Data science as this overly broad umbrella term is still akin to natural philosophy.
Q - That makes sense - in that context, any words of wisdom for Data Science students or practitioners starting out?
A - Just do it. There is no substitute for getting your feet wet and working on things. These days, there is no shortage of data (even the government has finally caught on: https://www.data.gov), free online courses, books, meetups of like-minded individuals and open source projects. Reading a book is one thing but getting real data and having to deal with missing data, outliers, encoding issues, reformatting, i.e. general data munging, which really can constitute 80% of time of a data science project are the kind of dues that you must pay to build the suite of skills to be a data scientist. Kaggle has some great introductory 101 competitions and http://scikit-learn.org/ has some great documentation that can help get you started.
Carl - Thank you so much for your time! Really enjoyed learning more about your background, your perspective on how data science is shaping the online retail landscape and what you are working on now at Warby Parker. Carl's blog can be found online at http://www.p-value.info and he is on twitter @LeapingLlamas.
Readers, thanks for joining us!
P.S.If you enjoyed this interview and want to learn more about
- what it takes to become a data scientist
- what skills do I need
- what type of work is currently being done in the field
then check out Data Scientists at Work - a collection of 16 interviews with some the world's most influential and innovative data scientists, who each address all the above and more! :)