Predict Customer Propensity with Machine Learning for Efficient Marketing
“Propensity: an inclination or natural tendency to behave in a particular way”
Modeling at a bra company
How likely is any particular person, on any given day, to log in to a website and make a purchase? Do her past interactions with the site predict this? Can this likelihood, or propensity, be quantified? Most importantly: can this quantity be used to market to her more efficiently?
ThirdLove is an ecommerce company without a brick-and-mortar store; most customer interactions start through its website. One of the benefits of having most interactions online is that a rich dataset consequently emerges. This data can be used to build predictive models which in turn enhance and personalize the customer experience.
We use machine learning for many applications at ThirdLove, for instance to recommend products and sizes. We recently used our arsenal of models to predict prospective customer behavior and tested this model on a real-life marketing campaign, the topic of this post.
Setting the scene for modeling
Every day, women come to ThirdLove’s website and leave without making a purchase. What can we learn about these women? Is she waiting for her paycheck to come in, is she shopping other sites to make comparisons, or is she browsing the web with no intent to purchase at all? I often find myself ogling shoes online that I cannot afford and have no intent to buy. Do these women fit this category of shopper?
Of course, we’ll never be able to know the answers to all these questions, but we can start by understanding a prospective customer’s propensity to make a purchase by building a model. Then we can use the model in combination with experimental marketing campaign data to understand where we can really deliver value with direct marketing, one of the levers we have when it comes to reaching out to customers.
Open call search: finding the best fitting model
The information that we have on a prospective customer at any given time can be separated into these categories:
- Web behavior: How long has it been since she last touched the website? How is she engaging with the website?
- Email engagement: Is she engaging with ThirdLove’s emails?
- Sizing info: Does she report having fit or size issues?
We want to build a model to answer the question: “will this customer make a purchase in the next 30 days?” This can be done using a “snapshot approach” where the model is trained on a “snapshot” of time, for instance, Aug 1, 2018. The features that we train on mirror the information that we have access to in real time when we are trying to make a future prediction.
Is one side bigger than the other? The balancing act
Any algorithm that supports binary classification can be used to build this model. We used a Random Forest due to its robustness and ability to capture non-linearity. We have two classes of data to train on for the Random Forest model:
- prospective customers who made a purchase 30 days after the snapshot date, and
- prospective customers who did not.
Most people don’t buy a bra every day, so the two classes are significantly unbalanced. A binary classifier that is trying to maximize accuracy will simply predict that no one will ever purchase, because that's the easy and greedy way for it to be right most often. We need to help it along by nudging it to give more weight/importance to those rare samples in the training dataset where a customer does make a purchase.
If our model is working well, we should see propensity scores in our test set that skew toward one for the “purchase” class and scores that skew toward zero for the “non-purchase” class. When training on the data holding the original class ratio, scores for both classes skew toward zero. We can balance the classes in the training data by undersampling the “non-purchase” class - and voilà! We see the score distribution that we expect to see for the two different classes.
What this bra service holds up – or holds out
At the end of the day, the purpose of this model is to market more efficiently, which in turn will produce a better customer experience and drive higher ROI on our marketing or outreach efforts.
To find out if the model is actually useful to us, we can leverage A/B tests done for our previous marketing campaigns. Specifically, we can analyze “holdout tests” where we withheld marketing from a randomized control group to find the correlation between our propensity model scores and the incremental orders (orders which were obtained through advertising and not “self-selection”). The magic number to find is a propensity score value that provides a cutoff between a loss on how much we spend on targeted ads and gaining a profit.
What we find (shown in the figure above) is that there is a huge correlation between the cost of incremental orders and propensity score! In fact, marketing to customers with the lowest propensity scores is costing over 20 times per incremental order more than those with the highest scores.
This is very encouraging, but let’s not stop here, because the bar chart above is just itching to be fit with a decaying exponential distribution to smooth out statistical fluctuations and squeeze out an extra order of precision. The function that best describes the data is a decaying exponential on top of a negatively sloping line. The resulting fitted function hugs the curves of the model quite beautifully (this is the last bra-inspired innuendo) and can be used to separate losses from profit by cutting off propensity scores where the function crosses the average order value.
Utilizing the model
There we have it! We can use the propensity-to-convert model to rank prospective customers by their likelihood to make a purchase. Marketers can use this information to hit a certain cost per incremental order from a campaign or reach a target number of conversions most efficiently. By weeding out customers who are unlikely to make a purchase and redirect marketing spend to a more efficient channel, we have the potential to significantly boost profit margins. That is a lot of power for one tiny model.