Cash Back Program

Set up experimentation framework

In order to know whether CCB was profitable, we had to have a reliable way of measuring impact. We implemented a core experiment, where a subset of the population would earn cash back on every vertical on which it was available (treatment) and the rest would earn no cash back (control). This would allow us to quantify the impact of cash back on key metrics.

Within the treatment variant, we could then experiment with different variants for each vertical through a series of A/B and multivariate tests.

I set up a detailed SQL-powered dashboard that tracked core success metrics by install cohort over time: unique conversion, repeat bookings, revenue per transaction, profit per transaction, ROI.

Scale to other verticals

My team revamped the cash back UI of the flights funnel, built CCB for hotels, and collaborated with the cars and homes verticals to introduce cash back for their products. This entailed influencing other PMs to prioritize the loyalty work by sharing impact data and resources with them.

Optimize the cash back percentages

In theory, the more cash back you give users, the more they’ll convert, but there are diminishing returns to this strategy, and you are constrained by the company’s margins.

For each vertical, I had to figure out the optimal cash back percentage to balance the benefit against the cost. I conducted competitor analysis to understand what percentages users were earning from other OTAs. I built simple models to estimate the ROI of different percentages based on the company’s forecast margins. Based on that, I designed experiments that tested the ROI impact of different cash back variants (e.g., 2% vs. 3% vs. 5% on hotels). This allowed me to choose the optimal cash back percentage for each vertical.

These experiments, along with UX-focused work (see the Wallet project) helped generate meaningful top line impact. The program was showing a lot of progress in terms of incentivizing incremental bookings (up to 15% incremental bookings per cohort), and analysis based on company margin forecasts suggested the program was reaching profitability. However, there were several headwinds:

• Other teams introduced incentive programs, which competed for user attention and profitability.

• Hopper’s hotel margins declined dramatically due to external and internal forces, which put pressure on ROI.

Profitability was a moving target, and it became imperative to experiment with the cost lever.

Reduce cost

Everyone who booked on Hopper earned cash back. However, survey and data analysis indicated that not everyone considered CCB an important reason to book on Hopper - they would book on Hopper with or without CCB. We wanted to focus on accelerating the impact on bookings from users whose behavior was altered by CCB, while minimizing costs from users whose behavior wasn’t.

We simultaneously ran two experiments:

• CCB Opt-in: Ask users to enroll in the CCB program in order to start earning cash back. Rewards liabilities would only be created for users who explicitly expressed their interest in joining the program. Notably, we wanted to test decoupling account creation from program enrollment - users who had already created an account would still need to explicitly tap a “Join” CTA to enroll. The benefit of this approach was that it could maximize the intentionality of earning cash back. After all, there were multiple areas in the app where users could be nudged to create an account (e.g., sign up to unlock free first-time booking offers), and we wanted to understand who wanted cash back specifically. On the flip side, we would create additional friction for users.

• CCB Claiming: Allow all users to earn cash back, but ask them to claim their earned cash back in order to be able to use it. Potential rewards liabilities would continue to be created for all users, but these liabilities would only be realized for users who chose to claim their rewards. This approach tested a more gamified experience where users would potentially experience the satisfaction of claiming their rewards every time they visited their wallets.

We then compared the metric impact of several cohorts: opt-in, claiming, opt-in + claiming, and control.