Written by Tim Kreienkamp, Director Measurement & Data Science at Funnel
At Funnel, we’re always pushing the boundaries of what’s possible in marketing measurement by leveraging AI and machine learning technologies. Our goal? To help businesses efficiently evaluate their marketing strategies and optimize budget allocations.
To enable modern measurement we focus on three major AI-driven pillars:
While each of these methods are useful in isolation, they work best when combined into a holistic measurement system. While multi-touch attribution is often solved using classification algorithms and tests can be analyzed using various frameworks such as Metas GeoLift, Media Mix Modeling, it demands more sophisticated AI/ML models to overcome the typical challenges and adhere to certain requirements.
Media Mix Models are typically used to:
In order to fit a realistic and “useful” model, at Funnel we cannot simply leverage any black-box ML algorithm, but rather need to conform to more explicit functional forms. Specifically, we need to build models that can assess and address the carry-over, shape effects and ongoing trends of the media response functions. Other than that, we need to assign constraints - media should not have a negative effect, and pricing is not always a positive effect (usually, hi Gucci, Apple…). Models also need to address changing media efficiency over time. So what’s needed is a modeling solution that allows us to assign constraints, model adstock, and/or carryover effects to also assess the delayed media impact over time, the saturation, or shape, of the media effect.
In addition to these there are other models which mainly relate to the various challenges and biases we need to address when trying to derive causal conclusions from observational data. For an in-depth discussion of these, refer to an excellent paper published by Google called “Challenges and Opportunities in Media Mix Modeling”.
Advanced AI-driven Media Mix Modeling can greatly increase the accuracy of MMMs by calibrating it with external results and benchmarks. These can - ideally - come from incrementality tests. Other sources on the measurement stack - like MTA - can be useful too. Even platform-reported attributions might be useful as a reasonable upper bound.
At Funnel we develop and maintain our own custom Media Mix Modeling packages that includes models based on different types of “base” models, including Ubers Orbit, Numpyro and even scikit-learn. While certain frameworks have certain advantages and disadvantages, we don’t want to be limited to any one solution, but always leverage the model - or combination of models - that is best suited to the dataset / modeling task at hand. Additionally, we wanted to include intuitive calibration capabilities, by minimizing the distance between a “ground” or "external" source of truth. This is where the Anyscale Platform built on Ray and Ray’s multi-objective hyperparameter search algorithms provide an advantage.
Anyscale and Ray remove several traditional AI/ML limitations; first it ensures we are not limited by any one algorithms or packages limitation, we can fit a) any parameter we want at scale, and b) include any secondary objective (next to model fit) to enable advanced calibration. With Anyscale and Ray we can find the pareto-optimal combination of hyperparameters (such as adstock, saturation, or specific hyperpriors) that optimizes the tradeoff between “machine learning accuracy” and “attribution accuracy”. This enables us to get closer to the true causal impact of media investments and helps our clients make better marketing decisions.
With the Anyscale Platform now also easily scale our models without requiring complex, costly infrastructure, iterate fast, run experiments and unify the developer experience from development to moving workloads to production. Given our otherwise complex decision surface, and our complex algorithms that can otherwise take several minutes to fit, we needed a solution that enabled us to run as many iterations as possible in a reasonable “wall clock” time. The Anyscale Platform and unique advantages of its capabilities and underlying Ray platform, provided us the perfect Platform to meet our AI/ML needs.
This is how our Ray / Anyscale Workflow currently looks like:
At each iteration, we try a different combination of hyperparameters (such as adstock, saturation-related parameters, which model to use, what (hyper-) priors to set…). Then we fit the model, calculate predictions and attributions, record the respective ML KPIs and secondary objectives (such as distance from the model’s attributions to an incrementality test). Then the meta-learner provided by Ray suggests a new set of parameters based on the success of the previous trial.
With over 10,000 iterations to run, and iterations taking on average 60 seconds. We can now empower our clients with real-time data to make better marketing decisions. The ability to continuously run Media Mix Modeling allows for always-on optimization and quicker access to critical insights.
At Funnel, we’re proud to be advancing the future of marketing measurement, constantly refining our approach to ensure that businesses can make smarter, faster decisions with greater confidence.