NCS's Leslie Wood on Monitoring Campaigns Using AI Causality

High on the wish list of most marketers is improving the measurement of campaign effectiveness. Fortunately, Leslie Wood, NCS's chief research officer, has been on a mission to perfect it. She has accomplished this with a combination of machine learning methodologies that interface with a campaign in flight, thus enabling marketers to modify and improve their ad placements in real time. This initiative, called Sales Lift Metrics, sheds light on the causal impact of specific campaign tactics by identifying the key sales drivers that amplify incremental sales for CPG brands.

"We have created a Super Learner, which includes multiple different machine learning methodologies, like 'random forest' and 'gradient boost,'" Wood said. "And when we execute the Super Learner, it runs all of those different models and sees which ones are winners — often up to three or four of them — and combines them into an ensemble model." It is apparently flexible enough to apply to any type of CPG advertiser and is constantly refining its application through the use of AI.

How It All Began
The idea and the efforts behind this initiative started about five years ago and, over that span of time, the methodology has evolved through testing and innovation. "When we first started, it was just television because you can't have a control if you have such high reach. After all, who is not exposed? People who were not exposed to the advertising look very different than TV viewers. They are very different kinds of people," she explained.

Wood discovered that this approach took the pressure off the researcher to constantly oversee the model. "Here was this machine learning method that allowed us to put this in places where data was siloed. We could set it up in 'free rooms,' places where there was no researcher," she said, adding that this allowed a door to open where Google, Facebook, and Apple could manage their proprietary datasets without a human interacting or overseeing it. "It is meant for computers to talk to computers."

In-Flight Optimization
Five years ago, the hardware just wasn't there to be able to apply in-flight optimization. But now, this can be executed on what Wood calls, "sales-lift metrics," which provides the ability to "use these machine learnings to quickly, in real time, deliver key performance indicators to tell you what about your campaign is working." It is now possible to compare multiple creative options, consumer targets, or sets of tactics for sales effectiveness and incremental causality.
"We are looking at the incremental value of each of them," she said, "and compare them" in a weekly cumulative sales report. An advertiser can make changes to the campaign as it rolls out and see the impact.

What Sets the Super Learner Apart?
"There is always a full set of statistics" within the Super Learner, Wood noted. This is unusual because "AI almost never has a full suite of statistics. Statistics were developed by professors on small data sets. For big data sets, we develop machine learning and AI methods and those were built by computer programmers. So, we don't usually have confidence intervals, significance tests, or standard error — those measures that you need to say that this is a good model. We built Super Learner, which does all of these steps."

In this way, as new machine learning methods are developed (and, according to Wood, they are being developed all the time), they are added to the system to see if there are new ways to improve campaign efficiency. "It allows us to continually improve without reprogramming or recoding," she said, adding that certain models never worked and others that worked too hard and delivered unreasonable answers. "We removed those."

Among the successful campaign indicators is incremental sales per household, which is a key performance metric. It also reports impressions delivery by target and by creative.

Going Forward
Since the Super Learner has the ability to integrate faster systems and new models, it advances every day. Thousands of new schedules have been run and rerun "to see what works and what doesn't work, what's the minimum threshold, what's feasible, what are indicators inside the statistics that let us know that it's not working."

Wood is committed to keeping up with "this fast-moving world" of data and machine learning with constant monitoring and asking why results change. In the fall, they will be adding more data, more brands, and a price/promotion feature.

And, according to Wood, this entire initiative has been a model for others. "We use the h2o machine learning language; they will tell you that we were the first to create a Super Learner and they followed suit and built one themselves," she said. "We have pushed the envelope on machine learning."

 This article first appeared in www.MediaVillage.com