Data Versus Conventional Wisdom: Lessons from a Recent Patient Recruitment Campaign

Shortly after I completed my master’s in social work, I met with a statistician as part of an interview for a research assistant position. Giddy with nervous energy, and fresh out of social work school, at one point I blurted out a question along the lines of asking him how he felt about statistics. (It’s true. I did.) He answered: Statistics is the stuff of life.

The statistician’s answer stayed with me over the years. I pictured mathematicians making decisions from the clear-eyed vantage point of probabilities: daily life as odds ratio. Pay more for organic produce? Decide which of your kids gets to go to college? All questions of calculation. While I have been working with data daily for a number of years now, I am still awed by how many questions can be answered by a little thinking and some simple math.

As part of a recent recruitment effort for a clinical trial involving patients who had significant pain, we implemented a strategy of distributing TV advertising throughout the day in order to reach a greater breadth of viewers. As a result, we aired TV ads as early as 5 to 8 AM. A few study coordinators at the study sites warned that this would be a waste of advertising dollars, as their own local efforts with early morning advertising had not brought in study patients. Prospective study patients with significant pain were unlikely to be watching television in the early morning hours, they argued, both because they were less likely to work 9-5 and because they were less likely to sleep comfortably at night (and therefore be up early in the morning).

After the study had successfully completed enrollment, our resident data guru at CAHG and I followed up on the sites’ initial concerns about early morning advertising with some data analysis. We first established that the response to TV ads tended to be immediate; local call activity spiked at the call center in the hour or so after a TV ad aired in that market. Using the study-wide average cost per TV referral, we then estimated the number of referrals we could expect in each market based on the size of the TV buy (e.g., the total spent). Finally, we compared this estimate to the actual number of referrals each market received. We found that the difference between the actual and expected number of referrals was essentially random. There was natural fluctuation in actual vs. expected referrals in each market, but no trend relative to the amount spent.

In other words, patients with significant pain were as responsive to TV ads in the early morning hours as they were the rest of the day.

Equipped with this data, we were able to let the study sponsor know that early morning advertising had been worth the investment for this study, and could safely be added to our arsenal for the next phase of clinical research for that drug.

I continue to be amazed by the extent to which logic and a little math can transform decision-making, particularly in the often hazy world of advertising. If the power of data-based decision-making can make a believer out of someone who once planned to practice social work for a living, it must have some value!