The Importance of Post-Study Recruitment Reporting and Analyses

There is a common trend in clinical research patient recruitment to tout the next big thing or widget that will most certainly help a sponsor fully enroll their next study on-time and budget. While it is certainly worthwhile to try and find new approaches to improve enrollment rates, I have found that it is prudent to pay more attention to basic principles in crafting a cohesive recruitment strategy for your trial.

While many studies and vendors at least pay lip service to the practice of collecting metrics during a recruitment campaign, the number of studies where this data is properly analyzed is relatively small. The rationale for this assertion is certainly anecdotal, but plausible in part because of the number of times that I have seen a sponsor or CRO communicate actionable intelligence from a previous study (maybe three times in ten years).

There are several steps you can take to ensure that your study is set up to produce actionable lessons learned at the end of enrollment:

      Design a data capture infrastructure to collect key metrics in advance of study launch

This seems intuitive enough, but often recruitment metrics are not planned far enough in advance to accommodate any special considerations for a specific study.  If you are relying solely on site-based reporting to measure the success of any initiatives, then it is vital to try and standardize your metrics to some degree up front. Otherwise, you may be stuck trying to aggregate varying levels of data between your sites.

For instance, some sites will precisely record every minor detail in the patient engagement process on a spreadsheet or other database, from first contact, the numbers of times they have followed up with the contact, to any appointment no-shows. Some sites, on the other hand, will be less exact with how they record this information. It is important to make sure you know what metrics you want to capture and communicate the expectations to all sites before FPFV.

Even if you are employing a CRO or patient recruitment vendor to capture data for a centralized campaign, it is still a good idea to finalize what they will be measuring and how in advance of study launch. These organizations often have template systems in place that capture the same metrics for every campaign they support. Be sure to ask about this up front and ensure that their metrics platforms meet your needs.

Ensure that reporting begins early and often in the recruitment campaign

It is a good idea to set the standards for reporting early before the study begins, as well. Most studies have regular check-in meetings between sponsor and vendor and/or sites. Use these discussions as an opportunity to create a reporting interval where current recruitment data can be shared and analyzed by the team. These forums can also be a valuable way to dig deeper into the data by generating regular reports that will allow you to make course corrections to enrollment strategy, if needed.

There are a number of ways to accomplish this within the confines of a study. It all depends on what your data requirements are. Perhaps you want to keep close tabs on cost per referral or screening. Or maybe you are curious about the relationship between site communication (eBlast, phone call, etc.) and screening activity. These are all things that can be measured and reported on throughout enrollment if planned for.

Don’t be afraid to call more frequent meetings or create reports outside of the normal schedule if it will benefit enrollment by prompting timely strategy decision-making. Your enrollment campaign should not be created in a vacuum. Use metrics and reporting to inform how you spend your recruitment dollars.

Insist on a robust final analysis meeting

On more than one occasion, I have encountered objections from key study leaders about the need or benefit of a final analysis meeting.

But why have one in the first place?

Even if recruitment goes exceedingly well for a study, any final analysis should be able to determine some lessons learned from a recruitment campaign, if the right metrics were captured from the beginning. Whether you are a project manager at a sponsor, CRO, site or recruitment vendor, the end of enrollment for any study should include a thorough review of every action that took place in trying to recruit patients to the trial, in an effort to improve those efforts in future campaigns.

Invariably, there will be naysayers who might decry the need for a long meeting to review reports and discuss lessons learned. Ignore these detractors and insist on conducting a thorough final analysis meeting. It will be time well-spent.

Deliver a thorough after-action report, including any key learnings from the final analysis meeting

At the end of any meeting, it is supremely important to recap the discussion and highlight any pertinent action items that were flagged. Once all key learnings are catalogued, a final report should be compiled, including any and all data and takeaways, for distribution to the entire team. If you are talking about a Phase II trial, the report should detail how these learnings can be applied to the next trial. If there are no further studies to leverage, then you should go the extra mile to apply the lessons learned to a therapeutic area or similar type of study where appropriate.

Paying close attention to metrics and reporting is not quite as sexy and cutting edge as touting the latest new trends in patient recruitment. And yet, all too often, data is overlooked or not emphasized enough when developing strategy for completing enrollment on time. If used effectively, reporting can be a way to build rapport and engage the entire study team that will benefit your entire clinical operations infrastructure.

How to Deal With Inconsistent IRB Requirements When Designing Recruitment Materials

One thing I like about working in patient recruitment for clinical trials is that no two trials are the same. Every study brings new challenges to overcome, resulting in new learning opportunities. One challenge that can be particularly daunting is drafting study recruitment materials for multiple institutional review boards (IRBs), or the independent ethics committees that approve the use of materials in clinical trials. Normally, inconsistent requirements happen when you are dealing with a central IRB and at least one local IRB. Local IRBs, functioning in support of academic institutions, generally have differing standards when it comes to messaging, terminology and tone. In most cases, any differences are very minor and attributed to simple word choices (think clinical trial vs. clinical study).

This can be an issue when you are drafting centralized materials or advertisements designed to support multiple sites in a research study. How do you standardize your copy so that it can be approved by all regulatory bodies? Following are a few hints on how to avoid IRB issues that can stall production of key recruitment materials.

1) Write any copy like it is being reviewed by the most conservative local IRB

The most important consideration when designing materials that will be reviewed by multiple IRBs is to ensure you do not take too many risks with your writing. Edgy copy might be inspiring to the reader, but could wind up passing through a central IRB while being scrutinized by a local one. While not a universal truth, it is usually the local IRB that will push back aggressively on any language that might be too speculative or promissory.

This is truly the worst-case scenario, because as we often find, academic centers traditionally are activated after dedicated clinical trial sites during study start-up. In that case you may have already had your materials approved by a central IRB and produced (which, in the case of a TV ad can be a significant investment), only to find that a local IRB has a problem with some of the wording. It is much better to write your copy to the most conservative review you can think of so that this is never an issue.

Ultimately, here are a few general tenets to adhere to when drafting copy for patient recruitment ads or materials:

• Do not use the word free when describing anything associated with the study
• Avoid phrases that speculate at all about the potential benefit or efficacy of a study medication
• Be mindful of key words in your copy that could be scrutinized, and limit their use (including clinical trial, study drug, etc.)
• Use limited reference to compensation for participation, including the amount of money someone could be reimbursed for travel (as this may vary by site)

2) Make your materials format as flexible as possible

It pays to have materials that can be easily reformatted to accommodate variable content, should you run into a situation where IRB standards conflict. This is not always easy, especially when it comes to broadcast ads. A good rule of thumb here is to always avoid live action commercials where you have actors with speaking parts. I know that very few clinical recruitment campaigns employ such a sophisticated level of ad, but the cost of reshooting a live action ad would most certainly be prohibitive for a sponsor. Even when using lower cost mediums (print ads, flyers, etc.), it is smart to make sure your creative files can be easily edited by a third-party designer, should you need to have something reformatted on the fly.

3) Communicate with your IRB if there are discrepancies

IRBs can sometimes seem like intractable, soulless bodies that rule with heavy-handed authority. However, if you find an IRB is mandating an edit that contradicts something another body has approved, it might be worthwhile to reach out to the reviewer and discuss the recommendation. Showing an IRB that another IRB has approved copy they are not comfortable with can sometimes result in a change in the ruling.

I recently ran across a situation where two different reviewers at the same IRB had different opinions about the same phrase used in a study flyer. One reviewer approved it, while a few months later, a different reviewer recommended a change. So, even within the same IRB, sometimes you can find variable standards. Ultimately, it never hurts to inquire about a specific decision, especially if not questioning it would result in having to reproduce materials at a significant cost to a sponsor.

Variable decisions by IRBs are something that everyone runs into at some point or another. Poor planning for unexpected deviations in approvals are an easy way to incur unexpected costs in for a study. Make sure you plan accordingly so that any surprise IRB issues don’t derail your recruitment budget or timeline.

Test driving a media campaign

In many situations, you are given the opportunity to test something before you commit.  Most people want to test drive a few cars before they settle on a final decision. Gyms often offer 2 weeks free to test the gym before signing a contract. This gives you an opportunity to evaluate the situation before you fully commit and ensure your money is being well spent.

In the clinical trial world, we always jump on any opportunity to test an initiative before launching a full program. One of the most valuable tests we conduct are pilot media campaigns.  A pilot media campaign can provide valuable information, helping to formulate a comprehensive media strategy.

There are some key reasons why running a pilot campaign can be beneficial:

1. The world of media is always changing. An advertising schedule that worked 3 years ago for a particular indication may not be the best fit for a current study.  New sources continue to sprout up and other sources may lose their impact so it is important to evaluate your current options as well as what has worked in the past.

1. Advertising budgets can vary. It is important to consider the past budget to provide the appropriate context. The site advertising budgets provided by the sponsor are generally not very generous and usually differ from an ad buy developed as part of a larger campaign. While site feedback is helpful, it is important to weigh all information when developing a media strategy.

1. Site responsiveness can be evaluated.  A pilot campaign can not only gauge the response we receive from the media but also it can test how sites are able to handle those referrals. Based on the site activity, you can determine an ideal media buying level and schedule.

A pilot campaign may demonstrate that advertising is not the answer

In a recent project with a small patient population, enrollment was lagging behind sponsor timelines.  The CRO and several sites insisted a radio campaign would accelerate enrollment.  We were hesitant to place a radio campaign because we were not confident broadcast advertising was the right fit for the target population.  Before launching a large campaign, we recommended a pilot campaign.  We placed radio advertising in 3 markets of varying size.

Though the pilot campaign yielded a decent response, only generated 2 screened patients.  Most of the respondents from the campaign clearly did not qualify but still inquired about participation. Ultimately, the high cost per screened patient confirmed that broadcast media did not provide the best ROI for this study population.  The pilot campaign allowed us to present data to the client and saved them the money they would have spent on a full-scale campaign.

A pilot campaign can help you develop a cost-effective comprehensive media strategy

A few years ago, we were engaged in a clinical trial where enrollment was slower than anticipated due to slow site start-up and challenging inclusion and exclusion criteria.  CAHG proposed a media campaign but the sponsor and most of the sites had no experience advertising for this specific indication.  We recommended a pilot campaign in 5 markets testing print, radio and television advertising. 

The pilot campaign provided us some key information:

• Television by far was the most cost effective medium over radio and print
• Sites became overwhelmed with too many referrals in a short amount of time, so we knew we needed to adjust future placements to match the site intake capabilities
• A question on the prescreen questionnaire at the call center and website was being misunderstood, so we proposed a small revision resulting in more referrals

Based on our analysis of the pilot campaign, we developed a full media strategy that successfully enrolled the trial.  The pilot campaign provided insights for us to make a sound strategy and develop a more cost effective campaign.  The graph below highlights the reduction of the cost per screen from the pilot campaign to the full-scale campaign.

We do not always have the luxury of conducting a pilot campaign for every clinical trial. However, if the budget and timeline allows, it can be an essential step in launching a successful advertising strategy. Next time you are ready to kick-off a patient recruitment media campaign, weigh the options of a pilot campaign.  Testing your campaign will lead to more cost-effective recruitment spending, which will likely result in accelerated enrollment timelines.

Data Versus Conventional Wisdom: Lessons from a Recent Patient Recruitment Campaign

Shortly after I completed my master’s in social work, I met with a statistician as part of an interview for a research assistant position. Giddy with nervous energy, and fresh out of social work school, at one point I blurted out a question along the lines of asking him how he felt about statistics. (It’s true. I did.) He answered: Statistics is the stuff of life.

The statistician’s answer stayed with me over the years. I pictured mathematicians making decisions from the clear-eyed vantage point of probabilities: daily life as odds ratio. Pay more for organic produce? Decide which of your kids gets to go to college? All questions of calculation. While I have been working with data daily for a number of years now, I am still awed by how many questions can be answered by a little thinking and some simple math.

As part of a recent recruitment effort for a clinical trial involving patients who had significant pain, we implemented a strategy of distributing TV advertising throughout the day in order to reach a greater breadth of viewers. As a result, we aired TV ads as early as 5 to 8 AM. A few study coordinators at the study sites warned that this would be a waste of advertising dollars, as their own local efforts with early morning advertising had not brought in study patients. Prospective study patients with significant pain were unlikely to be watching television in the early morning hours, they argued, both because they were less likely to work 9-5 and because they were less likely to sleep comfortably at night (and therefore be up early in the morning).

After the study had successfully completed enrollment, our resident data guru at CAHG and I followed up on the sites’ initial concerns about early morning advertising with some data analysis. We first established that the response to TV ads tended to be immediate; local call activity spiked at the call center in the hour or so after a TV ad aired in that market. Using the study-wide average cost per TV referral, we then estimated the number of referrals we could expect in each market based on the size of the TV buy (e.g., the total spent). Finally, we compared this estimate to the actual number of referrals each market received. We found that the difference between the actual and expected number of referrals was essentially random. There was natural fluctuation in actual vs. expected referrals in each market, but no trend relative to the amount spent.

In other words, patients with significant pain were as responsive to TV ads in the early morning hours as they were the rest of the day.

Equipped with this data, we were able to let the study sponsor know that early morning advertising had been worth the investment for this study, and could safely be added to our arsenal for the next phase of clinical research for that drug.

I continue to be amazed by the extent to which logic and a little math can transform decision-making, particularly in the often hazy world of advertising. If the power of data-based decision-making can make a believer out of someone who once planned to practice social work for a living, it must have some value!

Predicting Referral Conversion in Clinical Trial Advertising

Today we'll cover an issue that we've found to be vexing for a large number of study sponsors: modeling the rate at which pre-qualified potential trial participants will actually consent into the trial.

This issue has implications both for managing advertising budgets and assessing site enrollment performance. In this short video, I'll take you through a couple pitfalls in the standard approach to analyzing referral conversions, and lay out what we believe to be a clearer, fairer, and more reliable approach.

Comments and questions, of course, are welcome.

What Is the Right Mix of Recruitment Tactics for a Study?

As an avid runner, I am constantly obsessing about race times, trying to find ways to get faster even though I am nearing the tail-end of my running “prime”. In spite of professional and family demands, I sometimes think I can still run fast without putting in the training needed to shave minutes or seconds off of my personal best times, from 5K to marathon. However, with years of racing experience under my belt, I know that it just isn’t possible to make significant time gains without the proper work. Unfortunately, too often clinical trial sponsors make the same mistaken assumption when it comes to recruitment planning for their studies.

There continues to be a pervasive belief among sponsors and CROs that every therapeutic area has a predetermined selection of tactics that will help enroll a trial on time. Similarly, those same beliefs hold that you are not qualified to provide successful enrollment support for a study unless you have had previous experience on another similar trial. Instead, the focus should be on the process of developing a strategy that is best suited to maximize ROI and meet enrollment goals on time and on budget.

So, even generally speaking, what is the right mix of tactics to use to efficiently enroll your next study? Since we have established that there are no magic bullets to apply (i.e. print should be used on any patient audience over the age of 60+, etc.) universally across trials, there are some general tenets that should be adhered to when deciding what recruitment initiatives are going to work best for your study.

1)      Know your target audience

More than anything, it is imperative that everything you do, from messaging to engagement medium (media or online advertising, outreach, etc.), should be executed with the target audience in mind. Things like age, gender, ethnicity, and other demographics should be top of mind when developing study materials or advertisements. While there are certainly several other factors to consider, this is the most important in determining your recruitment strategy. Really knowing your target audience is not the same as simply having executed a recruitment campaign for a similar indication. The variables noted above change with every study, and it is critical that you understand them in an effort to know who the patient audience truly is for a given study.

2)      Never go ‘all-in’ with one initiative

Even in the most desperate rescue situations, it is never smart to put all of your eggs in one basket and rely on a single recruitment initiative to enroll your trial. Even in situations where you are confident that one medium (say, TV advertising) is going to provide enough referrals to meet your goal, until you have empirical evidence that an initiative is definitively performing the way you need it to, it does not make sense to spend significant amounts of money on one tactic. It is better strategy to pilot an initiative when possible. When that is not feasible, it still makes sense to stagger the rollout of your tactic in a way that allows you to collect data on its effectiveness before investing large amounts of money that could result in a poor return on investment.

Avoiding an ‘all-in’ strategy does not mean that it makes sense to throw money at a number of different tactics simultaneously. It is still prudent to do your homework and settle on a focused strategy that will efficiently enroll your study. My experience has been that a diversified suite of 2-3 tactics are generally sufficient to accommodate variable site capacity and ensure you are effectively attracting patients to a trial without breaking your budget.

One of the biggest reasons that it is next to impossible to accurately predict the performance of a given initiative is that no matter how successful you are in generating clicks, calls or referrals, unless the sites act on those leads and convert referrals into screenings and enrollments, your tactic may fall short of expectations. Which means you should…

3)      Communicate with your sites

As any seasoned clinical research professional can attest, no two sites are created equal. Some sites have robust recruitment teams that can handle dozens of referrals per day, while there are still some study centers that struggle to triage a dozen leads in a week. This disparity can be frustrating, but some simple upfront work to understand site capabilities along with robust engagement to manage expectations can go a long way in mitigating this issue. Any recruitment tactics you move forward with should be done so with these issues in mind. As an example, don’t run TV ads in a large market where you know you will get a lot of referrals, but have a site that might not be able to handle them.

Reaching out to the sites ahead of a recruitment campaign will not only allow you to determine their capabilities, but will also help you build rapport that can keep your trial top of mind and ultimately result in those sites working harder to enroll your study. You can make the sites feel even more invested by working with them to customize a given tactic in their market. Whether that is taking their feedback on which broadcast outlets to use or local patient advocacy groups to engage, this can go a long way in motivating your sites to go the extra mile for your study.

One potential communication pitfall to be mindful of when talking to sites is that they can sometimes tell you what they think you want to hear. This is where having data about how well your sites are executing on a given recruitment initiative can be very beneficial to give you the full picture, and objectively assess site performance. 

When a site or recruitment vendor says, “We’ve got lots of experience recruiting in this therapeutic area. We know exactly what will work to enroll your study”, you should be wary. No two studies are the same, and ultimately it comes down to carefully developing and executing a plan based on the myriad of variables that exist in any given study. While experience is helpful, it is more important to focus on the process of developing a sound strategy to ensure a successful recruitment campaign. Being mindful of this will help you cross the enrollment finish line of your next clinical trial in record time.