By Nisha Lewis, Antonia Berlingeri, and Heather MacArthur, Feb 7, 2024
Randomized Controlled Trials (RCTs) are considered one of the most credible approaches to assessing an intervention’s effectiveness, due to their ability to examine cause-effect relationships between an intervention and its outcomes. For workforce development programs, RCTs can show the measurable effect a program has on participant employment outcomes (i.e. employment status, income) – data that is increasingly desired by funders.
RCTs function by randomly assigning participants into one of two groups:
- A program group who can enroll in the program or,
- A comparison group, who receives no treatment or receives a program alternative (i.e. a modified version or another similar intervention).
The random assignment of qualified applicants helps to ensure that the only difference between groups is whether they participated in the program, reducing the likelihood that other factors could be responsible for any differences in outcomes.
While RCTs are highly desirable in providing rigorous evidence of program impact, denying people either full or partial access to a program that could benefit their career can be a difficult pill to swallow for stakeholders who believe in its benefits. This means that ensuring an ethical RCT design and gaining buy-in for the study are both critical elements of successful RCT implementation.
In this article, we focus on why an RCT is an important evaluation method and how to gain buy-in, particularly from program staff, when conducting an RCT. While the focus is on workforce development programs targeting underserved groups, using NPower Canada’s RCT as a case study, the learnings will be applicable to other initiatives aimed at producing social good.
Case study: NPower Canada’s RCT
NPower Canada is a non-profit workforce development organization that provides free digital skills training and employment programs to connect underserved jobseekers to new and rewarding career opportunities. In 2021, NPower Canada received funding from the Future Skills Centre (FSC) and partnered with Blueprint, a non-profit research organization, to learn how well NPower Canada’s programs are working.
As part of the Scaling Up Skills Development portfolio, FSC and Blueprint help organizations scale their programs to new geographic regions and/or population groups. Due to their unique ability to demonstrate a program’s causal impact, RCTs are a useful tool in supporting a program’s scaling journey. Thus, NPower Canada embarked on an RCT in 2022 to assess the effectiveness of its workforce development program. Eligible applicants were randomly assigned to a program or comparison group. Comparison group members were prevented from re-applying to the program for two years, to ensure robust longer-term comparison.
Gaining staff buy-in
Given the need to randomize some applicants out of program participation, an RCT is likely to be perceived as challenging by staff who believe in the work and seek to help as many underserved individuals as possible. An RCT also poses reputational risks; the organization may be scrutinized by partners, potential participants and the public for straying from its mission statement, which is particularly concerning for non-profits that rely on funding from other organizations to deliver programming. Staff need to believe in the project to confidently assuage the concerns of stakeholders and mitigate risks.
To gain staff buy-in and protect the most vulnerable program applicants, non-profit leaders and researchers wanting to undertake an RCT can prioritize flexibility in research design, detailed training and information-sharing to mitigate concerns.
Flexibility in research design
Before launching an RCT, researchers should engage with an organization’s staff to understand their concerns, particularly about denying vulnerable applicants access to the program, and to identify ways that these concerns can be addressed through study design.
In NPower Canada’s case, consultations with Blueprint led to three adjustments to study design. First, the RCT is only being conducted in provinces where there are many more applicants than seats in the program (Ontario and Alberta).
Second, instead of assigning 50% of applicants to the comparison group (as is typical), the design allowed for only 20% to be assigned to comparison. This is the average percentage of people who apply to the program over and above the number of available seats. Thus, no additional applicants are turned away from the program than would normally be the case.
Lastly, the design allowed for the most vulnerable groups – including Indigenous applicants and applicants receiving government disability benefits – to be excluded from randomization and offered direct access to the program. While excluding these groups from randomization means they will also be excluded from RCT data analysis, such compromises are helpful for gaining staff buy-in and ensuring that vulnerable groups continue to access the program.
Detailed training
In addition to the ethical considerations, RCTs also disrupt business as usual for the implementing organization. Training and resources need to be provided to help staff learn new processes, which may include changes to participant communication, participant tracking and data sharing.
Given that conducting an RCT can pose a reputational risk, difficult conversations are a given. For staff in stakeholder-facing roles (e.g. outreach and admissions) to feel comfortable, they need appropriate tools for discussing the research with applicants and partners. In addition to holding all-staff meetings and tailored information sessions, NPower Canada provided staff with FAQs and key messages to assist them in communicating about the RCT.
Information sharing
Information sharing and transparency are essential to gain staff buy-in for an RCT. Staff need to understand the RCT’s function, believe in its long-term benefits and accept that it may be uncomfortable in the short-term. To ensure transparency throughout the organization, researchers can host all-staff meetings before project launch to share general information about the RCT, help staff understand why it is important, how it will affect their roles and responsibilities, and to answer questions.
Tailored information sessions for each department can also help by speaking specifically to the aspects of the RCT that will affect each team. These sessions can also help researchers understand what is most important to staff, and thus reciprocally inform the research design.
Conclusion
Randomized Controlled Trials are a valuable method for assessing the true impact of interventions on participant outcomes and are highly sought after by funders and policymakers. However, RCTs present challenges that merit careful consideration, particularly the potential need to deny disadvantaged populations from accessing a service that is intended to serve them.
Through NPower Canada’s case study, we have found that flexibility in research design, detailed training and information sharing are key to successfully gaining staff buy-in to implement an RCT. These lessons can serve as a guide for any organization seeking to evaluate and enhance their interventions, especially those aimed at underserved populations.
