This study assessed whether women and minorities are discriminated against in the early stages of the recruitment process for senior positions in the Australian Public Service (APS). It also tested the impact of implementing a ‘blind’ or de-identified approach to reviewing candidates.
Over 2,100 public servants from 15 agencies participated in the trial. They completed an exercise in which they shortlisted applicants for a hypothetical senior role in their agency. Participants were randomly assigned to receive application materials for candidates in standard form or in de-identified form (with information about candidate gender, race and ethnicity removed).
Overall, the results indicate the need for caution when moving towards ’blind’ recruitment processes in the APS, as de-identification may frustrate efforts aimed at promoting diversity.
Authors: Professor Michael J. Hiscox, Tara Oliver, Michael Ridgway, Lilia Arcos-Holzinger, Alastair Warren and Andrea Willis.
Pre-registration information
Women are under-represented in management and executive level positions across the private and public sectors. In 2016, women comprised 59.0% of the APS, but accounted for 48.9% of its executive level officers and only 42.9% of its Senior Executive Service (SES) officers. These statistics may reflect gender discrimination in hiring and promotion processes as a result of unconscious cognitive biases that affect decision-making.
Addressing the gender imbalance across the APS is the key priority of the Australian Public Service Gender Equality Strategy 2016-19. Aimed at driving high performance and boosting productivity the strategy calls for the APS to reflect contemporary reality and states that the APS must embrace diversity and that it should benefit from people of all backgrounds.
Study
This study aimed to test the magnitude of gender and ethnic minority bias in APS shortlisting processes, and the impact of introducing a gender/ethnicity-blind approach to reviewing job applications for shortlisting purposes.
The study was a randomised controlled trial conducted in partnership with 15 APS agencies. Participants were drawn from senior and executive level officers in these agencies. To identify any effects participants were asked to complete a fictitious shortlisting exercise with 16 fictitious CVs. The aim was to see whether de-identifying a CV (by removing a candidate’s name and personal information) changed the way it was assessed.
Results
The results showed that overall, de-identifying applications at the shortlisting stage does not appear to assist in promoting diversity within the APS in hiring. Overall, APS officers discriminated in favour of female and minority candidates.
Can going blind help us make us less bias in recruitment decisions? This is not a new idea, to focus on what really matters, we hide the things that don’t.
Blinding ourselves from information that isn’t critical, helps protect us from our own unconscious biases. So we can make a decision without being influenced by irrelevant factors.
A curtain between musicians in a jury helps improve the chances that women are selected for the American Symphony Orchestra, suggesting an unconscious bias towards men. Reality TV uses a similar concept for people auditioning for a popular singing competition, by blinding the judges to their appearance.
Behavioural science tells us that we’re biased, that we get overwhelmed with large amounts of information, and that we use shortcuts to make quick decisions. So, can ‘going blind’ actually help?
We wanted to test it out, to see what works for recruitment in the Australian Public Service. We invited over 2,000 senior managers from 14 different Australian Public Service agencies to participate in a study. We randomised participants into three groups and asked them to review CVs and shortlist the best five candidates.
The first group saw the CVs including the names of each candidate. The second group saw the exact same CVs – but with one important difference: each name was substituted with a name of the opposite gender. For example, the CV for Mike Harris in group one, was Mary Harris in group two. The third group of reviewers saw the exact same CVs but instead of names, they were simply labelled CV1, CV2, and so on.
We then looked at the data to see how each CV was rated, and assess any differences between the three experimental groups. We found that blinding a CV resulted in a different short listing assessment. Making a CV blind actually decreased the chances that a female candidate would be shortlisted. Our results showed that people were actually favouring female candidates and rating these CVs more highly than the same CVs with male names.
We also tested whether ‘going blind’ could help improve diversity in our workplaces. We used the same principle and included ethnic minority names on some CVs, so that we could see if there was bias towards or against these candidates.
The results showed that most minority candidates were more likely to be shortlisted when their names were identified on their CVs, compared to when they were ‘blind’. These results surprised some people. It confirms that it is important to check how people actually behave, compared to how you think they will behave.
The overall take-home message is that agencies should proceed with caution before introducing blind recruitment. We thought ‘blinding’ CVs would help solve the problem of female participation and the diversity in senior levels of our workforce. If we hadn’t tested it, we would have probably held on to this belief.
There is more work that needs to be done to address gender balance and diversity in our workplaces. If bias isn’t being introduced during the shortlisting stage of recruitment, where else could this be happening? We’re still biased (sorry!), but now we know that rather than putting all the focus on bias in the initial reviewing of job applications, it may be more valuable to direct our attention to the other stages of recruitment, where bias may play a larger role.
Behavioural economics is all about testing assumptions and finding out what works. That’s what we do at BETA.
Intervention start and end date
Monday, 14 November 2016 to Tuesday, 22 November 2016
BETA ethics pre-registration number
BETA ETH 2016 – 001, 21 October 2016
Experimental design
The design of this trial is an individually randomised controlled trial. Specifically, the trial is a framed field experiment where participants will complete a task that is normal for employees at their levels – assessing candidates for an APS job vacancy. Participants have been informed that in this instance that the exercise is hypothetical.
Individuals who agree to participate in the study are randomly assigned to either the treatment or one of two control conditions. Pre-randomisation stratification is based upon agency, age, gender, and APS level and role type, conditional on the number of participants being large enough to stratify based on these characteristics.
Data is collected from participants at two points. Firstly, when candidates register to participate for the trial (24-31 October), and secondly when they complete the exercise (14-22 November).
Intervention(s)
The intervention (treatment) in this trial involves removing the names from CVs. By de-identifying CVs in this way, any indication of gender and ethnicity is removed.
Women are under-represented in management and executive-level positions in the private sector and in many areas of the Australian Public Service (APS). This may be a result of gender discrimination in hiring and promotion decisions. De-identifying CVs may offer an effective way to mitigate biases in hiring and promotion decisions.
In the 1970s and 1980s American symphony orchestras attempted to overcome biases in hiring by introducing a screen during auditions to conceal the identity of the musician from the jury evaluating the performance. In a well-known study analysing data on auditions and hiring by orchestras over this period, Goldin & Rouse (2000) found that the use of blind auditions had a major impact on gender bias in orchestras, increasing the likelihood of female musicians being selected by 25-40%.
A small number of public and private sector organisations in Europe have adopted anonymous job application procedures and several studies suggested that de-identification could significantly reduce bias. There have been no similar studies conducted in Australia and the APS to determine the effect of de-identifying CVs on shortlisting outcomes.
Control condition
In the control condition, participants will review the CVs in the usual identified form (i.e.: with names listed on each CV). There will be two control groups, the only difference being that each CV will have the name swapped to a similar name of the opposite gender (e.g.: Jane Smith’s CV in control group 1 will be John Smith’s CV in control group 2).
Having these two control groups will allow us to fully control for all other characteristics of the CVs while altering only gender (i.e.: the name in the CVs), and provides another level of assurance that applicant characteristics are balanced across gender.
Outcome(s)
Primary outcomes of interest are:
- gender discrimination effects
- the effect of CV de-identification on gender discrimination
- ethnic discrimination effects.
Secondary outcomes of interest include:
- A range of demographic characteristics and other variables that previous research has identified as being related to bias. These include (but are not limited to):
- Demographic characteristics: age, gender, APS classification level, role type/function.
- Other variables: how efficient the process was, confidence in shortlisted candidates, intensity of current workload.
Expected sample size
2,500 adults
Other
Hiscox, Michael. 2016. Unconscious Bias in Hiring in the Australian Public Service: Evidence from a Framed Field Experiment in Shortlisting of Job Candidates. AEA RCT Registry. November 22.