Matthew Davies: Welcome everyone my name is Matthew Davies, I'm Chair of the New Zealand Behavioural Insights Community of Practice and I'll be facilitating today's session. It's really exciting to have everyone here from across both Australia and New Zealand, as well as our colleagues from the OECD dialling in from Paris. This is an exciting session because it marks the first jointly hosted event we've had between Behavioural Insights Community of Practices in New Zealand, BICOP and BETA, and we hope this is going to be the first of many opportunities to share behavioural science knowledge and capability across the ditch.
I want to start out with a couple of acknowledgments before we crack into things, I'd like to firstly acknowledge the Indigenous peoples of the lands we join from today, Ngā iwi Māori Tangata Whenua in Aotearoa, New Zealand and Aboriginal and Torres Strait Islander peoples from Australia, and all Indigenous peoples represented here.
As I've noted, we've got many people dialing in from across the world from 7am in Paris through to 5pm in New Zealand and everything in between across Australia. A massive thank you to everyone for accommodating these slightly awkward time differences and especially to our presenters for getting up so early for this session.
The context for this session is about behavioural public administration, behavioural science, has become increasingly applied to help governments design better policy and services for the public, but today's session turns the lens inwards. We'll be hearing about how behavioural insights can be applied to the machinery of government, everything from recruitment through to organisational performance, in other words how governments can function better internally to deliver better outcomes externally.
I'm excited to introduce our speakers today we've got Haris Khan, who's a policy analyst and project lead leading the OECD's work on behavioural public administration. We've got Mia Drazilova, who is a behavioural economics advisor and policy analyst supporting governments to design and test behavioural approaches. Patrick Tehrani, a consultant specialising in behavioural analysis and economic modelling related to administrative burdens and public sector performance.
In terms of the agenda before I hand over to our speakers, we'll have about 30 or so minutes for presenting, then we'll have 15 minutes or so for Q&A and we'll be aiming to wrap up before the hour. If you could just please pop your questions into the chat as we go, hopefully we'll have time to get through them in the Q&A.
Note that this session is going to be recorded, so just be aware and the recording will be shared after the event. Last thing I'll say is if you could just keep your cameras off to support with the bandwidth of the call, that would be greatly appreciated because we've already had a couple of technical challenges so far. Looks like and we'll basically go, I'll hand over to Haris, Mia and Patrick. Thank you.
Haris Khan: Thank you, Matthew. Thank you to everyone who helped organise this call. It's a pleasure to be here, and we're happy to get up early to speak with our Australian and New Zealand colleagues, it's an honour. As Matthew mentioned we plan on talking for a few minutes and then we're hoping for a lively Q&A discussion. We invite you to put your questions in the chat, and then we're happy to answer them at the end of the call.
Before we do that we have kind of three goals in this presentation, one is to introduce the behavioural science at the OECD, which we'll do quite quickly, and then move into what we call the blueprint for behavioural public administration, we'll talk about what that is, when we talk about BPA, what are we talking about? The different applications that are possible with this tool and then we'll also get into a couple of projects that we've done recently on the topic that bring this kind of concept to life. That's the goal before we get into the discussion.
As Matthew mentioned, the team here today is me, Mia and Patrick, we’ll hear from all three of us and we all worked on these projects together. If there's any detailed questions, we're happy to take them on.
We at the OECD's Bureau of Science Unit say that we do three things, we convene, we enable, and we advise. We'll talk about each of those in more detail in a second. But we sit in the OECD's public governance directorate, which is the part of the OECD that's responsible for helping governments conduct their business better and implement reforms and other practical changes to the way that they function in order to make them more efficient, more effective, and more able to deliver for their citizens, our mandate sits within that.
Next let’s talk about the convening function that we do, bringing together over 500 government officials as part of the OECD Behavioural Research and Action International Network. It's an informal OECD network that's meant to be a place for knowledge exchange and peer learning on behavioural science issues around the world. At the end we'll have a little invitation for everyone on the call who would like to join us, we're having a meeting of this network on the 19th and 20th of May in the Netherlands, but also online, if possible, we'd love for you to join, and happy to provide more information at the end.
The second thing that we do is we enable the use of behavioural science with tools and templates, this is kind of classic OECD work. We write guidance documents, guidelines, methodologies, etc. that are intended to help people apply behavioural science to their work in their public administrations. the key ones for today are perhaps the basic toolkit, which is kind of a step by step methodology on how to do a behavioural science project, as well as the recently released BPA framework itself that will be going through in detail. So there is all kinds of tools on these topics and others including; measurement methodologies and ethical frameworks from behavioural science that we published in the last few years
Lastly in terms of our introduction, we do also advise policymakers on how to apply behavioural science in a very practical way, that means doing projects with country officials and you’ll see a summary of some of the projects that we've done recently that are particularly relevant to today's presentation, including a project in Slovakia trying to improve perceptions of corruption. The main BPA project that we've done is with Italy, helping them apply behavioural science to a whole set of internal government functions, and we'll talk about some of those. We also did a project in Greece where we helped apply behavioural science to making the public sector a more attractive employer. Then a big piece of work that we do is on sludge audits and sludge methodologies in general, we'll talk about each of these experiences as a way of animating the framework that we that we have today.
This is who we are, the next topic of behaviour public administration itself and turning the behavioural science toolkit on civil servants themselves, the key take away here is all the same biases, the same framing effects that apply to people in the real world, apply to us as public officials as well, and sometimes more so. I'm sure we can all think of times in our careers and in the public service where we've seen these biases at play. The basic premise of behavioural public administration is that if we are able to understand these biases that exist for policy makers or people who work in the government, and then create policies and tools and interventions to address those biases we're able to make the entire government function better and therefore make it more able to deliver for our citizens.
Next, we’ll talk a little bit about what the difference is between behavioural public administration and behavioural public policy. At a high level, behavioural public policy focuses on citizens or users or customers, people external to the government whose behaviour might be the object of behavioural change projects. For example, increase savings amongst the general population, the goal there is to improve the policy outcome, and the scope goes from the very beginning of the policy cycle, from policy design all the way to the actual implementation of policies.
With BPA, we're talking about using the same tools but using them on civil servants themselves, the goal is to improve public administration processes, this can be done through a variety of different, in a variety of different areas. We can talk about it in HR and policy development itself and the process of policy development. Also things like project management, procurement, IT, and the machinery that makes government function can be done in a more efficient way. The promise is that if we are able to do it more efficiently internally, then we're able to deliver better for citizens externally as well.
So that's what we're here to talk about, next we'll talk about the biggest project that we have on this topic, now I think I'll hand over to Mia.
Mia Drazilova: Thank you, Haris, we already introduced to you in the beginning, there will be few projects that we've done and we want to talk about. We'll start with this Italian one and one of our most recent projects, it was a big project, it was over two years. What we've tried to do here, we've tried to identify to what kinds of opportunities may be here for behavioural insights to really add value to different processes and practices, and we looked at the Italian public administration. We tried to take a behavioural lens, trying to see how the different civil servants operate, and tried to see how we can help them act more effectively, in whatever area it may be, and we prioritise those based on the most available evidence that we have from behavioural science.
A small disclaimer, these opportunities are not limited to Italy. So many governments, based on the findings you can see here, and you'll see in the presentation, you can find value in taking this behavioural lens to the work. And this is also something we've tried with this project in Italy.
Next this is something that Haris already mentioned in his presentation, that we often work with OECD's basic framework, this is a lens that we've applied in this project that we've started. Firstly when we try to think about what opportunities there could be for behavioural public administration, there could be plenty. We tried to follow this framework and see what are the behaviours we want to change? What are some of the barriers or enablers for this change? What could be the strategy to creating behaviour change? And then later, when we've defined this, we designed the interventions to test, whether this strategy that we were following succeeded. Lastly, focusing on the change, we've tried to disseminate the findings and see how these changes could be scaled up. So this is just the operational framework we picked.
Next as I've mentioned, there have been plenty of areas and I mean maybe it's a bit small, but as we've approached it, we've tried to talk to a lot of civil servants in Italy and involve different stakeholders from the government and try to see what are maybe some of the biggest areas where behavioural science could really make a difference. We've outlined a lot of these and then in the end we've considered a range of administrative areas and this is helpful for also for the other project you will hear, but essentially from hiring recruitment to things like performance reviews, also we considered internal communications.
We found that the strongest opportunity in this concrete use case was to focus on the planning process. On the way how the government, Italian public servants plan their processes internally, this is a very Italian specific thing, but to give you context, this relates to their integrated plan of activities and organisations. This is an internal planning Document or a scheme they call PIAO, and why we decided to focus our efforts here mostly because we recognise that there is an opportunity for Behavioural Science to help develop Italian civil servants to develop effective, timely, and transparent planning documents. This was the this was the approach we took for the projects. For the results, I'll hand over to my colleague Patrick to walk through what we've done in the project.
Patrick Tehrani: As Mia mentioned, we did see a lot of opportunity for behavioural science to be applied to this form of internal compliance, where people within the Italian public administration have to submit their PIAOs on a planning portal, at the central level they have to do it by 31st of January and on a local level by 30th of March.
However, many public administrators don't submit it. The challenge was that despite the requirement to publish this planning document on the government's website on the PIAO Portal, many didn't do so before the deadline, this was a classic behavioural problem. Many trials have shown that, you know, reminders can be very effective with increasing behaviour change among citizens, very few studies have gone and actually applied this technique internally. We then ran a randomised control trial with Italian public civil servants to evaluate the effect of reminders on PIAO publication.
We had two groups, a control group and a treatment group. The control group received just the standard reminder that they were receiving normally from the Italian government. Those receiving the treatment, they received 4 behaviourally informed reminders, and I'll unpack what this includes a bit shortly. We wanted to take a look at 2 outcome measures, the first being whether or not the PIAO was published and the second is what date the PIAO was published. We ran this for both central public administrations with a publication deadline of the 31st of January, a different deadline for the March ones, for the local ones.
For the reminders we leveraged behavioural science to motivate public administrators to publish on the PIAO Portal. We can see on the left-hand side of the screen, we have the control reminder, very standard, no clear visual support. For the first treatment, we had four reminders, as I mentioned previously, and each of those were different. The first reminder that we sent to the treatment group very clearly stated the task and the subject, it stated the deadline first, it had less text, it was a bit easier to read. It was attention grabbing with a visual calendar that demonstrated visually the deadline, of course the calendar image was adapted to different people with different deadlines. The purpose of this behavioural redesign was not to provide any new information, they had the same information set, but to reduce friction, increase attention and prompt timely action.
These were the reminders that we used for the second, third and fourth reminder. The second treatment included a step-by-step guide, and it clarified when the PIAO is ready to be published. The third reminder said that the non-publishing is noted and reminded where and how to publish the PIAO. Then the 4th reminder was much more official, reminded why it's required to publish on the portal, and it was signed by the very official department. It added this additional level of authority, this allowed us to compare not just the behavioural reminders versus the standard reminders, but also the impact of different behavioural approaches in each reminder.
Next we'll be turning to some of the results. Naturally, as you would expect, when we send reminders more public administrators published the PIAO on the portal. The control and the first treatment reminder was sent on the 27th of March, three days before the deadline for local public administrators to publish. Both reminders were associated with an increase in the PIAO publications. We see along that x-axis we have the publication date and the darker gray bars are demonstrating the weekends where we naturally expect lower publication rates. Then on the y-axis, we see the publications per day. We notice that upon that first treatment, we do see a somewhat higher spike for the treatment reminder relative to the control, then as we move along the x-axis, we see that those additional treatments, those additional reminders, were also associated with an increase in the PIAOs that were published. However, the first behavioural reminder led to a larger increase in publications than the control group reminder, we can also very clearly see the effect of those subsequent reminders as well.
We find that BPA reminders were more effective than standard reminders, and that this effect lasted over a month. The effect of the two treatments lasted for over two months in our post-treatment measurement period. And often with reminder trials, the treatment causes participants to act, but does not necessarily raise the success rate. They'll look at the time until the behavioural response and that will be lower in the treatment group, but then the overall response of the participant group does not significantly differ. We see that not only did they respond sooner in the behavioural treatment condition, but the overall response rate was much higher. The control group did not close the gap with the treatment group during the data collection period.
What do these findings suggest? These findings suggest that public administrators can benefit from clear deadlines, but also that more research is needed. We put together a few recommendations based on this, the first is that we encourage that, from the findings, internal public administration to simplify and shorten written communications between public administrators and with citizens where possible. To make deadlines more attention grabbing in the reminders. We also know that the step-by-step guide seemed a bit less effective and public administrators may already know how to use the PIAO Portal, they might already know how to comply, but they need more motivation to do that, that should be a broader focus. And the step-by-step guide was also the longest treatment of the separated treatments. I can pass back over to Mia.
Mia Drazilova: Thank you, Patrick. A small note before I go to the framework, essentially the learnings from what Patrick has introduced, they're just one example of an experiment that we've done as part of this project. Another experiment that we outlined in the report, related to public servants in Italy in the same use case, not only doing their planning document, but taking a step back and actually looking into how they're setting their objectives, so for their planning, whether the objectives are aligned with public value, they set these objectives, and has been also been behaviourally informed. This is just to give you an idea and also to motivate you to go and read the report if you're interested.
Moving into what we've done further in this report. We've tried to create a framework for behavioural public administration, and we have tried to, if you look at the slide you can see the visual. We've tried to outline that behavioural public administration, and its practices can be applied across the whole policy cycle. When we've looked into a lot of examples, not just the ones we've done in Italy as part of the project or in Greece, as Haris will be talking about, beyond that, we've identified that these practices are irrelevant across the whole policy cycle. This relates to the early stages, when we talk about the analysis and design, for example, the biases, can really affect how evidence is interpreted or options are weighed, as an example. When we look at the middle stage, the implementation and management, this could relate to being too optimistic in the planning, maybe having some [unintelligible] costs in the thinking, maybe the complexity can also delay the delivery and the different projects. Then when we look at the late stage, it's about monitoring and evaluation, for example, there are different behavioural factors that really influence how evidence is generated and how it is used. When we try to also talk about the different examples, we want to see them as part of the whole cycle.
Next some of the actionable insights for policy design that we've identified, we've tried to identify these for all of the stages of the policy cycle, for example, the goal of these is essentially to guide the policymakers as they do their problem analysis and policy design. The first one could be about reducing messenger effect, for example, when the identity of the messenger is not relevant and maybe anonymising the source of evidence or the source of recommendation. People are focusing on the message, not the messenger, for example, if a leader sends the message it could have an effect “this should be deprioritised or can be prioritised”. The second bit is using evidence effectively, this should guard against confirmation bias, for example, prompting teams to consider the opposite, and then stating some reasons why their view could be wrong in the first place. Other suggestion that we have made in the report, for example, relates to framing risk mindfully. Loss frames can really push the risk seeking, and gain frames can really push the risk aversion. Trying to pick the right frame also when talking internally and doing a policy design or discussions in the teams can really help to potentially match the policy goal that we may have. The last bit is a lot about the diversity of views and considering divided perspectives. Here is the goal and this is related directly to the project that we're having at [inaudible] Italy. I mentioned, we've tried to prompt them to, for example, consider public value as they're planning their objectives and their planning documents. Here it's about engaging citizens early and to counter the curse of knowledge that policy makers might be having and really bringing this lived experience at these insights into [inaudible] at every stage, for example, briefs or any white papers that policy makers might be working on at any stage of the of the process, for example of a project that is not our project, but it comes from the brain, the network that Haris introduced in the beginning.
Next here an example which is called Capacity to Act, this is an example from the Netherlands which illustrates how these findings can be incorporated. How this came about, a court challenges that policy makers sometimes overestimate the behavioural effects of the different policies and might underestimate what difficulties people might face when they have to comply with the different rules and procedures. of course, even though if people know what they have to do, they may act differently due to different factors, emotional, social, contextual, et cetera, different barriers they might be facing. To address this, the Netherlands had a great example to introduce a Capacity to Act test. This is a formalised quality standard in Dutch policy making, it prompts policy makers to, when thinking of a new policy, to analyse the feasibility in context of how people live their daily lives, through various tools like citizen journey mapping, consultation with the target audiences, involving the citizen early, and possible also piloting these measures to test for their behavioural impacts early on.
Next to stress that this framework, I was staying on the outer layer and we have plenty more examples in the paper, if we look more in the internal side of the framework that we've outlined, beyond the policy life cycle, there of course also, so people public administration can be applied to plenty more topics, that are relevant for public administration, such as integrity, for example. People management is something we'll talk about in more detail and internal government processes and some of the barriers that are in internal processes. We believe that really addressing these, so these different behavioural bottlenecks, can further improve also this public sector performance without maybe a need for major reform.
Just one more example and then I'll give over to Haris, for the integrity, one of these areas that we've identified, we recognised here in the paper that Bureau of Public Administration can support the public sector’s integrity. Some of the practices we outline here, for example, leveraging leaders to set and reinforce norms this is an example we've also tested with Slovakia, I'll show you the example in a second, then also things like fostering the identity of policymakers, or applying behavioural insights to employment psychology to help policy makers to recognise, but also report when they see, for example, corruption risks. More practices we've outlined in the paper, just to give you a quick example, we did a few years back with Slovak Republic.
Next the goal here in this project, was to support civil servants in Slovakia to report corruption risk when they see it. As part of this project, we tried to see how these different behavioural elements can inform the design implementation of risk management policies. The approached here, a similar way as in the Italian project, we've applied the basic framework to see what are the barriers at play. We found out that there's a culture of silence, many public servants didn't feel safe to raise corruption risks, there were two main barriers that we found. One was lack of understanding of “what is a corruption risk”. The second is not enough of a leadership of modeling or leadership not going by example, how to prevent and mitigate corruption. Based on this, we've designed a survey experiment which was running with more than 2,500 civil servants in Slovakia, and we've tested different behaviourally informed strategies to promote risk communication with the goal to increase the likelihood of reporting corruption.
Next we tested two messages here, we had a control group of no message and then we had a group treatment which was about encouraging civil servants to speak up, to report corruption risks they observed. The second one was about leadership, a message from the leadership stating that people should report the risks and that managers report the risk and encourage their teams to do the same. We found that both of these messages increased the stated willingness to report corruption risk, for this concrete use case, we found that the leadership message which had the strongest effect. Witj some of these findings we've been taken also into the report showing how behavioural science can then help make these integrity procedures, for example, in public administration more effective. Lastly note on the implications to be more practical, here the implications for practice within gave to the Slovak government was to make sure that there is communication process is easy and safe to use and well understood and that leaders set the standards for directions, that was the main second bit. Over to you Haris.
Haris Khan: Thank you, Mia, and thanks everyone for your attention. We have one more area to address, and one more example to show you, and then we're happy to open up to questions and discussion. One of the most fruitful areas for behavioural public administration is people management itself and how we how we operate our HR systems in government, this is a very well studied area of behavioural science, things like de-biasing hiring and candidate assessment is one area that's super important. Encouraging applications or encouraging reapplying, especially from populations that perhaps are discouraged from applying to government positions is another one, as well as creating a culture and behaviour of training and reskilling, especially in an era where our skills are constantly being redefined and there's new tools that we have at our disposal that we can use, it also means that we as public servants constantly have to be adjusting our skill set. I think behavioural science can be useful in creating the conditions that create a culture of constant learning and public administrations. These are just some examples of practices in which behavioural science can be helpful in creating a better people management system. but we'd like to talk about one specific example from a project that we've done in Greece.
We are helping Greece integrating behavioural science in their public sector, and that means two things at the moment. One is kind of institutionalising behavioural science in the public sector, providing them with a recommendation on how to create a unit and where it should sit and how it should function and items like this. Secondly, we're also conducting a couple of pilot projects with them, including on HR reforms.
Next we focused with our Greek colleagues on public sector employment attractiveness. What we mean by that is the attractiveness of the public sector as an employer competing with private sector employment in Greece and also with, in the context of Europe, employment outside of Greece. We did the usual behavioural mapping, and I don't want to go into all of the steps, but what we focused on was the first section of the map, was people finding out about the competition. What we're talking about is there's one big intake exam that potentially public servants would have to take and this happens every few years and so it's very important that someone who's interested in the public service as a career finds out about the exam and signs up for it in the window that it's available. Otherwise, they'll miss their chance for the next few years, the behavioural challenge is quite acute and what we tried to do. Next is to create a LinkedIn ad campaign in which we pushed LinkedIn ads to Greek professionals of roughly the right age to increase awareness of the sign-up phase or set up time and then invite them to apply.
The experiment went out to about 32,000 professionals in Greece, and we had a two by four different concepts for what would be attractive for a potential applicant. We had a neutral control message, another message that emphasised civic duty elements of a public service career, the idea that a career and the public service can help one serve their country. The third one was about job stability, public service employment is highly unionised, protected, it's potentially a very stable career, especially when compared to the private sector. Then lastly, the Greek public service offers quite a few opportunities for personal growth, we wanted to have an ad that talked about the opportunities for training and so on. Then we differentiated these four concepts for the general population and then for IT and STEM professionals, because the Greek public administration talked about how the IT and STEM professionals in particular were a hard to reach population.
One note on this is if you were a LinkedIn user and you received one of these ads, you only ever received the same ad again and again, you might have received the same ad, but a couple of times, you will never be sorted into a different into a different ad. Here are the images and the text, to illustrate we tried to align the pictures and the text. It's in Greek, obviously, in the picture, but then you have the translation below. With the concept that we were trying to test, these concepts came out of a literature review on what works in attracting people to a public service career. these are the general population ads on the
Next you see the ads for IT and STEM, which focus a bit more on the technology aspects of the various concepts that we're talking about. Quickly, in terms of results, we saw a big effect on the civic duty messaging which had a much higher click-through rate than the others and really drove engagement with the ads in a statistically significant way. This is surprising because when we asked when we asked the like our colleagues in the administration which ones they thought would work the best, they said “not civic duty.” so it was really interesting the example of kind of a counter-intuitive result and the reason why we do this kind of testing. We found that the job stability message worked just about as well as the neutral message, and then personal development worked much less than the neutral message. The neutral message click through rate is that red line that you see in the chart there, plenty of things to unpack. Perhaps there's a perception that the personal development is not available in the public administration, or that it's not as important to the people who might be applying for jobs, the real strong result was that the civic duty message was super appealing and drove a lot of interest in the public service career. We recommend that the Greek public administration take that on as a key part of their document strategy moving forward.
That's it from our results. I just want to talk about two quick things before we open the floor for questions. First is an initiative that we're doing here at the Public Governance Director at the OECD, it's called Government Unstuck and what it tries to do is to talk about the fact that in general, we're finding that governments are reporting that they are having difficulty delivering for their citizens in a tangible and at-scale way, and that this is having a negative effect on trust in government and the ability, or at least the perception of people in their government's ability to deliver.
We see that behavioural science really fits this to help unleash these reforms, we're able to analyse barriers and identify the bottlenecks that are causing reforms to get stuck. We're able to reveal frictions and sludge and base shape, which is, the sludge that affects the internal government processes that also perhaps hinder the delivery of real outcomes for citizens. We're able to test, measure and able to understand for a given policy, or program, or intervention of any kind, whether it's having the effect that we actually like to see in the world, or not. You'll be seeing hopefully soon more information from the OECD on government unstuck and more work on this on this point, and we really see behavioural science as a key plank in this work in terms of ensuring that governments are able to deliver on their on their promises and on their priorities, and that's kind of how we're orienting our behavioural science work at the moment is to show it as a way that governments can deliver moving forward.
Lastly I promised, you're all invited to this meeting that we're having on the 19th and 20th [May 2026]. You can sign up with this QR code. We can also send a link after the after the meeting or place it in the chat, but we're having a two-day meeting in the Hague in the Netherlands. The agenda is jam-packed with the topics ranging from behavioural public administration, to sludge audits, to tax and compliance, to mainstream of behavioural science, any kind of topic that you might be able to think of. The only requirement is that you are a government official and that you're interested in behavioural science. If you're those two things, we invite you to join as much as you possibly can given time zones. We're excited about this upcoming meeting and inviting anyone at the call to join us there, online or in person. Once you sign up you get the agenda and all the other information that you might want from the QR code there. With that, perhaps we can open up for questions and thanks again for your attention and for listening to us.
Matthew Davies: Wonderful. Thank you so much, Mia, Haris and Patrick. It was a really fantastic set of examples you've provided about how we can think about behavioural public administration. We've got a few questions in the chat and feel free to keep them coming. We've got about 10 or so minutes to get through them. We'll start with the first one from Sanket: given that the treatment arm used a range of reminder designs, I think this is in relation to the PIAO Italian example, given that the treatment arm used a range of reminder designs, while the control relied on the same reminder repeated over time. How did you think about the role of novelty or salience versus more sustained behavioural change in explaining the effects?
Patrick: Haris, do you want to take this one or should I?
Haris: Go for it, Patrick.
Patrick: Okay I just wanted to clarify one thing. The treatment arm had four reminders, whereas the control reminder just had one. There weren't those repeated reminders as well. What was the second part of the question?
Matthew: How did you think about the role of novelty or salience versus more sustained behavioural change and explaining the effects?
Patrick: Novelty and salience. I think novelty was definitely something that was thought about when we wanted to include the calendar. Suppose as you're going through your email, you're always looking at text, so that was something we wanted to really factor in when we were thinking about including that calendar and having that salient date and the image and making that a visual thing. I think many reminder trials don't actually do that, if they tell asking you to do something, it's usually just in written text. I would say, we wanted to address the salience and novelty element particularly with that, that visual of the calendar.
Matthew: Thanks, Patrick. We've got another question. This one is about ethical considerations. So how are ethical considerations and guidelines engaged within your interventions? So this is more broadly across all the set of interventions.
Haris: Maybe we can take the first stab, but then if we, Patrick, go and something, please, free. We have a paper on this it's called Tools and Ethics for Behavioural Science, published, I believe, in 2021. It provides a set of guidelines basically on how to do behavioural science projects in an ethical and responsible way. Throughout all these projects, in Slovakia and Greece and then in Italy, we really understood that we're working with real people's real behaviours, and so the goal was always been to ensure that we protect personal data issues, especially in Europe, we had the GDPR regulations, which can be quite strict, so that's part of it. We work really closely with our colleagues in each of those countries to understand what are the contextual factors that are at play with each of these interventions, especially, for example, in Slovakia, where we're talking about perceptions of corruption, we had to be really sensitive about making sure that we're not asking people to respond in a way that perhaps is dangerous or problematic for them. I think applying the guidelines at a high level is one thing and then secondly, being really sensitive to the context of the individuals themselves that we're talking to or trying to change the behaviour of each of those countries. There's another way of doing it.
Matthew Davies: Welcome everyone my name is Matthew Davies, I'm Chair of the New Zealand Behavioural Insights Community of Practice and I'll be facilitating today's session. It's really exciting to have everyone here from across both Australia and New Zealand, as well as our colleagues from the OECD dialling in from Paris. This is an exciting session because it marks the first jointly hosted event we've had between Behavioural Insights Community of Practices in New Zealand, BICOP and BETA, and we hope this is going to be the first of many opportunities to share behavioural science knowledge and capability across the ditch.
I want to start out with a couple of acknowledgments before we crack into things, I'd like to firstly acknowledge the Indigenous peoples of the lands we join from today, Ngā iwi Māori Tangata Whenua in Aotearoa, New Zealand and Aboriginal and Torres Strait Islander peoples from Australia, and all Indigenous peoples represented here.
As I've noted, we've got many people dialing in from across the world from 7am in Paris through to 5pm in New Zealand and everything in between across Australia. A massive thank you to everyone for accommodating these slightly awkward time differences and especially to our presenters for getting up so early for this session.
The context for this session is about behavioural public administration, behavioural science, has become increasingly applied to help governments design better policy and services for the public, but today's session turns the lens inwards. We'll be hearing about how behavioural insights can be applied to the machinery of government, everything from recruitment through to organisational performance, in other words how governments can function better internally to deliver better outcomes externally.
I'm excited to introduce our speakers today we've got Haris Khan, who's a policy analyst and project lead leading the OECD's work on behavioural public administration. We've got Mia Drazilova, who is a behavioural economics advisor and policy analyst supporting governments to design and test behavioural approaches. Patrick Tehrani, a consultant specialising in behavioural analysis and economic modelling related to administrative burdens and public sector performance.
In terms of the agenda before I hand over to our speakers, we'll have about 30 or so minutes for presenting, then we'll have 15 minutes or so for Q&A and we'll be aiming to wrap up before the hour. If you could just please pop your questions into the chat as we go, hopefully we'll have time to get through them in the Q&A.
Note that this session is going to be recorded, so just be aware and the recording will be shared after the event. Last thing I'll say is if you could just keep your cameras off to support with the bandwidth of the call, that would be greatly appreciated because we've already had a couple of technical challenges so far. Looks like and we'll basically go, I'll hand over to Haris, Mia and Patrick. Thank you.
Haris Khan: Thank you, Matthew. Thank you to everyone who helped organise this call. It's a pleasure to be here, and we're happy to get up early to speak with our Australian and New Zealand colleagues, it's an honour. As Matthew mentioned we plan on talking for a few minutes and then we're hoping for a lively Q&A discussion. We invite you to put your questions in the chat, and then we're happy to answer them at the end of the call.
Before we do that we have kind of three goals in this presentation, one is to introduce the behavioural science at the OECD, which we'll do quite quickly, and then move into what we call the blueprint for behavioural public administration, we'll talk about what that is, when we talk about BPA, what are we talking about? The different applications that are possible with this tool and then we'll also get into a couple of projects that we've done recently on the topic that bring this kind of concept to life. That's the goal before we get into the discussion.
As Matthew mentioned, the team here today is me, Mia and Patrick, we’ll hear from all three of us and we all worked on these projects together. If there's any detailed questions, we're happy to take them on.
We at the OECD's Bureau of Science Unit say that we do three things, we convene, we enable, and we advise. We'll talk about each of those in more detail in a second. But we sit in the OECD's public governance directorate, which is the part of the OECD that's responsible for helping governments conduct their business better and implement reforms and other practical changes to the way that they function in order to make them more efficient, more effective, and more able to deliver for their citizens, our mandate sits within that.
Next let’s talk about the convening function that we do, bringing together over 500 government officials as part of the OECD Behavioural Research and Action International Network. It's an informal OECD network that's meant to be a place for knowledge exchange and peer learning on behavioural science issues around the world. At the end we'll have a little invitation for everyone on the call who would like to join us, we're having a meeting of this network on the 19th and 20th of May in the Netherlands, but also online, if possible, we'd love for you to join, and happy to provide more information at the end.
The second thing that we do is we enable the use of behavioural science with tools and templates, this is kind of classic OECD work. We write guidance documents, guidelines, methodologies, etc. that are intended to help people apply behavioural science to their work in their public administrations. the key ones for today are perhaps the basic toolkit, which is kind of a step by step methodology on how to do a behavioural science project, as well as the recently released BPA framework itself that will be going through in detail. So there is all kinds of tools on these topics and others including; measurement methodologies and ethical frameworks from behavioural science that we published in the last few years
Lastly in terms of our introduction, we do also advise policymakers on how to apply behavioural science in a very practical way, that means doing projects with country officials and you’ll see a summary of some of the projects that we've done recently that are particularly relevant to today's presentation, including a project in Slovakia trying to improve perceptions of corruption. The main BPA project that we've done is with Italy, helping them apply behavioural science to a whole set of internal government functions, and we'll talk about some of those. We also did a project in Greece where we helped apply behavioural science to making the public sector a more attractive employer. Then a big piece of work that we do is on sludge audits and sludge methodologies in general, we'll talk about each of these experiences as a way of animating the framework that we that we have today.
This is who we are, the next topic of behaviour public administration itself and turning the behavioural science toolkit on civil servants themselves, the key take away here is all the same biases, the same framing effects that apply to people in the real world, apply to us as public officials as well, and sometimes more so. I'm sure we can all think of times in our careers and in the public service where we've seen these biases at play. The basic premise of behavioural public administration is that if we are able to understand these biases that exist for policy makers or people who work in the government, and then create policies and tools and interventions to address those biases we're able to make the entire government function better and therefore make it more able to deliver for our citizens.
Next, we’ll talk a little bit about what the difference is between behavioural public administration and behavioural public policy. At a high level, behavioural public policy focuses on citizens or users or customers, people external to the government whose behaviour might be the object of behavioural change projects. For example, increase savings amongst the general population, the goal there is to improve the policy outcome, and the scope goes from the very beginning of the policy cycle, from policy design all the way to the actual implementation of policies.
With BPA, we're talking about using the same tools but using them on civil servants themselves, the goal is to improve public administration processes, this can be done through a variety of different, in a variety of different areas. We can talk about it in HR and policy development itself and the process of policy development. Also things like project management, procurement, IT, and the machinery that makes government function can be done in a more efficient way. The promise is that if we are able to do it more efficiently internally, then we're able to deliver better for citizens externally as well.
So that's what we're here to talk about, next we'll talk about the biggest project that we have on this topic, now I think I'll hand over to Mia.
Mia Drazilova: Thank you, Haris, we already introduced to you in the beginning, there will be few projects that we've done and we want to talk about. We'll start with this Italian one and one of our most recent projects, it was a big project, it was over two years. What we've tried to do here, we've tried to identify to what kinds of opportunities may be here for behavioural insights to really add value to different processes and practices, and we looked at the Italian public administration. We tried to take a behavioural lens, trying to see how the different civil servants operate, and tried to see how we can help them act more effectively, in whatever area it may be, and we prioritise those based on the most available evidence that we have from behavioural science.
A small disclaimer, these opportunities are not limited to Italy. So many governments, based on the findings you can see here, and you'll see in the presentation, you can find value in taking this behavioural lens to the work. And this is also something we've tried with this project in Italy.
Next this is something that Haris already mentioned in his presentation, that we often work with OECD's basic framework, this is a lens that we've applied in this project that we've started. Firstly when we try to think about what opportunities there could be for behavioural public administration, there could be plenty. We tried to follow this framework and see what are the behaviours we want to change? What are some of the barriers or enablers for this change? What could be the strategy to creating behaviour change? And then later, when we've defined this, we designed the interventions to test, whether this strategy that we were following succeeded. Lastly, focusing on the change, we've tried to disseminate the findings and see how these changes could be scaled up. So this is just the operational framework we picked.
Next as I've mentioned, there have been plenty of areas and I mean maybe it's a bit small, but as we've approached it, we've tried to talk to a lot of civil servants in Italy and involve different stakeholders from the government and try to see what are maybe some of the biggest areas where behavioural science could really make a difference. We've outlined a lot of these and then in the end we've considered a range of administrative areas and this is helpful for also for the other project you will hear, but essentially from hiring recruitment to things like performance reviews, also we considered internal communications.
We found that the strongest opportunity in this concrete use case was to focus on the planning process. On the way how the government, Italian public servants plan their processes internally, this is a very Italian specific thing, but to give you context, this relates to their integrated plan of activities and organisations. This is an internal planning Document or a scheme they call PIAO, and why we decided to focus our efforts here mostly because we recognise that there is an opportunity for Behavioural Science to help develop Italian civil servants to develop effective, timely, and transparent planning documents. This was the this was the approach we took for the projects. For the results, I'll hand over to my colleague Patrick to walk through what we've done in the project.
Patrick Tehrani: As Mia mentioned, we did see a lot of opportunity for behavioural science to be applied to this form of internal compliance, where people within the Italian public administration have to submit their PIAOs on a planning portal, at the central level they have to do it by 31st of January and on a local level by 30th of March.
However, many public administrators don't submit it. The challenge was that despite the requirement to publish this planning document on the government's website on the PIAO Portal, many didn't do so before the deadline, this was a classic behavioural problem. Many trials have shown that, you know, reminders can be very effective with increasing behaviour change among citizens, very few studies have gone and actually applied this technique internally. We then ran a randomised control trial with Italian public civil servants to evaluate the effect of reminders on PIAO publication.
We had two groups, a control group and a treatment group. The control group received just the standard reminder that they were receiving normally from the Italian government. Those receiving the treatment, they received 4 behaviourally informed reminders, and I'll unpack what this includes a bit shortly. We wanted to take a look at 2 outcome measures, the first being whether or not the PIAO was published and the second is what date the PIAO was published. We ran this for both central public administrations with a publication deadline of the 31st of January, a different deadline for the March ones, for the local ones.
For the reminders we leveraged behavioural science to motivate public administrators to publish on the PIAO Portal. We can see on the left-hand side of the screen, we have the control reminder, very standard, no clear visual support. For the first treatment, we had four reminders, as I mentioned previously, and each of those were different. The first reminder that we sent to the treatment group very clearly stated the task and the subject, it stated the deadline first, it had less text, it was a bit easier to read. It was attention grabbing with a visual calendar that demonstrated visually the deadline, of course the calendar image was adapted to different people with different deadlines. The purpose of this behavioural redesign was not to provide any new information, they had the same information set, but to reduce friction, increase attention and prompt timely action.
These were the reminders that we used for the second, third and fourth reminder. The second treatment included a step-by-step guide, and it clarified when the PIAO is ready to be published. The third reminder said that the non-publishing is noted and reminded where and how to publish the PIAO. Then the 4th reminder was much more official, reminded why it's required to publish on the portal, and it was signed by the very official department. It added this additional level of authority, this allowed us to compare not just the behavioural reminders versus the standard reminders, but also the impact of different behavioural approaches in each reminder.
Next we'll be turning to some of the results. Naturally, as you would expect, when we send reminders more public administrators published the PIAO on the portal. The control and the first treatment reminder was sent on the 27th of March, three days before the deadline for local public administrators to publish. Both reminders were associated with an increase in the PIAO publications. We see along that x-axis we have the publication date and the darker gray bars are demonstrating the weekends where we naturally expect lower publication rates. Then on the y-axis, we see the publications per day. We notice that upon that first treatment, we do see a somewhat higher spike for the treatment reminder relative to the control, then as we move along the x-axis, we see that those additional treatments, those additional reminders, were also associated with an increase in the PIAOs that were published. However, the first behavioural reminder led to a larger increase in publications than the control group reminder, we can also very clearly see the effect of those subsequent reminders as well.
We find that BPA reminders were more effective than standard reminders, and that this effect lasted over a month. The effect of the two treatments lasted for over two months in our post-treatment measurement period. And often with reminder trials, the treatment causes participants to act, but does not necessarily raise the success rate. They'll look at the time until the behavioural response and that will be lower in the treatment group, but then the overall response of the participant group does not significantly differ. We see that not only did they respond sooner in the behavioural treatment condition, but the overall response rate was much higher. The control group did not close the gap with the treatment group during the data collection period.
What do these findings suggest? These findings suggest that public administrators can benefit from clear deadlines, but also that more research is needed. We put together a few recommendations based on this, the first is that we encourage that, from the findings, internal public administration to simplify and shorten written communications between public administrators and with citizens where possible. To make deadlines more attention grabbing in the reminders. We also know that the step-by-step guide seemed a bit less effective and public administrators may already know how to use the PIAO Portal, they might already know how to comply, but they need more motivation to do that, that should be a broader focus. And the step-by-step guide was also the longest treatment of the separated treatments. I can pass back over to Mia.
Mia Drazilova: Thank you, Patrick. A small note before I go to the framework, essentially the learnings from what Patrick has introduced, they're just one example of an experiment that we've done as part of this project. Another experiment that we outlined in the report, related to public servants in Italy in the same use case, not only doing their planning document, but taking a step back and actually looking into how they're setting their objectives, so for their planning, whether the objectives are aligned with public value, they set these objectives, and has been also been behaviourally informed. This is just to give you an idea and also to motivate you to go and read the report if you're interested.
Moving into what we've done further in this report. We've tried to create a framework for behavioural public administration, and we have tried to, if you look at the slide you can see the visual. We've tried to outline that behavioural public administration, and its practices can be applied across the whole policy cycle. When we've looked into a lot of examples, not just the ones we've done in Italy as part of the project or in Greece, as Haris will be talking about, beyond that, we've identified that these practices are irrelevant across the whole policy cycle. This relates to the early stages, when we talk about the analysis and design, for example, the biases, can really affect how evidence is interpreted or options are weighed, as an example. When we look at the middle stage, the implementation and management, this could relate to being too optimistic in the planning, maybe having some [unintelligible] costs in the thinking, maybe the complexity can also delay the delivery and the different projects. Then when we look at the late stage, it's about monitoring and evaluation, for example, there are different behavioural factors that really influence how evidence is generated and how it is used. When we try to also talk about the different examples, we want to see them as part of the whole cycle.
Next some of the actionable insights for policy design that we've identified, we've tried to identify these for all of the stages of the policy cycle, for example, the goal of these is essentially to guide the policymakers as they do their problem analysis and policy design. The first one could be about reducing messenger effect, for example, when the identity of the messenger is not relevant and maybe anonymising the source of evidence or the source of recommendation. People are focusing on the message, not the messenger, for example, if a leader sends the message it could have an effect “this should be deprioritised or can be prioritised”. The second bit is using evidence effectively, this should guard against confirmation bias, for example, prompting teams to consider the opposite, and then stating some reasons why their view could be wrong in the first place. Other suggestion that we have made in the report, for example, relates to framing risk mindfully. Loss frames can really push the risk seeking, and gain frames can really push the risk aversion. Trying to pick the right frame also when talking internally and doing a policy design or discussions in the teams can really help to potentially match the policy goal that we may have. The last bit is a lot about the diversity of views and considering divided perspectives. Here is the goal and this is related directly to the project that we're having at [inaudible] Italy. I mentioned, we've tried to prompt them to, for example, consider public value as they're planning their objectives and their planning documents. Here it's about engaging citizens early and to counter the curse of knowledge that policy makers might be having and really bringing this lived experience at these insights into [inaudible] at every stage, for example, briefs or any white papers that policy makers might be working on at any stage of the of the process, for example of a project that is not our project, but it comes from the brain, the network that Haris introduced in the beginning.
Next here an example which is called Capacity to Act, this is an example from the Netherlands which illustrates how these findings can be incorporated. How this came about, a court challenges that policy makers sometimes overestimate the behavioural effects of the different policies and might underestimate what difficulties people might face when they have to comply with the different rules and procedures. of course, even though if people know what they have to do, they may act differently due to different factors, emotional, social, contextual, et cetera, different barriers they might be facing. To address this, the Netherlands had a great example to introduce a Capacity to Act test. This is a formalised quality standard in Dutch policy making, it prompts policy makers to, when thinking of a new policy, to analyse the feasibility in context of how people live their daily lives, through various tools like citizen journey mapping, consultation with the target audiences, involving the citizen early, and possible also piloting these measures to test for their behavioural impacts early on.
Next to stress that this framework, I was staying on the outer layer and we have plenty more examples in the paper, if we look more in the internal side of the framework that we've outlined, beyond the policy life cycle, there of course also, so people public administration can be applied to plenty more topics, that are relevant for public administration, such as integrity, for example. People management is something we'll talk about in more detail and internal government processes and some of the barriers that are in internal processes. We believe that really addressing these, so these different behavioural bottlenecks, can further improve also this public sector performance without maybe a need for major reform.
Just one more example and then I'll give over to Haris, for the integrity, one of these areas that we've identified, we recognised here in the paper that Bureau of Public Administration can support the public sector’s integrity. Some of the practices we outline here, for example, leveraging leaders to set and reinforce norms this is an example we've also tested with Slovakia, I'll show you the example in a second, then also things like fostering the identity of policymakers, or applying behavioural insights to employment psychology to help policy makers to recognise, but also report when they see, for example, corruption risks. More practices we've outlined in the paper, just to give you a quick example, we did a few years back with Slovak Republic.
Next the goal here in this project, was to support civil servants in Slovakia to report corruption risk when they see it. As part of this project, we tried to see how these different behavioural elements can inform the design implementation of risk management policies. The approached here, a similar way as in the Italian project, we've applied the basic framework to see what are the barriers at play. We found out that there's a culture of silence, many public servants didn't feel safe to raise corruption risks, there were two main barriers that we found. One was lack of understanding of “what is a corruption risk”. The second is not enough of a leadership of modeling or leadership not going by example, how to prevent and mitigate corruption. Based on this, we've designed a survey experiment which was running with more than 2,500 civil servants in Slovakia, and we've tested different behaviourally informed strategies to promote risk communication with the goal to increase the likelihood of reporting corruption.
Next we tested two messages here, we had a control group of no message and then we had a group treatment which was about encouraging civil servants to speak up, to report corruption risks they observed. The second one was about leadership, a message from the leadership stating that people should report the risks and that managers report the risk and encourage their teams to do the same. We found that both of these messages increased the stated willingness to report corruption risk, for this concrete use case, we found that the leadership message which had the strongest effect. Witj some of these findings we've been taken also into the report showing how behavioural science can then help make these integrity procedures, for example, in public administration more effective. Lastly note on the implications to be more practical, here the implications for practice within gave to the Slovak government was to make sure that there is communication process is easy and safe to use and well understood and that leaders set the standards for directions, that was the main second bit. Over to you Haris.
Haris Khan: Thank you, Mia, and thanks everyone for your attention. We have one more area to address, and one more example to show you, and then we're happy to open up to questions and discussion. One of the most fruitful areas for behavioural public administration is people management itself and how we how we operate our HR systems in government, this is a very well studied area of behavioural science, things like de-biasing hiring and candidate assessment is one area that's super important. Encouraging applications or encouraging reapplying, especially from populations that perhaps are discouraged from applying to government positions is another one, as well as creating a culture and behaviour of training and reskilling, especially in an era where our skills are constantly being redefined and there's new tools that we have at our disposal that we can use, it also means that we as public servants constantly have to be adjusting our skill set. I think behavioural science can be useful in creating the conditions that create a culture of constant learning and public administrations. These are just some examples of practices in which behavioural science can be helpful in creating a better people management system. but we'd like to talk about one specific example from a project that we've done in Greece.
We are helping Greece integrating behavioural science in their public sector, and that means two things at the moment. One is kind of institutionalising behavioural science in the public sector, providing them with a recommendation on how to create a unit and where it should sit and how it should function and items like this. Secondly, we're also conducting a couple of pilot projects with them, including on HR reforms.
Next we focused with our Greek colleagues on public sector employment attractiveness. What we mean by that is the attractiveness of the public sector as an employer competing with private sector employment in Greece and also with, in the context of Europe, employment outside of Greece. We did the usual behavioural mapping, and I don't want to go into all of the steps, but what we focused on was the first section of the map, was people finding out about the competition. What we're talking about is there's one big intake exam that potentially public servants would have to take and this happens every few years and so it's very important that someone who's interested in the public service as a career finds out about the exam and signs up for it in the window that it's available. Otherwise, they'll miss their chance for the next few years, the behavioural challenge is quite acute and what we tried to do. Next is to create a LinkedIn ad campaign in which we pushed LinkedIn ads to Greek professionals of roughly the right age to increase awareness of the sign-up phase or set up time and then invite them to apply.
The experiment went out to about 32,000 professionals in Greece, and we had a two by four different concepts for what would be attractive for a potential applicant. We had a neutral control message, another message that emphasised civic duty elements of a public service career, the idea that a career and the public service can help one serve their country. The third one was about job stability, public service employment is highly unionised, protected, it's potentially a very stable career, especially when compared to the private sector. Then lastly, the Greek public service offers quite a few opportunities for personal growth, we wanted to have an ad that talked about the opportunities for training and so on. Then we differentiated these four concepts for the general population and then for IT and STEM professionals, because the Greek public administration talked about how the IT and STEM professionals in particular were a hard to reach population.
One note on this is if you were a LinkedIn user and you received one of these ads, you only ever received the same ad again and again, you might have received the same ad, but a couple of times, you will never be sorted into a different into a different ad. Here are the images and the text, to illustrate we tried to align the pictures and the text. It's in Greek, obviously, in the picture, but then you have the translation below. With the concept that we were trying to test, these concepts came out of a literature review on what works in attracting people to a public service career. these are the general population ads on the
Next you see the ads for IT and STEM, which focus a bit more on the technology aspects of the various concepts that we're talking about. Quickly, in terms of results, we saw a big effect on the civic duty messaging which had a much higher click-through rate than the others and really drove engagement with the ads in a statistically significant way. This is surprising because when we asked when we asked the like our colleagues in the administration which ones they thought would work the best, they said “not civic duty.” so it was really interesting the example of kind of a counter-intuitive result and the reason why we do this kind of testing. We found that the job stability message worked just about as well as the neutral message, and then personal development worked much less than the neutral message. The neutral message click through rate is that red line that you see in the chart there, plenty of things to unpack. Perhaps there's a perception that the personal development is not available in the public administration, or that it's not as important to the people who might be applying for jobs, the real strong result was that the civic duty message was super appealing and drove a lot of interest in the public service career. We recommend that the Greek public administration take that on as a key part of their document strategy moving forward.
That's it from our results. I just want to talk about two quick things before we open the floor for questions. First is an initiative that we're doing here at the Public Governance Director at the OECD, it's called Government Unstuck and what it tries to do is to talk about the fact that in general, we're finding that governments are reporting that they are having difficulty delivering for their citizens in a tangible and at-scale way, and that this is having a negative effect on trust in government and the ability, or at least the perception of people in their government's ability to deliver.
We see that behavioural science really fits this to help unleash these reforms, we're able to analyse barriers and identify the bottlenecks that are causing reforms to get stuck. We're able to reveal frictions and sludge and base shape, which is, the sludge that affects the internal government processes that also perhaps hinder the delivery of real outcomes for citizens. We're able to test, measure and able to understand for a given policy, or program, or intervention of any kind, whether it's having the effect that we actually like to see in the world, or not. You'll be seeing hopefully soon more information from the OECD on government unstuck and more work on this on this point, and we really see behavioural science as a key plank in this work in terms of ensuring that governments are able to deliver on their on their promises and on their priorities, and that's kind of how we're orienting our behavioural science work at the moment is to show it as a way that governments can deliver moving forward.
Lastly I promised, you're all invited to this meeting that we're having on the 19th and 20th [May 2026]. You can sign up with this QR code. We can also send a link after the after the meeting or place it in the chat, but we're having a two-day meeting in the Hague in the Netherlands. The agenda is jam-packed with the topics ranging from behavioural public administration, to sludge audits, to tax and compliance, to mainstream of behavioural science, any kind of topic that you might be able to think of. The only requirement is that you are a government official and that you're interested in behavioural science. If you're those two things, we invite you to join as much as you possibly can given time zones. We're excited about this upcoming meeting and inviting anyone at the call to join us there, online or in person. Once you sign up you get the agenda and all the other information that you might want from the QR code there. With that, perhaps we can open up for questions and thanks again for your attention and for listening to us.
Matthew Davies: Wonderful. Thank you so much, Mia, Haris and Patrick. It was a really fantastic set of examples you've provided about how we can think about behavioural public administration. We've got a few questions in the chat and feel free to keep them coming. We've got about 10 or so minutes to get through them. We'll start with the first one from Sanket: given that the treatment arm used a range of reminder designs, I think this is in relation to the PIAO Italian example, given that the treatment arm used a range of reminder designs, while the control relied on the same reminder repeated over time. How did you think about the role of novelty or salience versus more sustained behavioural change in explaining the effects?
Patrick: Haris, do you want to take this one or should I?
Haris: Go for it, Patrick.
Patrick: Okay I just wanted to clarify one thing. The treatment arm had four reminders, whereas the control reminder just had one. There weren't those repeated reminders as well. What was the second part of the question?
Matthew: How did you think about the role of novelty or salience versus more sustained behavioural change and explaining the effects?
Patrick: Novelty and salience. I think novelty was definitely something that was thought about when we wanted to include the calendar. Suppose as you're going through your email, you're always looking at text, so that was something we wanted to really factor in when we were thinking about including that calendar and having that salient date and the image and making that a visual thing. I think many reminder trials don't actually do that, if they tell asking you to do something, it's usually just in written text. I would say, we wanted to address the salience and novelty element particularly with that, that visual of the calendar.
Matthew: Thanks, Patrick. We've got another question. This one is about ethical considerations. So how are ethical considerations and guidelines engaged within your interventions? So this is more broadly across all the set of interventions.
Haris: Maybe we can take the first stab, but then if we, Patrick, go and something, please, free. We have a paper on this it's called Tools and Ethics for Behavioural Science, published, I believe, in 2021. It provides a set of guidelines basically on how to do behavioural science projects in an ethical and responsible way. Throughout all these projects, in Slovakia and Greece and then in Italy, we really understood that we're working with real people's real behaviours, and so the goal was always been to ensure that we protect personal data issues, especially in Europe, we had the GDPR regulations, which can be quite strict, so that's part of it. We work really closely with our colleagues in each of those countries to understand what are the contextual factors that are at play with each of these interventions, especially, for example, in Slovakia, where we're talking about perceptions of corruption, we had to be really sensitive about making sure that we're not asking people to respond in a way that perhaps is dangerous or problematic for them. I think applying the guidelines at a high level is one thing and then secondly, being really sensitive to the context of the individuals themselves that we're talking to or trying to change the behaviour of each of those countries. There's another way of doing it.
Matthew: Fantastic. Thank you. I might ask the next question to Samir and then obviously others may have a view on this as well, we've got a question from Katrina Kent who asks, in an environment of high pace and resource constraints in the policy world, monitoring and evaluation is often not prioritised. Did you encounter resistance to monitoring and evaluation of your tests, and if so, how did you overcome it?
Mia: I think when we talked about the Italian examples, maybe this is, it's important to say here in this case that the teams that we were dealing with doing this project, for them in general, behavioural science was a novel topic. Of course, all these elements about monitoring and evaluation, everything related to our work was a new thing for them, so I think to be frank, I think we have encountered difficulties across the whole part of the project, because of course this experimentation part was also a new thing, monitoring evaluation as well. I think in this concrete project, we've supported the team in doing so. But I think maybe, Haris, you can reflect more on the Greece example, because I think there you have a stronger capability building component. So maybe you can describe a little bit more how you're addressing it there, because in the Italian trials, it was more us helping the teams, but maybe you can talk about Greece.
Haris: Sure, thanks, Mia. I think it's different for every partner and every situation for sure, but I think what we were able to do in Greece was to convey the importance of finding out what actually worked. For, example, in the experiment that we did, I think I mentioned, the public officials in Greece were convinced that the civic duty message wouldn't work, because of whatever prior assumptions they may have or experience they might have, and so I think that, that experience kind of helps challenge those assumptions and then has made ensuring that there's a strong monitoring and evaluation component in the future trials much easier. I think for us what seems to have worked is if you make sure that you get one good experiment where the result is robust and maybe a bit surprising, then it kind of opens up the opportunities for more of that kind of work.
Matthew: There's a related question that Katrina asked actually, which is, do you typically use the language of behavioural science in your work and try to educate and lift the capability of teams along the way, or do you use plain language and at key times reveal evidence practice behind it?
Mia: I can start on this one, I think it really depends, right? Maybe I'll now take a step away from just the behavioural public administration behavioural standards at large, for example, a big chunk of our work relates to sludge audits, which is a very unique terminology. I think this is an example which we've been dealing with recently where we find both words being triggering at certain stages in some governments. For example, we step away more into things like sludge research, where we call it frictions. We try to play around with the terminology and the same of course applies to behavioural science. I think in this concrete project in Italy, we were lucky that the project was about bringing behavioural science into the administration. Of course, we've tried to, from the beginning of the project, introduce the topic to them, of course, we didn't start by talking about, not just choice architecture, whatever at hand, but we tried to really bring on more examples that they could to from their areas and then we started speaking more the common language, it's definitely a challenge. If we say “we want to address optimism bias in your project planning” that would not fly in this concrete project, but we've more said “we notice your projects are running over time and over budgets” and so on, and we would like to work with you to help you with that, that language then resonated way more. I don't know if my colleagues want to add something.
Matthew: Might just jump in as well, because I think having read through the report, it's really fascinating and encourage everyone to read it. It's great to actually see so many examples, cited from Australia and a couple from New Zealand. It's good to see some of this is actually already being done in practice, I guess my question is, for those policy teams who are completely new to this. Where do you suggest they start when thinking about thinking about public administration with the behavioural science lens? There's obviously a lot that they have to contend with and working under constrained resources and time. Do you have any suggestions about where to start thinking about all of this? I might hand that to Patrick.
Patrick: Maybe just one note on the constrained resources point, I think more and more of our work is really being about trying to demonstrate the value of our work, trying to be able to put numbers to the interventions to really say that, if we were to generalise this effect to our broader population, what would the monetary or time savings, whatever it is, result in? I think just being able to try and translate the work that we do into something that's measurable and use that for case making is really important and an essential part of each project. But maybe I can pass to my other colleagues if they have anything additional they'd like to add.
Matthew: Might just, in the interest of time, we've got time for one last question. I can see one from Tom Emparato, and he's asked. Have you seen the appeal to civic duty have success in any other jurisdictions in recruitment messaging? So, if Haris, if you've got any thoughts on that.
Haris: Short answer is yes, we did an extensive literature review before we go and put those ads and we found that civic duty seems to work in general with caveats, obviously it depends, I think I think the effect is a bit contextualised with like the trust that people have in their governments and also their perception of the public service and [inaudible] environment, a job in public service to actually have an impact on people, but we found that it worked really well. Actually, I actually did a similar trial when I was in Canada, and that was about recruiting women to the armed forces, and there also we found that appealing to like a kind of a humanitarian angle on an armed forces career was also found to be effective or at least more effective than control for increasing interest in a job there. We seem to find it, we seem to find it does work, I think, and maybe not universally and maybe not in all situations, but as a general idea that seems to be effective.
Matthew: Fantastic. Thank you very much, Haris, Mia and Patrick, I can see we're almost out of time, I've just posted the link to the report. This has been really informative. I can see we've got lots of questions that we didn't have time to get around to, so it shows we've had lots of interest and engagement. Really appreciated your time and thank you as well to BETA and everyone else who's taking the time out of their schedules to join us today. We hope to see you at future BICOP and BETA events soon, and I'll just wrap up with a closing Fakatoki to acknowledge the space we're in nāu te rourou, nāku te rourou, ka ora ai te iw, which translates as by sharing our knowledge, we enable others in our communities to thrive. So thank you very much, everyone. Enjoy the rest of the day wherever you are. Thank you.
Fantastic. Thank you. I might ask the next question to Samir and then obviously others may have a view on this as well, we've got a question from Katrina Kent who asks, in an environment of high pace and resource constraints in the policy world, monitoring and evaluation is often not prioritised. Did you encounter resistance to monitoring and evaluation of your tests, and if so, how did you overcome it?
Mia: I think when we talked about the Italian examples, maybe this is, it's important to say here in this case that the teams that we were dealing with doing this project, for them in general, behavioural science was a novel topic. Of course, all these elements about monitoring and evaluation, everything related to our work was a new thing for them, so I think to be frank, I think we have encountered difficulties across the whole part of the project, because of course this experimentation part was also a new thing, monitoring evaluation as well. I think in this concrete project, we've supported the team in doing so. But I think maybe, Haris, you can reflect more on the Greece example, because I think there you have a stronger capability building component. So maybe you can describe a little bit more how you're addressing it there, because in the Italian trials, it was more us helping the teams, but maybe you can talk about Greece.
Haris: Sure, thanks, Mia. I think it's different for every partner and every situation for sure, but I think what we were able to do in Greece was to convey the importance of finding out what actually worked. For, example, in the experiment that we did, I think I mentioned, the public officials in Greece were convinced that the civic duty message wouldn't work, because of whatever prior assumptions they may have or experience they might have, and so I think that, that experience kind of helps challenge those assumptions and then has made ensuring that there's a strong monitoring and evaluation component in the future trials much easier. I think for us what seems to have worked is if you make sure that you get one good experiment where the result is robust and maybe a bit surprising, then it kind of opens up the opportunities for more of that kind of work.
Matthew: There's a related question that Katrina asked actually, which is, do you typically use the language of behavioural science in your work and try to educate and lift the capability of teams along the way, or do you use plain language and at key times reveal evidence practice behind it?
Mia: I can start on this one, I think it really depends, right? Maybe I'll now take a step away from just the behavioural public administration behavioural standards at large, for example, a big chunk of our work relates to sludge audits, which is a very unique terminology. I think this is an example which we've been dealing with recently where we find both words being triggering at certain stages in some governments. For example, we step away more into things like sludge research, where we call it frictions. We try to play around with the terminology and the same of course applies to behavioural science. I think in this concrete project in Italy, we were lucky that the project was about bringing behavioural science into the administration. Of course, we've tried to, from the beginning of the project, introduce the topic to them, of course, we didn't start by talking about, not just choice architecture, whatever at hand, but we tried to really bring on more examples that they could to from their areas and then we started speaking more the common language, it's definitely a challenge. If we say “we want to address optimism bias in your project planning” that would not fly in this concrete project, but we've more said “we notice your projects are running over time and over budgets” and so on, and we would like to work with you to help you with that, that language then resonated way more. I don't know if my colleagues want to add something.
Matthew: Might just jump in as well, because I think having read through the report, it's really fascinating and encourage everyone to read it. It's great to actually see so many examples, cited from Australia and a couple from New Zealand. It's good to see some of this is actually already being done in practice, I guess my question is, for those policy teams who are completely new to this. Where do you suggest they start when thinking about thinking about public administration with the behavioural science lens? There's obviously a lot that they have to contend with and working under constrained resources and time. Do you have any suggestions about where to start thinking about all of this? I might hand that to Patrick.
Patrick: Maybe just one note on the constrained resources point, I think more and more of our work is really being about trying to demonstrate the value of our work, trying to be able to put numbers to the interventions to really say that, if we were to generalise this effect to our broader population, what would the monetary or time savings, whatever it is, result in? I think just being able to try and translate the work that we do into something that's measurable and use that for case making is really important and an essential part of each project. But maybe I can pass to my other colleagues if they have anything additional they'd like to add.
Matthew: Might just, in the interest of time, we've got time for one last question. I can see one from Tom Emparato, and he's asked. Have you seen the appeal to civic duty have success in any other jurisdictions in recruitment messaging? So, if Haris, if you've got any thoughts on that.
Haris: Short answer is yes, we did an extensive literature review before we go and put those ads and we found that civic duty seems to work in general with caveats, obviously it depends, I think I think the effect is a bit contextualised with like the trust that people have in their governments and also their perception of the public service and [inaudible] environment, a job in public service to actually have an impact on people, but we found that it worked really well. Actually, I actually did a similar trial when I was in Canada, and that was about recruiting women to the armed forces, and there also we found that appealing to like a kind of a humanitarian angle on an armed forces career was also found to be effective or at least more effective than control for increasing interest in a job there. We seem to find it, we seem to find it does work, I think, and maybe not universally and maybe not in all situations, but as a general idea that seems to be effective.
Matthew: Fantastic. Thank you very much, Haris, Mia and Patrick, I can see we're almost out of time, I've just posted the link to the report. This has been really informative. I can see we've got lots of questions that we didn't have time to get around to, so it shows we've had lots of interest and engagement. Really appreciated your time and thank you as well to BETA and everyone else who's taking the time out of their schedules to join us today. We hope to see you at future BICOP and BETA events soon, and I'll just wrap up with a closing Fakatoki to acknowledge the space we're in nāu te rourou, nāku te rourou, ka ora ai te iw, which translates as by sharing our knowledge, we enable others in our communities to thrive. So thank you very much, everyone. Enjoy the rest of the day wherever you are. Thank you.