Categories
Not part of a category How do you do research? Promoklip

Full circle research

Yesterday, more than 100 researchers working in the Dutch government met in Utrecht. The government-wide research community held its first real-life event, and the organization had to ask participants to attend with just 5 persons per government organization. If you had told me this 10 years ago, I wouldn’t have believed you. So many researchers already work in government. Super.

In this blog, I share my presentation: about the context in which we work and why researchers are necessary for good service delivery. We look back at how far we’ve come, and also ahead: what will it take to let the lifeworld of citizens guide the delivery of government services? I call that doing full circle research.

An update in your mailbox every month about my research? Subscribe to my Dutch newsletter.

The circle of democracy and service delivery

In my research, I use a division between the system world and the living world, with both a collective and an individual side. This creates 4 quadrants. On the collective side, Cabinet, together with representatives of the people, shape their vision of society. This vision is then translated into laws, policies and services for individuals. Only after interaction between citizens and government does this become a reality in the context of the people themselves, or in the living world. And also in the living world, individuals group together and try to implement their ideas about society, thus coming full circle.

Drawing the circle of democracy and service delivery
Drawing the circle of democracy and service.

Of this circle, people experience mostly the individual side. “People don’t experience policies, they experience services,” says researcher Sabine Junginger (2016). That means the job of public service providers is to translate collective value into individual experiences that are also valuable.

Want to know more about this collective and individual side? Then read the blog Executive and service provider.

That’s not easy. The practice of the system world now is often a big waterfall. It goes clockwise in a circle, from policy to service and pours out over citizens and society. If you disagree, you can show it in your voting record.

It may be different if we collect citizens’ experiences and introduce them, counterclockwise, into the system world. Those experiences can shape how we offer services and how we design services at all. For the policies enacted for it, and yes, perhaps also for the legislation and associated ideas about interventions in society and the effect it will have.

We have to turn it completely around. And we can, because that’s exactly our job as government researchers.

We extract experiences from the lifeworld. We have several methods and techniques to do this properly. Moreover, we are able to share these stories within our organizations in a way that motivates colleagues to take action.

We have shown that over the past 10 years.

How it began

In most organizations, it started with usability testing of websites and other screens. At least for me, too. I started working at the Executive Agency of Education in 2013. One of the first things I learned was how to set up and run a usability test. I visited schools with my laptop under my arm. In the office, I showed videos to colleagues and talked about how students or school employees experienced our digital services.

Look, so cute: my first steps in the research world.

In 2017, I started my blog. I shared what I was learning and what we were trying out. You also started sending me your experiences.

I saw that we were growing as researchers. We went from testing screens to exploring how to improve all interaction moments with government. We learned new research methods and devised more creative ways to share the insights within our organizations. We took a more holistic approach and also tried to get to know the person behind the user.

We worked smarter. In order to scale up, we started to bundle and archive our insights. We started to get better organized and research positions for that logistics side came up.

And then some of us started getting occasional phone calls from stray policy officials who needed to take make sure people could ‘do’ their policies. “I got your name through, maybe you can help me?” We certainly can.

And so with the insights from the living world, we went deeper and deeper into the system world.

So let’s dream on. Where do we go from here? What do I wish for our field?

More quality

To better understand the whole living world, both the individual as well as the collective side, we need to improve our work. For this, we need more diversity in research roles. The field is so large that no one is good at all the research methods we need. Neither is necessary. In the past, we had research teams consisting of one person doing everything, but that is no longer possible. A good research team includes usability researchers, strategic researchers, customer journey experts, behavioral scientists and more. We must embrace the full range of research methods.

Scholars from Sneek think with DUO

Diversity also says something about who we are ourselves. I still too often see a very homogeneous group when I look around. Especially we, who want to test the bias of the government, must also know our own bias. We have far too few people of color on our teams. One in 5 people has a visible or invisible disability, but looking around now, it seems that everyone has an invisible disability. Our teams are not diverse and that is a problem. As a result, we have too many blind spots that affect how we do our work.

Talking to people

I need to get something off my chest. It really should no longer be a problem to speak to respondents. Really. Come on. This is still far too often a problem in organizations. “Then what are you promising them?” “No, the GDPR.” “It has to be efficient.” Human contact, by definition, is not efficient; indeed, it gets better the less efficient it is. We can only do our job if we are allowed to have real contact with citizens. We shouldn’t put up with this stuff anymore.

Scaling up

If we really want to improve services to citizens, we need to expand our activities. It’s great that more and more development teams want to do usability testing in a sprint, but now imagine if all the teams in your organization wanted to do usability testing in every sprint? How are you going to manage that?

So we need to invest more in the organizational side. Contact with people does not have to be efficient, but we can organize our research work efficiently. This requires a different approach for most of us. The creative and human side is strong among most of us, but now it is time to embrace the blue side we know so well in civic service again as well.

Making policies and services together

I see more and more collaboration between policy and implementation, and research is the basis for that. I hope this becomes standard operating procedure in government. So that policies are based on insights from users and along with the developed services are always tested with citizens.

For this we need to stop that waterfall. I know I am raising a familiar point with this, and that we often complain about it. But we are not helpless on the sidelines. We can help stop the waterfall.

Sharing insights = working in the open

As far as I’m concerned, the best way to stop the waterfall is to share your work. Share citizens’ stories. Share how you do your work and what it brings. Also share the moments when things are not going well. In particular, stories about research insights getting stuck in a cumbersome process or on a system that is already finished help us understand how to work differently.

It often falls short of making a good shareable story as well. We are already so busy. To illustrate, I have spent about 1 day a week for years on this blog and giving and sharing presentations. And even now, on Friday afternoon, I am typing out my presentation from yesterday. I do this because I know it helps our profession move forward. Join me and contribute. After all, I only know what I happen to experience and see in Groningen here. Together we can learn much more.

It’s not real until it’s real

It is great that we are conducting pilots, living labs and experiments. But only when a citizen actually experiences better service, we can get coffee. Therefore, it is not strange to spend as much time sharing your work and following what happens with it as you do the actual research. What good is it if you put in all that effort and then nothing happens with it?

Let us strive to do full circle research. We start with the living world, with the citizen, of course. With the insights, we enter our organization and climb up the waterfall. We can. We work with others to adapt processes and systems, to adjust policies and, if necessary, legislation. We then re-examine how to translate those adjustments into individual experiences.

Full circle research begins and ends in the living world.

Full circle researching begins and ends with the citizen.

When I became a civil servant, my then manager Theo said, “Maike, it’s going to take a very long time for you to change anything in government. But if you succeed, you will have really accomplished something.”

So let’s start continue.

Continue reading?

This blog is full of tips on how to research and get your organization on board. For example:

Categories
How do you do research? Promoklip

Fingers on the keyboard

Writing an academic article on government services is very different from writing a blog. I have noticed that by now. I just sent the draft of my second article to my supervisor and it feels great.

While writing, I had a lot of help from other researchers who shared on Instagram, Youtube or on their blog how they go about it. So I do something in return of course.

In this blog, I show my writing process, how I prepare and what tools I use.

An update in your mailbox every month about my research? Subscribe to my Dutch newsletter.

A different ball game

I am not writing a book, blog or newspaper article, but a scholarly article. This is a very functional text and very different from what I am used to. You always write for your reader, and in this case my reader wants to know as quickly as possible, “what knowledge is in this piece and can I use it for my research?” There are rules associated with writing a scientific paper, which may also vary by research field.

For example, articles have a set structure: introduction, research question and aim, method, findings, discussion and conclusion. This took a lot of getting used to at first. If I shared a nice anecdote in the discussion, my promoters mercilessly moved it to the findings, or worse, deleted it completely from the article. If I gave even a small hint of the findings and what you could do with them, it was moved to the discussion. But as I plow through many articles myself, I am grateful for that logical format.

Photo taken from above of my desk with laptop and all sorts of notes on paper
Notes ready, tapping away

In April 2023, based on the data that 7 years of blogging gave me, I wrote an empirical article. Now I’m working on a conceptual piece. I now have a little more freedom to shape the article, but that also makes it immediately more difficult. The argument for my conceptual framework determines the structure of the article, so that argument must be excellent.

I use two articles as a guide for how to build such a conceptual piece. I make a theoretical synthesis of concepts found in the fields of design, services and public administration (Jaakkola, 2020). I bring these concepts together like an architect building a house (MacInnis, 2011).

Building arguments

PhD research is actually an education: with every step you take, you must first learn how to take it. So I took two courses to help me with writing.

  1. Philosophy of Science at Erasmus University. I learned what makes a good argument and how to check that the structure is sound. We had to analyze others’ texts and dissect them completely.
  2. Creative tools for scientific writing. For example, how to create overview for yourself, how to extract the main message from a paragraph and write towards it, and how to draw the reader into a text and hold their attention. And also: how to avoid writing those scientific sentences that seem like monstrosities. So that, even though it contains a lot of knowledge, it is also a bit of fun to read your article.

My writing process

After all this preparation, nothing stopped me from keeping my fingers on the keyboard. In January, I had three weeks, nonstop. I had also lost my voice, so I really couldn’t plan anything else.

During the second course, I wrote the introduction to the article, describing the research question, the purpose of the article and the approach to arriving at answers. This introduction covered about 500 words and I discussed it at length with my supervisors. After some scraping, I also sent it to Servsig, a service conference in June. It would be nice if I could give a presentation about the article there; I’ll hear about that later in the spring.

I had already done a lot of reading and made notes. As a library for scientific articles, I use Zotero, which is also useful for keeping track of references while writing. I read through all my notes again. I then created a visual outline in Miro with the main arguments and figures I was going to use, as well as the flow from one point to another.

Picture of my screen with all the figures while holding a tasty cup of coffee in front of it so you can't read everything
Expanded Miro board with all figures

So the real writing began quite functionally. I started with the main research questions and built the answer step by step in blocks. I added the main sources.

Then I worked out the blocks point by point. Each sentence I placed on a separate line with a number in front of it. The sentences were short and staccato. The important thing was that the argument was clear, and if it was, I marked the most important line. That was the point I was trying to make. I shuffled back and forth some more, looking critically to see if all the sentences were building toward that conclusion. And then I moved on to the next block.

I want to publish in an English-language journal, but I don’t really think in English yet. So I started in Dutch and translated sentences into English when I was satisfied with the structure. I used Deepl as a translation assistant to help me.

Making beautiful sentences

Now it was time to do what I used to call writing: make beautiful sentences. I removed the numbers and wrote the sentences together. Some sentences I added together. Other sentences I worked out and became as many as three sentences. I divided the information into paragraphs and again marked the main message for each paragraph, building toward the conclusion of the entire block. The sentences were much less staccato, but carried full-blown information. And of course I added all the references of the sources I relied on.

For editing, I used Edit.gpt. Mainly for English grammar, but also to improve overall flow. Some adaptations I adopted, others I did not. Then I checked that the sentences ran smoothly and were not too long. I did this by reading the text aloud.

When that was all done, I moved on to the next block. After completing all the blocks within a segment, I read through everything a few times. I watched the transitions from one point to another. I checked that all the research questions were answered and that the whole thing invited further reading.

And so last month I wrote a whopping 8,000 words.

Photo of my desk and window, with my legs stretched out in the window frame
8,000 words later

This writing is very different from typing a blog. At first, I had to get used to it tremendously. And it required an awful lot of discipline. With a blog, I usually wait for inspiration, but writing an article went step by step. In December, I resolved to take one step every day, or say, 300 words, so that I would have a draft ready by the end of January. And it succeeded. Even on days without inspiration.

Now what I wrote goes “into the mix. That means receiving feedback from my promoters. They will ask a lot of questions about the argument and rearrange things again. Some of the blocks also need better elaboration, but after 8,000 words I had a toasty brain. I am very curious to see what they come back with.

And about the content itself, because I understand that you are also extremely curious about that, there will be a blog later too :).

References

Jaakkola, E. (2020). Designing conceptual articles: Four approaches. AMS Review, 10(1-2), 18-26.

MacInnis, D. J. (2011). A framework for conceptual contributions in marketing. Journal of Marketing, 75(4), 136-154.

Categories
How do you do research? Not part of a category Visual working

How we can see an automated decision

With increasing talk of algorithm supervision, it is important to know how a third-party study of algorithms can be conducted in government. Preferably in such a way that day-to-day implementation processes are not disrupted but according to the principle that government shows itself.

Since January, I have been working with a very nice group to design a work method for this. We focus mainly on the fixed, ‘dumb’ algorithms and not yet on self-learning algorithms. In this blog, a behind-the-scenes look at our approach, initial observations and two questions for you.

I can just see it

When I had just finished the study The compassionate civil servant, I was sitting with Marlies van Eck on a terrace in Utrecht toasting. She mentioned that she had a follow-up idea to her dissertation (on legal protection in automated chain decisions). Her main conclusion was that, as a lawyer, she could not say whether citizens were sufficiently protected because she could not see the algorithms. The government was a black box, so how can you test whether the decisions it makes are right?

Over my beer I said “but, how weird, because at the Executive Agency of Education, where I worked, I can just see them’. Of course, I immediately began to doubt myself when I said that, but I looked up the photo interview of Cees-Jan talking about decision rules from which the code is then derived. And I mentioned that we were also working to derive the calculation tools on duo.nl from these decision rules so that, as a student, you can simulate how the computer will decide in the future how to pay off your student debt.

Opening and viewing

I am reminded of our conversation frequently because “being able to see the algorithm” is central to this research.

Marleen Stikker writes in her book The Internet is Broken that you only really own something when you can open it up to fix or change it. If you can’t, as with most contemporary technology, the device owns you. Can we open up government’s computers and see how they decide and possibly change that if necessary?

This area is still relatively unknown. The Dutch Tax Service has developed a method for making algorithms testable and explainable. The State Audit Office and accounting firms are preparing for this new task. The General Audit Office has developed a review framework. Recently, the State Council issued a report on automated law enforcement, among other things.

But what is suitable for a financial expert may not work for a researcher who wants to do a legal check. It is also tedious for an implementing organization when different disciplines and organizations consider the use of algorithms at different times and burden the implementation with questions. Both for researchers, and for implementing organizations, a more integrated approach is important.

Say no more. So with a group that I spontaneously get imposter syndrome from, we started in January to experimentally design a method for this. The group: Marlies van Eck, Steven Gort, Abram Klop, Robert van Doesburg, Mariette Lokin, Carlijn Oldeman, Giulia Bössenecker and me. Colleagues from the Executive Agency of Education and the Social Secutiry Bank are cooperating and offering us their automated decisions as trial material to design the method on.

What we want to achieve

Allright. We want to develop a working method for conducting third-party research on a government organization’s use of algorithms. To this end, we have formulated a thought scheme:

  • We study how the use of algorithms can be examined simultaneously legally, financially and model-wise[topic].
  • because we want to know which discipline would like to see which research questions answered[rationale].
  • so that we understand the least burdensome approach to a multidisciplinary and comprehensive assessment[significance].

To keep the research sufficiently concrete but also manageable, we propose that the working method should meet the following four objectives:

  • the working method allows a lawyer to make a judgment on the legality and propriety of the system,
  • the working method allows a data scientist/ information scientist to make a judgment about the quality of the system,
  • the working method enables an accountant or internal controller to make a judgment about …[to be completed, see also help question at the end of this blog],
  • the working method is suitable for repeated use in various public organizations.

Designing the method

In a series of on-site making days, we will work with a government organization. Lucky for me: the first 2 making days are at the Executive Agency of Education in Groningen where I live.

To arrive at a working method that can stand on its own within a few months, we take an embedded and iterative approach. But we must remember to design ourselves out of the method as well, and thus critically examine our bias. We learn by doing: we dive into an algorithm and record our process. Then we reflect on our process and make it explicit. What comes out of this is potentially the method we want to develop.

This is how we shape the creation process:

Outline of the design process divided into the 4 making days and intermediate actions

Together with the Executive Agency of Education, we chose a few automated decisions that are part of the big decision whether or not a student will receive a student loan: the nationality test, the age test and the partner test. Beforehand, we were given the sets of decision rules of these three tests to study firmly.

We decided to look from three perspectives or levels of abstraction. Different questions are important for each perspective.

Perspective 1 is the algorithm itself. How does the decision come about?

  • What are the decision rules dealing with the nationality test?
  • What sections of the law are the decision rules based on?
  • How are the decision rules programmed?
  • How are they included in the work instruction?
  • What data is needed and what are sources?
  • How is the decision explained to the student (user) in personal and general communication?
  • What interaction does a student (user) have with the algorithm and how does this influence the decision?

Perspective 2 is the creators of the algorithm. How does the algorithm come about?

  • Who (what roles and also individuals) are involved in the creation of the algorithm?
  • What considerations were made in the creation of the decision rules?
  • On what basis do these individuals make these trade-offs? What personal bias is there?
  • Is there documentation of this creation, and if so, what is recorded here and for what purpose?
  • What is implicit and difficult to make explicit here? In other words, what do we not know (yet) or cannot know (anymore)? How does this come about?

Perspective 3 is ourselves, the supervisors so to speak. How does the method come about?

  • How did we proceed? What questions did we ask? What worked well and what didn’t?
  • Which questions belong to which discipline (lawyer, accountant and information scientist)? Is there any overlap? Were all disciplines able to get their answers? How do they reinforce each other?
  • What knowledge and expertise do we have that allows us to ask these questions?
  • Can we reproduce our process next time? What would be different?

‘How crazy it is here’

The first day of making was wonderfully chaotic. We were flying in all directions, asking all kinds of questions, and Jean and Cees-Jan from the Executive Agency of Education were infinitely patient to answer all questions. Sometimes we ended up in heated discussion about the way the organization was organized, only to conclude after an hour that that shouldn’t matter at all for the method. Or as Steven so aptly noted, “I couldn’t care less how it is put together.” :’)

After this day of making, Marlies wrote a first blog with her observations and I made a video for the scientific guidance committee and the sounding board group.

Video (in Dutch) showing the experience of the first day of making at the Executive Agency of Education.

The second day of making went a lot more structured. In preparation, I visualized on a digital board all the steps from law to decision. I asked Cees-Jan if he could fill in everything from another decision (the partner test). This was not so simple because the information and docs had to come from all sorts of nooks and crannies of the organization.

I printed what I got (or left a step open) and put the steps on the floor. Underneath I put 3 long sheets of paper for the questions the lawyer, accountant or developer could ask. Perspective 1 and 3 are below each other. We took all afternoon to fill them out and see if we could get answers directly from Cees-Jan and colleagues (perspective 2).

An impression of what that looked like:

In two weeks we will have make day 3 at the Social Security Bank. That will be uncharted territory for me, fun! We are going to look at the age and partner test in the state pension.

In preparation, I digitized the large board we made on the ground at the Executive Agency of Education. It became a version with questions and a blank version for the Social Security Bank colleagues to fill in themselves. We are going to test this prototype and see if this is already a tool for self-assessment to show as an organization how your algorithm comes about and makes a decision.

Through this miro board, you can better zoom in on the prototype. You can also post comments here, and feedback is welcome!

Prototype of the working method we will soon test with the Social Security Bank

Two questions

You can see that the accountant’s perspective has no questions yet. Here we are still searching. Auditors are very busy around February and March because of wrapping up the previous year. So we haven’t been able to include much input from that side. Do you know more about this or know someone who can help, let me know.

And of course: feedback in general. In this blog, I describe our making process and show the prototype. I’m curious what you think of this and how it would work in your organization. After design day 3, we would like to update our prototype and share it with organizations to use without us for self-assessment. So we want to learn from your findings in turn.

If you would like to help with this, please send me a message. You can do so through maike @ klipklaar.nl or the other known ways. Thanks!

Categories
(Un)understood citizens How do you do research? Visual working

The cash flow maze

Almost a year after the floods in Limburg-South, how is the damage compensation going? Have people already returned to their homes and “is everything back to normal”? Or not? In early March, I went with a colleague for three days to Valkenburg aan de Geul, Gulpen-Wittem and Meerssen to see for myself the aftermath of the floods. I took my camera with me.

In this blog, you can read about how we approached this visit, about my musings on these types of events in the future and what effect they have on citizens’ relationship with government.

The trigger

In January, Reinier van Zutphen, the National Ombudsman together with Jan, a colleague, paid a regional visit in Limburg as they often do. Among others, Reinier spoke with the mayor of Valkenburg, Daan Prevoo. He told him that with that aftermath, things did not go well at all. The following week, Jan asked if I wanted to go with him to Limburg on short notice for a bit longer to see exactly what was going on. I went along because I did research on the effects of gas extraction last year and we saw some overlap at first glance.

From Monday to Wednesday, we were guests of the three municipalities most affected during last year’s floods: Valkenburg aan de Geul, Gulpen-Wittem and Meerssen. Staff who are working a lot with the aftermath of the floods had created a program for us.

But first: what happened last year?

During the week of July 14, it rained tremendously in Belgium and Germany. This led to floods that then flowed into South Limburg. I found this clip from the youth news that shows well what residents had to deal with:

On July 16, after a crisis council, the cabinet formally declared the flood in South Limburg a national disaster. On BNR that day: “This means that the cabinet will put into effect the Damage Compensation Act. That way, victims will quickly get clarity on whether their damage will be compensated by the government, if their insurance does not cover it.”

But so it’s not going well. We went out for three days with employees of the three Limburg municipalities to get a picture of this ourselves.

Beforehand, Jan and I looked for all kinds of things. I visualized this on a timeline. Approximately what has happened and what bottlenecks do we already see based on what we find online?

Timeline with initial assumptions of the problems – click for larger

We used this timeline as a conversation starter on Monday. Our questions in this regard were:

  • Who are the key parties in the aftermath and how do they relate to each other? How do the most affected areas differ?
  • What are causes and effects that have happened in recent months? Are there chain reactions?
  • What are money flows and who has access to what?
  • What bottlenecks are there and do they have relationships with each other?
  • What do residents and business owners notice about this? How do consequences impact them and those around them?

On large flip charts we mapped out the answers. By the end of the afternoon, my head was full of all the money flows and dead ends in the arrangements. How do we make sense of this?

Real stories from real people

The following days we visited residents and business owners. I had my camera with me to immediately capture what we encountered.

For example, with a man my age in Valkenburg who will probably be living in a cottage with his girlfriend and baby until next year. He is caught between the insurer and the Netherlands Enterprise Agency (RVO), no one wants to get burned on the likely problems with his foundation. He is now receiving an allowance from a private fund to find out what the problem is so that things can move forward again.

Man points to the Geul behind his house and explains how in 7 minutes the water forcefully rushed into the house and ripped out the facade.

At a couple in Gulpen-Wittem who are still doing a lot of work in their home. After last June’s rising water, they again placed sandbags at the doors, the now sweetly babbling brook is flowing directly behind their home. They would like organizations to help them how to proactively protect their homes from future high water but there are no arrangements for that. They are for damage afterwards though.

Husband leans against the garden table as he talks about the interventions he wants to do around the house. The sandbags are still up against the house.

A gentleman in Meerssen who still lives in the garage because the house has not yet been renovated. He is left with a gap between damages and compensation of 80,000 euros that he has nowhere to invoice.

Man shows how he is temporarily living in the garage because the house is not yet habitable again after the floods last year.

All indicated that they had different expectations, perhaps naively, when it was said last year that the floods were a disaster and that the government was going to help them. A year later, that turns out not to be so simple.

How do we move forward?

I see two routes in front of me.

In the short term, I think we need to make the maze understandable. The maze of money flows and schemes that are there for residents and business owners but have dead ends where citizens get stuck. The stories we encountered are examples of this, these people are at the end of such a dead end and cannot go on.

How can the government design such a maze? Probably with good intentions, I know few rogues in government. But still: the government promises something, which is positive, only to design squishy bureaucracy. Why are we doing this? And how come it turned out this way? What can the National Ombudsman do in this regard?

Street in Meerssen where there is still a lot of tinkering and bulky trash and building materials on the street.

In the long run, I think it would be interesting to delve into future events related to this kind of climate conflicts. The people we spoke to, both resident and mayor, are all concerned with the next time this will happen. Because there is no question in their minds that this is not an incident. Something needs to be done in the Geul Valley.

Who is in charge? In part, if still possible, to prevent such major floods from happening again, to warn when they happen and to deal with the consequences quickly and appropriately?

In the hedge along the Geul, you can still see the washed-up grass hanging which was carried away with the flood last year.

Last year, I read many books about climate change and what lies ahead. What should citizens expect from government in this type of climate conflict? And what does this mean for how government designs itself?

This is still a new area for me. So I would love to hear who I can talk to further about that and what to read or listen to about it. Let me know.