Category: How do you do research?

At the Executive Agency of Education I did research into the target group: students, people integrating and employees of schools. I devised new methods for this, brought their stories into the organization and trained colleagues to do the same. Now I’m learning at the National Ombudsman how you can conduct government-wide research.

You will find out how to approach this in this line of research. I share methods and material. I share success stories and also failures.

Categories
How do you do research? Promoklip

Starting from the whole experience of your target audience

Foto door Aline de Nadai via Unsplash

For the past few months, I have been following a team in the government that is redesigning services to help people repay their debts to the government. I am looking at what helps and hinders them in making these services human-centered. To find an answer to that question, I must first have a good understanding of what the principles and activities of human-centered design actually mean.

So in a number of blogs in the coming period I will explore these principles. Earlier I wrote about what these principles and activities are. Today we are going to go in depth on the principle: “starting from the whole experience of your users. What is it, how do you do it, and how do you start? Of course, with examples!

What is “the whole user experience”?

I use the ISO standard 9241-210 on human-centered design of interactive systems for definitions. User experience, according to this standard, is the outcome of an interaction a person has with a technological system. This experience is influenced by previous experiences, attitudes and skills of the user. Thus, when we talk about user friendliness (usability), it is more than just making systems easy to use. It is about whether someone can achieve a certain goal effectively, efficiently and to their satisfaction by using a system, service or product.

When you design a service, you make choices: what will the user do and what can the system do? When we look first at what the technology can do, and then leave the rest to the user, we create ineffective services. Instead, the point is to keep as much complexity away from the user as possible. So the tasks for people when interacting with government must, as a whole, be meaningful to them.

Designing from the whole experience therefore means that as a government you take into account the goals people have, the opportunities they have, and short-term (e.g., comfort and enjoyment) and long-term (health and well-being) satisfaction.

In the literature, that is actual Service with a capital letter. That with your service someone can achieve their own goals, and thus create value.

Be-goals and do-goals

So at first glance, user experience seems to be about when someone reads a letter, logs into a portal or uses another system. But it goes beyond that. It’s about whether someone can achieve a goal effectively, efficiently and to their satisfaction.

You have to distinguish between be-goals, do-goals and tasks. No one’s goal is to call CJIB on Tuesday afternoon. That is a task. And that is part of a do-goal, which is to see if, because of an unexpected setback, you can still arrange to pay your debt. And that in turn belongs to a be-goal: that you want to be in control of your own finances and be able to get by every month. A task like “calling CJIB” can support a lot of different do- and be-goals. User experience is as much about these higher goals as it is about tasks.

So what is the whole service?

At DUO, young status holders who are both integrating and studying have to log into two different portals: one for their study loan, which is accruing, and one for their civic integration loan, which is counting down. This is because different departments at DUO implement different legislation: the Student Loan Act from the Ministry of Education and the Integration Act from the Ministry of Justice. DUO is no exception. It often happens that departments are the mirror image of the organizational rake.

Any organization designing a system will create a design whose structure copies the organization’s communication structure.

Conway’s Law, from Platformland by Richard Pope

So to make services that are good for people, you have to start not from your own structure, but from how the user experiences it.

The way the government now collects debt from citizens is the mirror image of how the government itself is set up. The Tax Office collects tax debts, the Benefits Service collects overpayments of benefits, and the UWV offsets debts against the benefits you receive from them. Most customer journeys the government uses assume one claim the government has with someone. While someone can have many claims at the same time with multiple organizations. That’s what you have to design for. And that can include different do and be goals. In the program, we use a spectrum that looks something like this:

A schematic representation of the spectrum "debt in government.

Different things are needed at different points on the spectrum. At one end are people who experience a great deal of autonomy, and at the other end are in need of duty of care or repression. The services that are developed, such as a statement of receivables and a payment plan, help people achieve their goals in their situation. By thinking from the whole experience, you can also examine which tasks and goals are not yet supported well enough, and thus come up with new additional services.

Services that support different aspects on the spectrum.

I am discovering that thinking from the whole experience of someone in debt, and then designing services for it, is one of the greatest strengths of the program I am following. By eliminating their own organizational logic, they saw how much overlap there can be in debt. And problems around debt easily spill over into other domains, such as health and performance at school or work.

Looking from the whole experience of the citizen makes everyone look very differently at the systems and processes that are in place, at organizational structures and at policies and political assignments. Those are no longer leading, but you can change them all to work from that whole experience.

You don’t start seeing it until you realize it, but when you do, you can’t really go back either.

How do you do this?

How can you work from this principle? The first two activities of human-centered design from the ISO standard provide the perfect starting point:

  1. Understand and specify the context of use. This context involves both the user and all other parties involved in the problem. Exploratory usage research allows you to identify the actual Service. My blog is full of methods you can use for this. But you can also consider sources of information such as CBS.
  2. Identify the specific needs of users and other stakeholders. These may include opposing needs that you later need to reconcile in the design. In short, map out the being, doing goals and tasks.

Tips for getting started

  1. First, take off your organizational blinders. This is the most difficult step. Realize that the structure of your organization does not necessarily reflect the reality outside the office. Shake off Conway’s Law.
  2. Map out the being and doing goals of your target audience. Use sources that look across domains, such as the Court of Audit or the Ombudsman. Or look at what universities and lectorates write about general topics such as “Poverty” or “Housing.
  3. Seek out experiential expertise. This may well be on a small scale. Walk along for a day with someone from your target group or with a professional who works a lot with your target group. I recently did that with a bailiff and learned to look at the subject of debt much more broadly than I did before. Invite a few people from your target group to the office for a cup of coffee and swap stories. Listen. It will inspire you to see how broadly you actually need to look at the topic and how much overlap there is with other “files.
  4. Then actively (and professionally) engage in the first two activities of human-centered design.
  5. Good luck!

Continue reading

  • What is human-centered design? All the principles and activities based on the ISO standard at a glance.
  • The being and doing goals come from this article: Hassenzahl, M. (2008, September). User experience (UX) toward an experiential perspective on product quality. In Proceedings of the 20th Conference on l’Interaction Homme-Machine (pp. 11-15).
  • Conway’s law and inspiration on making services from the whole experience I got from: Pope, R. (2019). Playbook: government as a platform. Ash Center for Democratic Governance and Innovation, Harvard Kennedy School, Cambridge, Massachusetts.
Categories
How do you do research? Promoklip

Making sense of your research data

Photo by Bud Helisson via Unsplash.

This summer I am immersing myself in all the observations, conversations and activities I have participated in over the past six months. Since February, I have been following the Clustering Government Collection program. In early July, I got the bright idea to take a look at the data I had collected so far. That was quite a lot. In this blog I will tell you how I collected this data, and how I analyze it.

If you want to follow my research, subscribe to my monthly newsletter.

Last month I wrote about how I set up the empirical study. As an action researcher, I observe what is happening but also actively participate. I am following a government program that is redesigning services in the debt domain based on human-centered principles. You can read this blog here: case selection done.

PhD research timeline. The unfolded gray area runs from February this year untill now, which is what I am focusing on this summer.

In that blog, you’ll also see that I divide my research into several phases. The outcome of one phase determines the approach to the subsequent phase. I determine the approach together with the participants who participate in the research. This summer I am focusing on the “unfolded” gray area.

You always start with a question

There is, of course, a reason why you engage in a particular group and context. You want to know something. This semester I have focused mainly on 2 sub-questions:

  1. How do organizations create a service together?
  2. What helps and hinders them in the human-centered design of such a service?
Almost vintage post-it with the research question so I am reminded of it every week.

Before I began this doctoral research, I had some experience doing research. This blog is full of it! This is mostly applied qualitative research for organizations. Over the past two years, I have read a lot about the requirements of science for new knowledge based on qualitative research. In particular, I looked at how data is collected and analyzed in ethnographic and participatory action research.

An important principle, for example, is triangulation: using different data sources that complement each other. If you observe something in behavior, discuss it in interviews and find it in the organization’s documents used, your story is better substantiated. This also applies the other way around: someone can say something, but then if you don’t observe it or can’t find it anywhere in the organization’s archives, well….

My data-gathering strategy

I collected data in several ways. Some were intended as input to another way of collecting data, which were then important sources for my analysis.

I kept observations each week in a diary. In all, I have 22 diaries, ranging from 4 to 10 A4 sheets. I joined meetings, had lunch with participants and talked with colleagues between activities. I wrote down what was happening and what I noticed.

In early April, I organized a workshop with the program’s core team. Everyone made a timeline with moments that were defining for them. We then exchanged experiences in a group discussion and responded to each other’s timeline. I made a start on a joint timeline during the discussion.

The observations and timelines provided the input for personal interviews with 13 colleagues involved in the program. The interviews were free format. We discussed the timeline they had made and I asked about their general ideas about what good services are and how to make them.

I myself collaborated on a sub-project. I kept my own reflections in the same diary. I already wrote the blogs Reframing leads to new solutions? and A day with a court bailiff about this project.

I used program documentation such as mail exchanges, meeting notes and vision documents. There is even a Whatsapp message in my data collection! I used these documents primarily as additional substantiation for observations or statements from the interviews.

How the sources support each other looks like this:

Visual overview of how I use different data sources in my research.

Through the pile of data

In early July, I temporarily stopped collecting data. Time to see if I could already formulate answers to the questions. In 2 ways I went through all the data thoroughly.

My research focuses on change, and this requires time. Therefore, I chose a process lens to look at all the data. I explain how this works in the blog Studying a process. It involves plotting my data on timelines to see the sequence of events and actions in order to find underlying mechanisms. This timeline approach is the first way I went through the data.

On the wall of my home office I made a large timeline of a few years on which I placed the defining moments of colleagues. I added stickers with their names when they talked about such a moment or period in their interview. Then I divided the timeline into periods that made sense to me.

Me this summer. If you know, you know.

One such period I experienced up close: the past six months. I made a detailed version of this by going through all my diaries and noting the important events per week. Events that I influenced got an extra black dot. Judging by the growth of the black dots, my role changed from passive observer to more active team member the longer I was involved in the program. That’s fine, and I’m glad I captured this so well and kept my reflections on it in the journal.

1988 unique codes

The second way I went through all the data is by encoding it. Almost all of my data consists of text. Even the visual data have been “translated” into text as colleagues talk about it in the group discussion and their interview.

Of the conversations I recorded in audio, I made a verbatim transcript. For some, that took some getting used to, because every um is in there. No one talks in neat full sentences. Often we search for words and come up with a new ending mid-sentence. I wanted to follow the thought processes and excursions. And I wanted to stay as close as possible to the exact wording of the participants.

For coding all the data, I use ATLAS.ti. On their website they have a handy explanation of what coding in qualitative research entails. In short: I go through each piece of text to see what it is about. This I give a label, a code in other words. An average one-hour interview gave me 125 codes. In total, a first round of coding all the texts yielded 1988 unique codes.

While coding the texts, I was constantly thinking about my research question. So I paid attention to whether participants told how they made services. What processes are there and what steps do they consist of? Each step is given a code. Do they use design methods, and which of these are human-centered? Everything I encountered was given a code. What are they running into? And what exactly helps them? You get the point: a code!

This is what that looks like: a piece of interview with multiple codes.

In addition, I also coded a lot “en vivo. This means that you use the literal words of the participant as code. You do this when you want to stay as close to the data as possible. The reason I do this is that for designers outside government, government is often a closed world with its own language. To apply design in a public bureaucratic environment, we also need a common language. That’s why I don’t want to “translate” myself already while coding, but use the language of the government itself.

Examples of this type of en vivo codes that can be very instructive for designers:

  • An individual administrative decision: this is a term from the General Administrative Law Act. It is an official legal decision that an organization has a legal mandate to make. For example, the decision that you are not entitled to an allowance and must pay it back. You can object and appeal against a decision. These are measures of legal protection for citizens. Such a decision is a core part of a government service.
  • A front portal: a type of meeting in which decisions are prepared by officials. There can be all sorts of different types of front portals. Sometimes a decision has to go through several tables and therefore through several front portals to those tables. Especially if different organizations have to decide something.
  • the Minister’s bag: every weekend the minister is given the so-called weekend bag with memos for information or with something that needs to be decided. For civil servants, this bag represents a kind of deadline you work toward.

These en vivo codes can be combined to form a category, for example, a category describing how government decision-making works when you redesign services. Or a category on the legal aspects of service delivery. Some codes can also belong to both categories. This is something I’m going to work out over the rest of the summer. Then in a codebook, I’ll keep track of what main categories I have and what their definitions are. This will also allow me to start with a preliminary answer to my 2 questions: how do these organizations make a service together, and what helps and hinders them in designing it human-centered?

So how to proceed?

In early September, I will first discuss all the insights with the participants who participated. In qualitative research, this is called the member check. This is a crucial step in action research because you are working with participants in their own context. Do they recognize my preliminary answers? Do they have additions? Have I overlooked anything? At the same time, this is also an intervention because we reflect together on our way of working. This influences how the research will proceed. What do they see as the next step? We also discuss this together.

Are you also curious about the first insights? On September 12 in The Hague, I get to talk a little about my research at an event hosted by PONT with Kustaw Bessems of the podcast Stuurloos. You can sign up here!

Continue reading?

  • Coghlan, D. (2019). Doing Action Research in Your Own Organization. Sage.
  • Miles, M. B., Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook. Sage.
  • Saldaña, J. (2021). The coding manual for qualitative researchers. Sage.
  • Small, M. L., Calarco, J. M. (2022). Qualitative literacy: A guide to evaluating ethnographic and interview research. Univ of California Press.
Categories
How do you do research? Promoklip

Case selection done

Foto door

I have been working on my doctoral research for more than a year and a half now, and am now working on the emperical part. In this blog, an update.

You’ll read about what I research, how I select the case study and which one it turned out to be. At the end a timeline of what the next time looks like. To stay in the travel theme, the route is now pretty much set.

Status quo of government services

I often show the figure below to explain how government organizations make services now, and thus what I want to change with my action research. It is a oversimplification of a month from my own relationship with the government.

I kept track of what I experienced with the government and what is involved in government organizations. It struck me that most organizations create and offer government services as if they have a 1-to-1 relationship with a citizen, but in practice that is not the case. Someone is constantly switching with different organizations to do all sorts of things. And the outcome of one service often impacts another. The question is whether the collective values we start with at the top still work out that way for the individual. The answer: well, no.

I wrote this blog about it that kicked off my doctoral research.

The status quo when it comes to services the government makes for people.

How can this be done differently? And how can we learn that as government organizations?

To examine how government organizations are growing in making services that are good for people, I think it is important to be as close to such a service as possible. I don’t want to explore it in theory or in an isolated innovation lab. No, as close as possible to practice, with my paws in the mud, in the middle of the status quo. That’s where it has to happen.

By zooming in on one government service, I can examine how the organizations involved are developing themselves in human-centered design. Since February, therefore, I have been following the Program on Clustering Government Collections (CRI). They are redesigning the status quo around debt issues. The motto: one citizen, one payoff capacity, one government collection. This video shows how they do it.

How do you select a case study?

Actually, I have only known for a few weeks that CRI and paying off debt is really going to be my case study. The first few weeks I spent at the program were spent exploring their practices and formulating case criteria from the literature. In qualitative research, it is important that you do that very well selectively and can substantiate it.

In the end, I arrived at the following criteria:

  1. The organization/team redesigns a government service, in this case “paying off your debt(s).
  2. This service is provided by multiple organizations (service bundling).
  3. This service is in the social domain and is typical of a national semi-digitized public service.
  4. There is room to adapt existing structures, roles and processes (and organizations).
  5. The service is redesigned based on human-centered design principles (or there is room for that).
  6. I may participate as an action researcher.
  7. The team goes “full circle”: society, politics, policy, implementation and citizens are involved.
  8. There is organizational and political will for change.
  9. The organization/team supports and participates in an open approach.
  10. There is funding for the case.

I must honestly confess that I also cheated a bit. After all, I already knew the program and intuitively knew: something interesting is happening here. So I made a list of why I really believed this should be the case. I then turned CRI everywhere into “the organization.

I ended up with the above list which was far more ambitious than I could have hoped for a government case I would have access to as a researcher. If I didn’t know this existed, I wouldn’t have known this was possible, so to speak. And then my supervisor said, “Well, Maike, in science you also have such a thing as ‘opportunity,’ so go for it!”

If I adjust the status quo to what that would look like in CRI logic, you get this artist impression:

Broadly explored how organizations can work together to create services that are good for people. Services that allow citizens to get a grip on their debt(s) and prevent them from accumulating.

I have kept a journal of observations for 22 weeks now. Along with a series of interviews, I will analyze them this summer. That sets the stage for a second cycle of research that I am shaping with the team beginning in September. Soon I will write a blog on how I have collected the data so far and what my analysis strategy is.

Focus!

In the first month of my PhD, my supervisor explained to me that you start with big plans that get smaller and smaller, until you only have a postage stamp left. That’s what you write your disseration on. He sent me this comic that illustrates it perfectly. Now at first I thought that was just annoying because of course I was going to solve all the government’s problems, granted.

Meanwhile, I do need focus, so in June I decided to devote my entire PhD to creating services around this theme: paying off a debt to the government.

This does not necessarily mean that I am going to explore this from the CRI program all the time. I prefer to look broadly from this program this year and switch perspectives sometime during 2025. Then I want to zoom in deeper on a government organization that is part of that broad debt approach. This is what that might look like schematically:

Zooming in on the perspective of 1 government organization: how can it evolve to make services that are good for people from the whole?

It does mean that I won’t switch to other services that the government also makes. My insights on how government can make good services will probably apply to those other services, but it gives me peace of mind to go into the summer with an increasingly concrete stamp in sight.

Are you also engaged in services around “paying off debt” and would like to share your insights with me? Yes, please! Send me a message at maike@klipklaar.nl. I will also share it with the CRI program team.

Timeline

So the research is taking shape. And I have to, because in October I will be halfway there – in terms of time. This is what my timeline looks like right now:

The picket stakes are in the ground. Nice!

Want to follow my research and not miss anything? Then subscribe to my monthly newsletter.

Categories
How do you do research? Promoklip

Studying a process

Poster van een van de principes van goede diensten uit het boek van Lou Downe

In my research, time plays an important role. Or rather, processes that take place over time. To get a better handle on this, I recently took the Process Research course at the Vrije Universiteit of Amsterdam.

In this blog, I share how I plan to use this way of looking at data and knowledge in my research. I see 3 possibilities for that and I will work them out in the coming time.

Actions and processes

But first: what is process research? If you have been following this blog for some time, you may be thinking, “hey, but you did action research, right?” Yes. In fact, you can combine the two perfectly.

If you want to understand something, you have to change it. This is a famous quote from Kurt Lewin, the founder of action research. With action research, you work with participants in real situations. Together you try to change something and reflect on it. Things just rarely go from A to B overnight; it takes time. You can study this change in 2 ways.

  1. A variants approach looks at the differences between A and B were they 2 separate pictures and thus explains the change.
  2. A process approach looks at how A becomes B. This is the moving image, the whole film. It is precisely the fluid process between A and B that is interesting here. This process lens allows you to better understand the mechanisms that make change possible.

Action research and process research both assume that organizations are made up of change and that the best way to examine this is from the perspective of the inside. That suits me perfectly, as a public servant who wants to explore how government implementing organizations are transforming into service organizations.

In the Process Research course, I learned about the theory behind this method of research. And in what ways to collect data and analyze it with a process lens to arrive at knowledge. I found it super interesting! If you would like to know more about this, or are looking for reading tips, send me a message.

My research consists of 3 layers and on each layer I think I can use this process approach well. I look at what good services are, how organizations make them and how, in this research, I learn about this along with you.

A good service is a process

Lou Downe writes, in the best book on good service I ever read, that good services are verbs. Often in government we call our service a registry, a portal or – even worse – an abbreviation that leaves no one actually knowing what the person on the other end, the citizen, wants to accomplish with it. What they wanted to do.

I have the poster myself at home!

A good service is a process with all kinds of actions in it. We visualize this in a customer journey. You can also link the organization’s processes to this. It’s called a service blueprint. So you can see over time how someone experiences a particular government service and what your organization does and organizes for it.

In my research, I look at what constitutes good service. I am doing that this year in a case study at the CJIB, in the Clustering National Debt Collection program. Here I look at what a good service looks like for people who have to pay a debt to the government.

I also look one layer above this. I look at how the government is redesigning this service and thereby changing it.

Redesigning a service is also a process

I participate in the team working to improve this service. I do that because I want to learn how organizations get better at making services that are good for people.

To track the growth in government in this, I record events over time. Together with the team I follow, I reflect on past events. For example, the other day we all made a timeline of the important moments in the project. In interviews, I then ask how people proceeded around those events. What the strategy is, what they care about and how they make choices. I’ve blogged about this method before: visual interviewing.

I also made my own timeline, because I am part of the team myself and think/work with them how the service can be better. This is my timeline, redrawn.

Drawing of my timeline at the CJIB

I’m still figuring out how to properly anonymize the data, but this is a little too much of it, I think, haha.

Later, I will put the timelines and stories side by side and you can see from multiple perspectives how the service is created and the organization(s) grow. Together we can then see what mechanisms are involved. And, of course, I hope to learn lessons that can work in other places.

My promotion is also a process

You can follow this research in a transparant way. I regularly show what it looks like behind the scenes. I do that through this blog and through a monthly newsletter. I get regular feedback from you on what I do and I find that very valuable. This open way of working, I call it open action research, is also a process and this is the third layer in my research.

The way I learn, collect and analyze data and translate this to new knowledge together with your input, can also be drawed out as a process. I am now writing a paper on my conceptual framework for the research. Secretly, all kinds of building blocks of this have been on this blog for a long time. Sometimes I don’t even realize it myself. Retrospectively, you can read back the entire thought process, the knowledge in the making, here. For example, this process looks like this:

Schematic representation of how I go from idea to outcome through the open-ended approach.

In the coming time, I will work on this approach and, with the team and together with you, I will analyze the initial findings. I am very curious to see what we will learn from it!

Continue reading?

  • Coghlan, D. (2019). Doing Action Research in Your Own Organization. SAGE.
  • Langley, A. (2007). Process thinking in strategic organization. Strategic Organization, 5(3), 271-282.
  • Tsoukas, H., & Chia, R. (2002). On Organizational Becoming: Rethinking Organizational Change. Organization Science, 13(5), 567-582.
Categories
Not part of a category How do you do research? Promoklip

Full circle research

Foto van een waterval die echt super steil is
Foto door

Yesterday, more than 100 researchers working in the Dutch government met in Utrecht. The government-wide research community held its first real-life event, and the organization had to ask participants to attend with just 5 persons per government organization. If you had told me this 10 years ago, I wouldn’t have believed you. So many researchers already work in government. Super.

In this blog, I share my presentation: about the context in which we work and why researchers are necessary for good service delivery. We look back at how far we’ve come, and also ahead: what will it take to let the lifeworld of citizens guide the delivery of government services? I call that doing full circle research.

An update in your mailbox every month about my research? Subscribe to my Dutch newsletter.

The circle of democracy and service delivery

In my research, I use a division between the system world and the living world, with both a collective and an individual side. This creates 4 quadrants. On the collective side, Cabinet, together with representatives of the people, shape their vision of society. This vision is then translated into laws, policies and services for individuals. Only after interaction between citizens and government does this become a reality in the context of the people themselves, or in the living world. And also in the living world, individuals group together and try to implement their ideas about society, thus coming full circle.

Drawing the circle of democracy and service delivery
Drawing the circle of democracy and service.

Of this circle, people experience mostly the individual side. “People don’t experience policies, they experience services,” says researcher Sabine Junginger (2016). That means the job of public service providers is to translate collective value into individual experiences that are also valuable.

Want to know more about this collective and individual side? Then read the blog Executive and service provider.

That’s not easy. The practice of the system world now is often a big waterfall. It goes clockwise in a circle, from policy to service and pours out over citizens and society. If you disagree, you can show it in your voting record.

It may be different if we collect citizens’ experiences and introduce them, counterclockwise, into the system world. Those experiences can shape how we offer services and how we design services at all. For the policies enacted for it, and yes, perhaps also for the legislation and associated ideas about interventions in society and the effect it will have.

We have to turn it completely around. And we can, because that’s exactly our job as government researchers.

We extract experiences from the lifeworld. We have several methods and techniques to do this properly. Moreover, we are able to share these stories within our organizations in a way that motivates colleagues to take action.

We have shown that over the past 10 years.

How it began

In most organizations, it started with usability testing of websites and other screens. At least for me, too. I started working at the Executive Agency of Education in 2013. One of the first things I learned was how to set up and run a usability test. I visited schools with my laptop under my arm. In the office, I showed videos to colleagues and talked about how students or school employees experienced our digital services.

Look, so cute: my first steps in the research world.

In 2017, I started my blog. I shared what I was learning and what we were trying out. You also started sending me your experiences.

I saw that we were growing as researchers. We went from testing screens to exploring how to improve all interaction moments with government. We learned new research methods and devised more creative ways to share the insights within our organizations. We took a more holistic approach and also tried to get to know the person behind the user.

We worked smarter. In order to scale up, we started to bundle and archive our insights. We started to get better organized and research positions for that logistics side came up.

And then some of us started getting occasional phone calls from stray policy officials who needed to take make sure people could ‘do’ their policies. “I got your name through, maybe you can help me?” We certainly can.

And so with the insights from the living world, we went deeper and deeper into the system world.

So let’s dream on. Where do we go from here? What do I wish for our field?

More quality

To better understand the whole living world, both the individual as well as the collective side, we need to improve our work. For this, we need more diversity in research roles. The field is so large that no one is good at all the research methods we need. Neither is necessary. In the past, we had research teams consisting of one person doing everything, but that is no longer possible. A good research team includes usability researchers, strategic researchers, customer journey experts, behavioral scientists and more. We must embrace the full range of research methods.

Scholars from Sneek think with DUO

Diversity also says something about who we are ourselves. I still too often see a very homogeneous group when I look around. Especially we, who want to test the bias of the government, must also know our own bias. We have far too few people of color on our teams. One in 5 people has a visible or invisible disability, but looking around now, it seems that everyone has an invisible disability. Our teams are not diverse and that is a problem. As a result, we have too many blind spots that affect how we do our work.

Talking to people

I need to get something off my chest. It really should no longer be a problem to speak to respondents. Really. Come on. This is still far too often a problem in organizations. “Then what are you promising them?” “No, the GDPR.” “It has to be efficient.” Human contact, by definition, is not efficient; indeed, it gets better the less efficient it is. We can only do our job if we are allowed to have real contact with citizens. We shouldn’t put up with this stuff anymore.

Scaling up

If we really want to improve services to citizens, we need to expand our activities. It’s great that more and more development teams want to do usability testing in a sprint, but now imagine if all the teams in your organization wanted to do usability testing in every sprint? How are you going to manage that?

So we need to invest more in the organizational side. Contact with people does not have to be efficient, but we can organize our research work efficiently. This requires a different approach for most of us. The creative and human side is strong among most of us, but now it is time to embrace the blue side we know so well in civic service again as well.

Making policies and services together

I see more and more collaboration between policy and implementation, and research is the basis for that. I hope this becomes standard operating procedure in government. So that policies are based on insights from users and along with the developed services are always tested with citizens.

For this we need to stop that waterfall. I know I am raising a familiar point with this, and that we often complain about it. But we are not helpless on the sidelines. We can help stop the waterfall.

Sharing insights = working in the open

As far as I’m concerned, the best way to stop the waterfall is to share your work. Share citizens’ stories. Share how you do your work and what it brings. Also share the moments when things are not going well. In particular, stories about research insights getting stuck in a cumbersome process or on a system that is already finished help us understand how to work differently.

It often falls short of making a good shareable story as well. We are already so busy. To illustrate, I have spent about 1 day a week for years on this blog and giving and sharing presentations. And even now, on Friday afternoon, I am typing out my presentation from yesterday. I do this because I know it helps our profession move forward. Join me and contribute. After all, I only know what I happen to experience and see in Groningen here. Together we can learn much more.

It’s not real until it’s real

It is great that we are conducting pilots, living labs and experiments. But only when a citizen actually experiences better service, we can get coffee. Therefore, it is not strange to spend as much time sharing your work and following what happens with it as you do the actual research. What good is it if you put in all that effort and then nothing happens with it?

Let us strive to do full circle research. We start with the living world, with the citizen, of course. With the insights, we enter our organization and climb up the waterfall. We can. We work with others to adapt processes and systems, to adjust policies and, if necessary, legislation. We then re-examine how to translate those adjustments into individual experiences.

Full circle research begins and ends in the living world.

Full circle researching begins and ends with the citizen.

When I became a civil servant, my then manager Theo said, “Maike, it’s going to take a very long time for you to change anything in government. But if you succeed, you will have really accomplished something.”

So let’s start continue.

Continue reading?

This blog is full of tips on how to research and get your organization on board. For example:

Categories
How do you do research? Promoklip

Fingers on the keyboard

Foto van persoon die de vingers op het toetsenbord houdt haha
Photo by Daniel Thomas on Unsplash

Writing an academic article on government services is very different from writing a blog. I have noticed that by now. I just sent the draft of my second article to my supervisor and it feels great.

While writing, I had a lot of help from other researchers who shared on Instagram, Youtube or on their blog how they go about it. So I do something in return of course.

In this blog, I show my writing process, how I prepare and what tools I use.

An update in your mailbox every month about my research? Subscribe to my Dutch newsletter.

A different ball game

I am not writing a book, blog or newspaper article, but a scholarly article. This is a very functional text and very different from what I am used to. You always write for your reader, and in this case my reader wants to know as quickly as possible, “what knowledge is in this piece and can I use it for my research?” There are rules associated with writing a scientific paper, which may also vary by research field.

For example, articles have a set structure: introduction, research question and aim, method, findings, discussion and conclusion. This took a lot of getting used to at first. If I shared a nice anecdote in the discussion, my promoters mercilessly moved it to the findings, or worse, deleted it completely from the article. If I gave even a small hint of the findings and what you could do with them, it was moved to the discussion. But as I plow through many articles myself, I am grateful for that logical format.

Photo taken from above of my desk with laptop and all sorts of notes on paper
Notes ready, tapping away

In April 2023, based on the data that 7 years of blogging gave me, I wrote an empirical article. Now I’m working on a conceptual piece. I now have a little more freedom to shape the article, but that also makes it immediately more difficult. The argument for my conceptual framework determines the structure of the article, so that argument must be excellent.

I use two articles as a guide for how to build such a conceptual piece. I make a theoretical synthesis of concepts found in the fields of design, services and public administration (Jaakkola, 2020). I bring these concepts together like an architect building a house (MacInnis, 2011).

Building arguments

PhD research is actually an education: with every step you take, you must first learn how to take it. So I took two courses to help me with writing.

  1. Philosophy of Science at Erasmus University. I learned what makes a good argument and how to check that the structure is sound. We had to analyze others’ texts and dissect them completely.
  2. Creative tools for scientific writing. For example, how to create overview for yourself, how to extract the main message from a paragraph and write towards it, and how to draw the reader into a text and hold their attention. And also: how to avoid writing those scientific sentences that seem like monstrosities. So that, even though it contains a lot of knowledge, it is also a bit of fun to read your article.

My writing process

After all this preparation, nothing stopped me from keeping my fingers on the keyboard. In January, I had three weeks, nonstop. I had also lost my voice, so I really couldn’t plan anything else.

During the second course, I wrote the introduction to the article, describing the research question, the purpose of the article and the approach to arriving at answers. This introduction covered about 500 words and I discussed it at length with my supervisors. After some scraping, I also sent it to Servsig, a service conference in June. It would be nice if I could give a presentation about the article there; I’ll hear about that later in the spring.

I had already done a lot of reading and made notes. As a library for scientific articles, I use Zotero, which is also useful for keeping track of references while writing. I read through all my notes again. I then created a visual outline in Miro with the main arguments and figures I was going to use, as well as the flow from one point to another.

Picture of my screen with all the figures while holding a tasty cup of coffee in front of it so you can't read everything
Expanded Miro board with all figures

So the real writing began quite functionally. I started with the main research questions and built the answer step by step in blocks. I added the main sources.

Then I worked out the blocks point by point. Each sentence I placed on a separate line with a number in front of it. The sentences were short and staccato. The important thing was that the argument was clear, and if it was, I marked the most important line. That was the point I was trying to make. I shuffled back and forth some more, looking critically to see if all the sentences were building toward that conclusion. And then I moved on to the next block.

I want to publish in an English-language journal, but I don’t really think in English yet. So I started in Dutch and translated sentences into English when I was satisfied with the structure. I used Deepl as a translation assistant to help me.

Making beautiful sentences

Now it was time to do what I used to call writing: make beautiful sentences. I removed the numbers and wrote the sentences together. Some sentences I added together. Other sentences I worked out and became as many as three sentences. I divided the information into paragraphs and again marked the main message for each paragraph, building toward the conclusion of the entire block. The sentences were much less staccato, but carried full-blown information. And of course I added all the references of the sources I relied on.

For editing, I used Edit.gpt. Mainly for English grammar, but also to improve overall flow. Some adaptations I adopted, others I did not. Then I checked that the sentences ran smoothly and were not too long. I did this by reading the text aloud.

When that was all done, I moved on to the next block. After completing all the blocks within a segment, I read through everything a few times. I watched the transitions from one point to another. I checked that all the research questions were answered and that the whole thing invited further reading.

And so last month I wrote a whopping 8,000 words.

Photo of my desk and window, with my legs stretched out in the window frame
8,000 words later

This writing is very different from typing a blog. At first, I had to get used to it tremendously. And it required an awful lot of discipline. With a blog, I usually wait for inspiration, but writing an article went step by step. In December, I resolved to take one step every day, or say, 300 words, so that I would have a draft ready by the end of January. And it succeeded. Even on days without inspiration.

Now what I wrote goes “into the mix. That means receiving feedback from my promoters. They will ask a lot of questions about the argument and rearrange things again. Some of the blocks also need better elaboration, but after 8,000 words I had a toasty brain. I am very curious to see what they come back with.

And about the content itself, because I understand that you are also extremely curious about that, there will be a blog later too :).

References

Jaakkola, E. (2020). Designing conceptual articles: Four approaches. AMS Review, 10(1-2), 18-26.

MacInnis, D. J. (2011). A framework for conceptual contributions in marketing. Journal of Marketing, 75(4), 136-154.

Categories
How do you do research? Promoklip

How to survive your first conference: Frontiers in Service 2023

The sun was shining, I was in one of my favorite cities (Maastricht) and had breakfast with Limburg pie every morning. Last week was Frontiers in Service 2023. My first academic conference. Four days, with a day especially for PhD candidates, dedicated to the most current research on services.

In this blog, I share what I learned from this conference for my own research on public services that are good for people, in my case study on Dutch national public service providers. I do so in the form of 4 reasons why it is a smart idea as a beginning researcher to attend conferences right from the start of your PhD.

Reason 1: get a good overview quickly

The service field is actually not my own domain. I do research from the Technical University of Delft, Industrial Design Engineering. Design is my home base, and I do research on how to design services in government. My research connects three disciplines: design, services and public administration. To fully understand the services part, I went to Frontiers and regularly visit CTF, Karlstad University’s service research center. (I wrote this blog about my first visit at CTF.)

Through conversations at CTF, I stumbled into the service field. From there I looked for a route like a snowball finding its way. Of course, I read the classics, for example, on service-dominant logic. I practiced how to apply this way of thinking in my research context by writing the blog the value is in the classroom.

A conference highlights the most current state of a research field. One look at the program and you see how much there is. The services field is much larger than I thought!

I found out that public services are a niche here, and government services may be the niche within the niche.

Entering through a side door

Frontiers in service is a conference that deals with services, including the creation and delivery of services in the broadest sense. The keynotes focused on the child care benefit scandal in the Netherlands and Starbucks’ new service concepts in Asia. We saw experimental examples of VR glasses in education and service robots in healthcare. Nearly 250 studies were presented in 60 sub-sessions. View the entire program here.

I discovered that “transformative service research” is very hot right now and, as a separate research topic, even has its own acronym: TSR. From my own reading nook, I had not yet come across this. This is something I will be looking up more about in the near future.

Reason 2: free advice

If at the beginning you still nervously walk with your coffee to a table where no one is standing, by the end of the conference you know that you can join a busy table. Indeed, that is a very good idea. After your name and your university, you drop that you have only just started your PhD and it immediately starts raining tips from the seniors at the table. They ask questions, talk about their own early days and give advice on writing good articles for top journals. That they sometimes contradict each other we will never let them know of course.

Coffee and tips

This free advice makes sense because the purpose of a conference is to exchange knowledge. Everyone seeks each other out and talks about each other’s research. Rarely have I received such good (and critical) questions about my research: what I want to contribute to it, how I want to approach it and more.

In line for the ice cream truck, for example, I was engaged in a discussion about what public value is with Jakob Trischler (whose articles I have read before). Is that really well defined yet (we thought not) and does public value actually have a place within the marketing-dominated world of service research? I’ll have vanilla with raspberry in a cone, please.

Extra helpful was the advice of Mike Brady, Professor of Marketing at Florida State University and editor-in-chief of the Journal of Service Research (JSR). In a small ask-me-anything session, he talked at length about JSR’s editorial process and how to write and submit your first articles to top journals as a young researcher. Those were a lot of useful tips.

Reason 3: practice pitching and learn from others

Most people gave presentations themselves; I was there as a consumer. If my paper will be accepted, I may present at a design conference this fall. During Frontiers, I was able to watch how others show their work. What are the codes, the unwritten rules? How do others structure their presentation? How do they deal with critical questions?

Very interesting presentation by Koskela-Huotari et all (2023) on systemic dynamics in (un)sustainable behavior

Yet I could not consume alone.

Regularly, someone asked what my research is about. I am now over six months in and exactly at that stage when you start doubting everything. My rehearsed pitch “I’m doing research on how to design government services that are good for people” is starting to come out slicker and slicker, but by now I’m having doubts about the definition of government services, about what is design, what is good and what are people really??!

That’s okay. By telling what you do and why it is relevant over and over again, you learn to put it into words better and better. After the fifth pitch (and because of the questions I received), I subconsciously adjusted the pitch a bit. By the fourth day, the insecurity was gone and I surprised myself how much better I could explain what I want to do.

I am still on “how to make government services that are good for people” haha. But also on how important it is to properly define that “public value. Maybe I can do that by just delving more into the public administrative side: the principles of good governance and the Rule of Law of course! The radars are already turning at full speed again.

Session on thinking from ecosystems in service research

Reason 4: meet nice people in a new city

As an external PhD candidate, I am not at the university much. I live in Groningen, which is 3+ hours by train to Delft. My supervisors live in Delft and Karlstad, so most of my research happens online, from home or from my research context, in the Dutch government. I don’t know that many other PhD candidates to exchange experiences and joke around with. However, this can be done at a conference (Della and Mike, I see you), especially if a special Doctoral Consortium for PhD candidates is organized.

Maastricht is a super fun city is and the University of Maastricht knows how to organize a cool conference. Attending a multi-day conference is also getting away from the grind, from behind your desk, and getting new inspiration. Rent a bike, have breakfast on a terrace, take your new friends into town. I even got to row on the Maas River with the Maastricht rowing team.

Rowing on the Maas River

In short: many new experiences bring new questions to delve into in the coming time. The next conference will hopefully be IASDR2023 in October. But first, in August, I am going to CTF in Sweden again :).

Want to follow my research on Dutch government services? You can! Through my newsletter (in Dutch) or follow this blog via my Linkedin.

References and reading tips

Koskela-Huotari, K., Svärd, K., Williams, H., Trischler, J., & Wikström, F. (2023). Drivers and Hinderers of (Un) Sustainable Service: A Systems View. Journal of Service Research, 10946705231176071.

Jakob Trischler & Jessica Westman Trischler (2022). Design for experience – a public service design approach in the age of digitalization. Public Management Review, 24:8, 1251-1270.

This page lists all the books and articles I use in my research.

Categories
Not part of a category How do you do research? Promoklip

Open action research

This month I write about my new research on government services that are good for people. I wrote the big plan and the journey so far. In this blog you can read about the approach and an initial planning for the coming years.

You can sign up for my newsletter. Once a month you will receive a summary of the research in your mailbox. This way you won’t miss anything and you can easily respond and participate in the research.

Choosing the approach

A number of personal considerations quickly helped me decide how to approach this research. For several years now I have enjoyed working across government, from the perspective of citizens who have to deal with the entire government. During my master’s, I really enjoyed the critical sparring with an external institute. I therefore contacted TUDelft to find such a construction again. In the form of a PhD research I was able to find both an excuse to work government-wide and to collaborate with a university.

But… I am not concerned with ‘just’ producing new knowledge. No, I want us to learn how to work from a human perspective in practice at the government. And of course I have developed my own design and research skills in a certain direction in recent years, which you have been able to read on this blog since 2017.

An approach that fits well with all of this is action research. To me that is open action research because I blog about every step. Not only the result is open, but the process while we are working. So you have every chance to adjust the process!

Combining practice and theory

Action research is not your average scientific research. It is a much more practical approach, which is why I will not be working at the university in the coming years, but at the government’s implementing organizations itself. To research from the inside in practice together with colleagues.

Research that starts from the problems generated by organizational contexts and focuses on change requires a radical reappraisal of the relationship between knowledge and action, and of the related image of the ‘academic researcher in an armchair’.

Van Marrewijk, A., Veenswijk, M., & Clegg, S. (2010) ‘The organizing reflexivity in designed change: The ethnoventionist approach’, Journal of Organizational Change Management, 23(3): 212‹29

Action research has a number of properties that suit me and the issue well. Action research is:

  • in the situation: it requires direct involvement with real and complex problems in the natural context
  • relationship-based: we learn through relationships with stakeholders who all have different perspectives and contribute in their own ways to understanding and solving problems
  • focused on change: together we look for ways to initiate, promote and manage change
  • reflexive: we continuously (in action) critically consider our own practice; I as a researcher and together with all participants. We learn together and an open and explicit learning process is created.

From: Giuseppe Scaratti, Mara Gorli, Laura Galuppo and Silvio Ripamonti. Action research: knowing and changing (in) organizational contexts. In: The SAGE handbook of qualitative business and management research methods: history and traditions, 2019.

Reading tips

In the near future I will learn more about this way of doing research. During my master’s I learned a lot of practical skills and now I’m also immersing myself in the methodological background. In addition to the article from the previous section, the following books are very helpful to me:

  • Introduction to action research, social research for social change by Davydd Greenwood en Morten Levin. This book provides a good overview of the background of action research and a number of examples of different trends and approaches.
  • The reflective practitioner by David Schön. I already read this while making the photo interviews from The compassionate civil servant. Very good book about reflecting while you are in the middle of it all, about reflection-in-action.
  • Doing action research in your own organization by David Coghlan. I’m reading this one at the moment. Much corresponds to the approach I also used with The compassionate civil servant, but it is much more extensive and explains action research in a more fundamental way.

Role of this blog

If you’ve been following my blog for some time, you might find it super logical that I’m going to write about this research. That’s how it feels to me too. But in the near future this blog (and the newsletter) will have an important function.

I once started writing as my own archive, for myself and the handful of colleagues who might also find it useful. Later it became a place to document my research results and after that a place to participate in the social discourse about the human dimension in government.

Since October I speak to my supervisors Maaike Kleinsmann and Jasper van Kuijk every month. They noticed that I often quickly determine how something should be interpreted when it comes to government. “Yes, that’s just how the government works,” I say. I have been a civil servant for 10 years and, through this blog, I hear so many stories from you about how your organization is doing, that many things are so self-evident to me. That’s all tacit knowledge that I myself sometimes don’t even know I know.

By writing I make my own thoughts and choices explicit. And by sharing you can react to it and add knowledge and new questions. This is how we reflect together in action.

Then the schedule

This is a long-term project. It will certainly take several years. Overall, I look at it something like this:

Year 1, we are in the middle of this, is a year of preparation. Important is:

  • setting up the project and connecting with you and other stakeholders
  • dive into the literature and build a foundation for the years to come
  • make a concrete research plan for the following years, including agreements with organizations, make a data management plan, pass the ethics committee, and probably more that I can’t yet oversee
  • working on my own skills, because it’s quite a different story doing PhD research

Year 2 and 3 I will work in practice together with public service organizations. For example, I might come to work in your organization and together we create a service from A to Z from the perspective of the citizen. Together we reflect and learn. In the coming year I will develop this further and I will also share which criteria such a case preferably meets.

Year 4 consists of finishing. Insights become shareable and beautiful end products are produced. This will be very practical and applicable for everyone who helped and theoretically in the form of a dissertation.

In the next blog, which will be online soon, I will share more about the first year, especially about diving into literature.

Categories
How do you do research? Visual working

Visual interviewing

Tijdens een workshop bij DUO over datavisualisatie.

I increasingly use diagrams, drawings and infographics when I interview someone. Since corona, I often interview digitally. At first I found this very inconvenient and impersonal, but I also discovered the benefits. Precisely because it is digital you can very easily draw together and organize information schematically.

In this blog some examples of how I approach visual interviewing and some tips for getting started yourself. I think it would be fun to organize a workshop or meet-up about this sometime, so I’d love to hear if there’s enthusiasm for that.

https://open.spotify.com/episode/5ojiEGVi4QwMZemZEbLrUX?si=4df36a8b08854547

Drawing together

“I can’t draw at all” is the first sentence that rolls out of many people when I introduce this method. That changes when we get down to business and the interviewee loosens up and finds it easier and easier to tell and participate. And at the end, “now it’s much clearer for myself, too.”

I call it visual interviewing, drawing together, schematic conversation, picture talking… Think of a nice name. It means that as we talk, we directly visualize how things fit together. This can be done, for example, by making a mind map together, a timeline or some other useful arrangement. During the conversation, you put all the information directly in the right place, which helps both of you get to the point and immediately see if the story is complete.

And yes, you may also just use text while drawing.

A few examples

Janet Ramesar and I created a timeline together. That one went back ten years to the present. On the timeline, we posted her experiences with child care subsidies. You could see it getting more and more complicated over time. By lining it up like that, you saw cause and effect very clearly. It gave us a foothold to broadly capture the story in a short period of time. We created the entire timeline in a 1-hour zoom session. Janet told, I shared my screen and posted what she told directly on the timeline. If I misplaced it, she corrected me: ‘no, that should be first. That one should be more to the right’. It gave us both something to hold on to.

A snippet from the timeline Janet and I created together about her experiences with the child care subsidy.

I later used the timeline format myself when I charted a month out of my relationship with the government. And I applied it in a project for Werk aan Uitvoering where, together with Maureen Hermeling, I interviewed someone from Loket Geldzaken of the Municipality of The Hague (on my blog I called him Puzzler Patrick). The three of us thus mapped out complex case histories involving debt.

A snippet from the timeline of Puzzler Patrick and Mr. N.

At the Ombudsman, during a series of interviews with colleagues, I used not a timeline but a drawing of the organization. The key question was “how should we organize our own activities to have a greater impact on the government?” In each interview, we drew together a new picture of our ideal organization. While making that drawing, I kept asking why-questions. What I learned from those conversations I described in the blog about a professional listening standard. This is the basic drawing I used as a start for the conversations (i.e., before everyone started shuffling around with it and making a new drawing):

At the beginning of the corona crisis, I got to work for a while with the team that made the corona apps at the Ministry of Health. From Jasper, my husband I had just received an Ipad at that time. I took it along on my visits to the local health organizations (GGD) and outlined with the staff the source and contact tracing and the bottlenecks they experienced. For example, we made these kinds of sketches:

Last example, also with the Ombudsman. In a short brainstorm about signals and complaints as input for research topics, we drew them so you could immediately see the extent of the signals and the overlap with the rest of the topics. This was a first sketch that we could develop later.

Why it works

Drawing together just works nicely. Even in on-site conversations, it’s easy to grab a flipchart: “Shall I draw it out?” and the confusion of speech disappears like snow in the sun.

Digitally, it can be done just as easily. You share your screen, you draw how you envision it, and the other person can immediately respond, “no, you have it all wrong, it should be just like this.”

Creating together means organizing and structuring the story together. It is immediately tangible. You can see it before you. You catch the words in focus before they slip away.

The input is the output. You have the basis for the result and documentation ready immediately after the interview. After talking with Janet, I only needed an extra hour or so myself to make everything neat and write out the keywords. After a check from her, it was finished.

Talking digitally is much less impersonal this way. You are working together. You don’t necessarily have to see each other at all, because you share your story equally and are creative together. That activates even though you are both sitting behind a remote screen. And as an added bonus, you won’t have to type out another stack of post-its or flip charts after the event.

This is what you need

  • a digital drawing tool. I use Miro, but you can also, for example, connect your Ipad to your laptop, draw along and share your screen. Or use another digital tool, there are plenty of them.
  • a basic architecture. Think about how you will set it up in advance. Are you creating a timeline? A schematic of the organization? A mind map of questions and answers? Or something else? In Miro, you have handy ready-made templates that you can use.
  • Skills to sketch visualization. It doesn’t have to be perfect right away. Thinking out loud “hmm how shall I portray this” is fine. The other person then also helps, and you give the other person a chance to make together. In a master class taught by Stefanie Posavec (known for Dear Data), I learned the basic techniques for working visually. I wrote this blog about it and later gave this workshop about it at DUO.
  • program to video call and share your screen

For advanced visual interviewers:

  • also give the other person editing rights in your program. Then take time for a brief intro and explain how the drawing program works. I set aside at least 10 minutes for this and do some practice together. Sometimes it is difficult and it might work easier if I draw and ask questions myself. The other person then thinks with me and gives me drawing instructions. That’s why I call it visual interviewing, because you help the other person tell their story in pictures or diagrams.

Getting Started

Enthusiastic to also draw together with your colleagues or the person you are interviewing? I hope this is of some use to you. It is fun to see how you tackle it, so please share your results with me (and others!) too.

Categories
How do you do research? Not part of a category Visual working

How we can see an automated decision

With increasing talk of algorithm supervision, it is important to know how a third-party study of algorithms can be conducted in government. Preferably in such a way that day-to-day implementation processes are not disrupted but according to the principle that government shows itself.

Since January, I have been working with a very nice group to design a work method for this. We focus mainly on the fixed, ‘dumb’ algorithms and not yet on self-learning algorithms. In this blog, a behind-the-scenes look at our approach, initial observations and two questions for you.

https://open.spotify.com/episode/61MQ5s4ce95L0bMGHiotnF?si=b6a251a59fc040f3

I can just see it

When I had just finished the study The compassionate civil servant, I was sitting with Marlies van Eck on a terrace in Utrecht toasting. She mentioned that she had a follow-up idea to her dissertation (on legal protection in automated chain decisions). Her main conclusion was that, as a lawyer, she could not say whether citizens were sufficiently protected because she could not see the algorithms. The government was a black box, so how can you test whether the decisions it makes are right?

Over my beer I said “but, how weird, because at the Executive Agency of Education, where I worked, I can just see them’. Of course, I immediately began to doubt myself when I said that, but I looked up the photo interview of Cees-Jan talking about decision rules from which the code is then derived. And I mentioned that we were also working to derive the calculation tools on duo.nl from these decision rules so that, as a student, you can simulate how the computer will decide in the future how to pay off your student debt.

Opening and viewing

I am reminded of our conversation frequently because “being able to see the algorithm” is central to this research.

Marleen Stikker writes in her book The Internet is Broken that you only really own something when you can open it up to fix or change it. If you can’t, as with most contemporary technology, the device owns you. Can we open up government’s computers and see how they decide and possibly change that if necessary?

This area is still relatively unknown. The Dutch Tax Service has developed a method for making algorithms testable and explainable. The State Audit Office and accounting firms are preparing for this new task. The General Audit Office has developed a review framework. Recently, the State Council issued a report on automated law enforcement, among other things.

But what is suitable for a financial expert may not work for a researcher who wants to do a legal check. It is also tedious for an implementing organization when different disciplines and organizations consider the use of algorithms at different times and burden the implementation with questions. Both for researchers, and for implementing organizations, a more integrated approach is important.

Say no more. So with a group that I spontaneously get imposter syndrome from, we started in January to experimentally design a method for this. The group: Marlies van Eck, Steven Gort, Abram Klop, Robert van Doesburg, Mariette Lokin, Carlijn Oldeman, Giulia Bössenecker and me. Colleagues from the Executive Agency of Education and the Social Secutiry Bank are cooperating and offering us their automated decisions as trial material to design the method on.

What we want to achieve

Allright. We want to develop a working method for conducting third-party research on a government organization’s use of algorithms. To this end, we have formulated a thought scheme:

  • We study how the use of algorithms can be examined simultaneously legally, financially and model-wise[topic].
  • because we want to know which discipline would like to see which research questions answered[rationale].
  • so that we understand the least burdensome approach to a multidisciplinary and comprehensive assessment[significance].

To keep the research sufficiently concrete but also manageable, we propose that the working method should meet the following four objectives:

  • the working method allows a lawyer to make a judgment on the legality and propriety of the system,
  • the working method allows a data scientist/ information scientist to make a judgment about the quality of the system,
  • the working method enables an accountant or internal controller to make a judgment about …[to be completed, see also help question at the end of this blog],
  • the working method is suitable for repeated use in various public organizations.

Designing the method

In a series of on-site making days, we will work with a government organization. Lucky for me: the first 2 making days are at the Executive Agency of Education in Groningen where I live.

To arrive at a working method that can stand on its own within a few months, we take an embedded and iterative approach. But we must remember to design ourselves out of the method as well, and thus critically examine our bias. We learn by doing: we dive into an algorithm and record our process. Then we reflect on our process and make it explicit. What comes out of this is potentially the method we want to develop.

This is how we shape the creation process:

Outline of the design process divided into the 4 making days and intermediate actions

Together with the Executive Agency of Education, we chose a few automated decisions that are part of the big decision whether or not a student will receive a student loan: the nationality test, the age test and the partner test. Beforehand, we were given the sets of decision rules of these three tests to study firmly.

We decided to look from three perspectives or levels of abstraction. Different questions are important for each perspective.

Perspective 1 is the algorithm itself. How does the decision come about?

  • What are the decision rules dealing with the nationality test?
  • What sections of the law are the decision rules based on?
  • How are the decision rules programmed?
  • How are they included in the work instruction?
  • What data is needed and what are sources?
  • How is the decision explained to the student (user) in personal and general communication?
  • What interaction does a student (user) have with the algorithm and how does this influence the decision?

Perspective 2 is the creators of the algorithm. How does the algorithm come about?

  • Who (what roles and also individuals) are involved in the creation of the algorithm?
  • What considerations were made in the creation of the decision rules?
  • On what basis do these individuals make these trade-offs? What personal bias is there?
  • Is there documentation of this creation, and if so, what is recorded here and for what purpose?
  • What is implicit and difficult to make explicit here? In other words, what do we not know (yet) or cannot know (anymore)? How does this come about?

Perspective 3 is ourselves, the supervisors so to speak. How does the method come about?

  • How did we proceed? What questions did we ask? What worked well and what didn’t?
  • Which questions belong to which discipline (lawyer, accountant and information scientist)? Is there any overlap? Were all disciplines able to get their answers? How do they reinforce each other?
  • What knowledge and expertise do we have that allows us to ask these questions?
  • Can we reproduce our process next time? What would be different?

‘How crazy it is here’

The first day of making was wonderfully chaotic. We were flying in all directions, asking all kinds of questions, and Jean and Cees-Jan from the Executive Agency of Education were infinitely patient to answer all questions. Sometimes we ended up in heated discussion about the way the organization was organized, only to conclude after an hour that that shouldn’t matter at all for the method. Or as Steven so aptly noted, “I couldn’t care less how it is put together.” :’)

After this day of making, Marlies wrote a first blog with her observations and I made a video for the scientific guidance committee and the sounding board group.

Video (in Dutch) showing the experience of the first day of making at the Executive Agency of Education.

The second day of making went a lot more structured. In preparation, I visualized on a digital board all the steps from law to decision. I asked Cees-Jan if he could fill in everything from another decision (the partner test). This was not so simple because the information and docs had to come from all sorts of nooks and crannies of the organization.

I printed what I got (or left a step open) and put the steps on the floor. Underneath I put 3 long sheets of paper for the questions the lawyer, accountant or developer could ask. Perspective 1 and 3 are below each other. We took all afternoon to fill them out and see if we could get answers directly from Cees-Jan and colleagues (perspective 2).

An impression of what that looked like:

In two weeks we will have make day 3 at the Social Security Bank. That will be uncharted territory for me, fun! We are going to look at the age and partner test in the state pension.

In preparation, I digitized the large board we made on the ground at the Executive Agency of Education. It became a version with questions and a blank version for the Social Security Bank colleagues to fill in themselves. We are going to test this prototype and see if this is already a tool for self-assessment to show as an organization how your algorithm comes about and makes a decision.

Through this miro board, you can better zoom in on the prototype. You can also post comments here, and feedback is welcome!

Prototype of the working method we will soon test with the Social Security Bank

Two questions

You can see that the accountant’s perspective has no questions yet. Here we are still searching. Auditors are very busy around February and March because of wrapping up the previous year. So we haven’t been able to include much input from that side. Do you know more about this or know someone who can help, let me know.

And of course: feedback in general. In this blog, I describe our making process and show the prototype. I’m curious what you think of this and how it would work in your organization. After design day 3, we would like to update our prototype and share it with organizations to use without us for self-assessment. So we want to learn from your findings in turn.

If you would like to help with this, please send me a message. You can do so through maike @ klipklaar.nl or the other known ways. Thanks!