Categories
(Un)understood citizens A compassionate future Not part of a category

This is not about customization

‘Nice this story about customization, but surely that can never be done for everyone?’ That was the first question I got from the audience the other day when I gave a lecture at a government organization about The Compassionate Civil Servant and the relationship citizens have with government. I was confused. I hadn’t mentioned customization in services I thought. I get questions like that a lot, by the way.

Why do people keep thinking I’m talking about customization when I talk about more empathy in government?

Hence this blog, which is NOT about customization. Okay, a little bit then. But it is mostly about standard services that are good for people because it is better to go full steam ahead with that than with customization.

What is customization?

If you ask me, I say ‘anything that deviates from the process because the situation calls for it’. A very human and fine process does not have to be customized if that is always how it goes.

Customization involves a real deviation from the standard. For example, a government employee makes a decision that allows someone to get or do something even though they are not entitled to it under the regulation. Or a staff member (or sometimes an entire team) looks at the situation separately with the person to figure out what is going on when in the normal process there is no time for that. An exception will be made for someone.

This is directly also the tricky thing about customization, because you have to justify the decision anyway. Therefore, in some regulations there is a hardship clause that provides a legal basis for exceptions. But even then you should be able to explain why the exception is justified, because otherwise the neighbors will complain ‘how come she got it, and I didn’t’. Unequal cases should be treated equally unequally, so to speak. How you do that is quite tricky and can also put a lot of mental pressure on the deciding officials.

I started with ‘if you ask me’, because there is no official definition of customization in the Dutch government yet. We are working on it, but right now each organization, and perhaps each team – you know how it goes – uses its own definition. That makes it difficult to have a good conversation about this, I think.

Instead of focusing on customization, I prefer to talk about customization’s antithesis: the standard service. When we invest heavily in government services that are already so good for people you don’t necessarily need customization. The service itself can be good.

Rather a good standard process than customization.

Citizens like that too, because no one likes being the exception and going through extra hassles (customization is also more hassle for citizens than if it had been appropriate right away).

What is a standard service?

I draw the steps that are usually in there.

An example. You want to apply for student loans because you are going to start studying. You search for information, online, through school, through your dad. Finally, you log in to MyDUO and fill out the digital form. You get a confirmation email and a while later the student finance in your account.

It is not rocket science. Government services often mean: seeking information, filling out forms, possibly with supporting documents and then waiting for you to ‘get the thing (settled)’ you are concerned about, possibly with confirmation. Sometimes you go to a counter for this, but usually the counter is online at a website.

Another example. You can argue about whether it is really a service but fine: you get a fine from the CJIB, the Dutch debt collection centre. Which starts on your door mat (or actually on the road when you started the offense by going beyond the speed limit maybe). You may be looking for some additional information and pay. Done.

As a service designer, I call these examples the happy flow. How someone experiences this can be called a customer journey (but I kind of hate that word ‘customer’ in government). You can expand this diagram to include the government side and show the work process behind it. This is called a service blue print, a schematic of a service from the perspective of the customer (citizen) and the provider (government). For example, like this.

Want to see a much more elaborate version than this sketch? I once tracked for a month everything that I was doing with the government and then mapped out the government’s side of things as well as my own. I made a big schematic drawing of it.

You could add to this the standard steps the government offers for when you disagree. You can appeal the decision, for example, if you think the fine is unjustified, and then even go to court. All standard.

So when then customization? If this happy flow doesn’t work?

No, not yet.

The policy has also come up with exceptions as to why something might not work. There are then additional arrangements for that and you can incorporate them into a standard service. A kind of variation on the happy flow.

For example, with student loans. Suppose you become ill during your studies, physically or mentally. Your studies are not going that good anymore and you get delayed. At the end of your studies, you get into a jam, because you are basically entitled to x number of years of student finance and it runs out, but your studies are not finished. With a letter from your dean or doctor, you can apply for an additional year of student loans. No customization! It is just a regulation that is a side branch of the happy flow.

An example from the CJIB then. You receive a fine, but the letter is still sent to your old address. That’s where your ex lives, you broke up and your ex didn’t forward the fine. Very annoying and now it has accumulated to a much larger amount due to the reminders. You call the CJIB, explain the situation, and the CJIB says ‘if you pay the original amount, it’s fine’. This must be customized, surely.

No, it’s not, this situation happens more often and the work instruction for CJIB employees tells them how to resolve it. (I can well imagine that this was once a custom process and then CJIB included it in the work instruction when it turned out not to be so incidental – hooray for CJIB).

What we can learn from CJIB is that you can adjust services. Based on feedback from citizens through the telephone, the website, social media or in many other ways, you can learn, add interventions in your process and thus make your service better and better. Perfecting your standard. Love it.

You might already notice. These are standard services. Employees do not have to make their own consideration of whether it is lawful and not arbitrary. That consideration has already been done by including it in the standard service – for everyone. Parts of these types of services can even be left to computers by the government so that you, as a citizen, get instant confirmation that it is in order and do not have to wait for an official to look.

But the problem in government is that most services are not created from the user’s perspective.

The happy flow that the government comes up with does not necessarily make people happy. The government invents services that are mostly good for the government itself.

Processes must be as efficient as possible and are outsourced entirely to computers. Changes require a business case first, preferably for savings. KPIs (success metrics) are about whether people can deal with it on their own and most importantly Don’t Call. To help with that, we are setting up points in libraries to make people more digitally proficient (oh the irony).

Organizations carry out regulations that they have as a statutory duty and how this intersects with other regulations of other organizations… well, that’s not their responsibility now, is it? And that someone could arrange something with multiple organizations in one fell swoop, and they won’t have to go from site to site, or that the government proactively thinks along: ‘oh gosh, this will also affect your rent allowance if you apply for this with us’… Wouldn’t that be something?

And so people get into trouble and in the newspapers. Everyone angry, the House of Representatives angry: the social pendulum is swinging toward More Human Size. We get high on stress and exclaim: ‘okay, we’ll solve it with customization.’

We don’t have to if we start with services that are good for people.


How to make good services? That’s for other blogs. But you can read Good services by Lou Downe to get a start (or this summary of it I wrote earlier).

Categories
How do you do research? Not part of a category Visual working

How we can see an automated decision

With increasing talk of algorithm supervision, it is important to know how a third-party study of algorithms can be conducted in government. Preferably in such a way that day-to-day implementation processes are not disrupted but according to the principle that government shows itself.

Since January, I have been working with a very nice group to design a work method for this. We focus mainly on the fixed, ‘dumb’ algorithms and not yet on self-learning algorithms. In this blog, a behind-the-scenes look at our approach, initial observations and two questions for you.

I can just see it

When I had just finished the study The compassionate civil servant, I was sitting with Marlies van Eck on a terrace in Utrecht toasting. She mentioned that she had a follow-up idea to her dissertation (on legal protection in automated chain decisions). Her main conclusion was that, as a lawyer, she could not say whether citizens were sufficiently protected because she could not see the algorithms. The government was a black box, so how can you test whether the decisions it makes are right?

Over my beer I said “but, how weird, because at the Executive Agency of Education, where I worked, I can just see them’. Of course, I immediately began to doubt myself when I said that, but I looked up the photo interview of Cees-Jan talking about decision rules from which the code is then derived. And I mentioned that we were also working to derive the calculation tools on duo.nl from these decision rules so that, as a student, you can simulate how the computer will decide in the future how to pay off your student debt.

Opening and viewing

I am reminded of our conversation frequently because “being able to see the algorithm” is central to this research.

Marleen Stikker writes in her book The Internet is Broken that you only really own something when you can open it up to fix or change it. If you can’t, as with most contemporary technology, the device owns you. Can we open up government’s computers and see how they decide and possibly change that if necessary?

This area is still relatively unknown. The Dutch Tax Service has developed a method for making algorithms testable and explainable. The State Audit Office and accounting firms are preparing for this new task. The General Audit Office has developed a review framework. Recently, the State Council issued a report on automated law enforcement, among other things.

But what is suitable for a financial expert may not work for a researcher who wants to do a legal check. It is also tedious for an implementing organization when different disciplines and organizations consider the use of algorithms at different times and burden the implementation with questions. Both for researchers, and for implementing organizations, a more integrated approach is important.

Say no more. So with a group that I spontaneously get imposter syndrome from, we started in January to experimentally design a method for this. The group: Marlies van Eck, Steven Gort, Abram Klop, Robert van Doesburg, Mariette Lokin, Carlijn Oldeman, Giulia Bössenecker and me. Colleagues from the Executive Agency of Education and the Social Secutiry Bank are cooperating and offering us their automated decisions as trial material to design the method on.

What we want to achieve

Allright. We want to develop a working method for conducting third-party research on a government organization’s use of algorithms. To this end, we have formulated a thought scheme:

  • We study how the use of algorithms can be examined simultaneously legally, financially and model-wise[topic].
  • because we want to know which discipline would like to see which research questions answered[rationale].
  • so that we understand the least burdensome approach to a multidisciplinary and comprehensive assessment[significance].

To keep the research sufficiently concrete but also manageable, we propose that the working method should meet the following four objectives:

  • the working method allows a lawyer to make a judgment on the legality and propriety of the system,
  • the working method allows a data scientist/ information scientist to make a judgment about the quality of the system,
  • the working method enables an accountant or internal controller to make a judgment about …[to be completed, see also help question at the end of this blog],
  • the working method is suitable for repeated use in various public organizations.

Designing the method

In a series of on-site making days, we will work with a government organization. Lucky for me: the first 2 making days are at the Executive Agency of Education in Groningen where I live.

To arrive at a working method that can stand on its own within a few months, we take an embedded and iterative approach. But we must remember to design ourselves out of the method as well, and thus critically examine our bias. We learn by doing: we dive into an algorithm and record our process. Then we reflect on our process and make it explicit. What comes out of this is potentially the method we want to develop.

This is how we shape the creation process:

Outline of the design process divided into the 4 making days and intermediate actions

Together with the Executive Agency of Education, we chose a few automated decisions that are part of the big decision whether or not a student will receive a student loan: the nationality test, the age test and the partner test. Beforehand, we were given the sets of decision rules of these three tests to study firmly.

We decided to look from three perspectives or levels of abstraction. Different questions are important for each perspective.

Perspective 1 is the algorithm itself. How does the decision come about?

  • What are the decision rules dealing with the nationality test?
  • What sections of the law are the decision rules based on?
  • How are the decision rules programmed?
  • How are they included in the work instruction?
  • What data is needed and what are sources?
  • How is the decision explained to the student (user) in personal and general communication?
  • What interaction does a student (user) have with the algorithm and how does this influence the decision?

Perspective 2 is the creators of the algorithm. How does the algorithm come about?

  • Who (what roles and also individuals) are involved in the creation of the algorithm?
  • What considerations were made in the creation of the decision rules?
  • On what basis do these individuals make these trade-offs? What personal bias is there?
  • Is there documentation of this creation, and if so, what is recorded here and for what purpose?
  • What is implicit and difficult to make explicit here? In other words, what do we not know (yet) or cannot know (anymore)? How does this come about?

Perspective 3 is ourselves, the supervisors so to speak. How does the method come about?

  • How did we proceed? What questions did we ask? What worked well and what didn’t?
  • Which questions belong to which discipline (lawyer, accountant and information scientist)? Is there any overlap? Were all disciplines able to get their answers? How do they reinforce each other?
  • What knowledge and expertise do we have that allows us to ask these questions?
  • Can we reproduce our process next time? What would be different?

‘How crazy it is here’

The first day of making was wonderfully chaotic. We were flying in all directions, asking all kinds of questions, and Jean and Cees-Jan from the Executive Agency of Education were infinitely patient to answer all questions. Sometimes we ended up in heated discussion about the way the organization was organized, only to conclude after an hour that that shouldn’t matter at all for the method. Or as Steven so aptly noted, “I couldn’t care less how it is put together.” :’)

After this day of making, Marlies wrote a first blog with her observations and I made a video for the scientific guidance committee and the sounding board group.

Video (in Dutch) showing the experience of the first day of making at the Executive Agency of Education.

The second day of making went a lot more structured. In preparation, I visualized on a digital board all the steps from law to decision. I asked Cees-Jan if he could fill in everything from another decision (the partner test). This was not so simple because the information and docs had to come from all sorts of nooks and crannies of the organization.

I printed what I got (or left a step open) and put the steps on the floor. Underneath I put 3 long sheets of paper for the questions the lawyer, accountant or developer could ask. Perspective 1 and 3 are below each other. We took all afternoon to fill them out and see if we could get answers directly from Cees-Jan and colleagues (perspective 2).

An impression of what that looked like:

In two weeks we will have make day 3 at the Social Security Bank. That will be uncharted territory for me, fun! We are going to look at the age and partner test in the state pension.

In preparation, I digitized the large board we made on the ground at the Executive Agency of Education. It became a version with questions and a blank version for the Social Security Bank colleagues to fill in themselves. We are going to test this prototype and see if this is already a tool for self-assessment to show as an organization how your algorithm comes about and makes a decision.

Through this miro board, you can better zoom in on the prototype. You can also post comments here, and feedback is welcome!

Prototype of the working method we will soon test with the Social Security Bank

Two questions

You can see that the accountant’s perspective has no questions yet. Here we are still searching. Auditors are very busy around February and March because of wrapping up the previous year. So we haven’t been able to include much input from that side. Do you know more about this or know someone who can help, let me know.

And of course: feedback in general. In this blog, I describe our making process and show the prototype. I’m curious what you think of this and how it would work in your organization. After design day 3, we would like to update our prototype and share it with organizations to use without us for self-assessment. So we want to learn from your findings in turn.

If you would like to help with this, please send me a message. You can do so through maike @ klipklaar.nl or the other known ways. Thanks!