Designing for Autonomy, Part 1: A Conversation with Jelger Kroese

This is part 1 of a two part post around designing for autonomy. Read part 2 here.

As part of our deep dive into the challenge of creating a digital twin of students from HvA’s physical education (ALO) and Elite Student Athlete programs to keep track of their wellbeing, mental health, and self-development skills, we held a conversation with Jelger Kroese. Jelger is a designer, creative technologist, and ambassador of Digital Society School’s Digital Twin track. Reflecting upon our conversations with Jelger helped our team challenge the idea of digital twins and deepen our approach in order to rethink digital twins as a reflexive tool. Read part 2 here for further considerations on digital twins for mental and emotional wellbeing.

Having worked with mixed reality for medical education uses at Leiden University, as well as creating interactive art installations —both independently and as a member of Shock Forest Group—, Jelger focuses on the responsible use of emerging technologies and interactive experiences for education, research, and storytelling. For example, he is currently working in a collaboration with researchers at Twente University on a project at the intersection between mental and emotional development with interactive media.

Providing digital tools and contexts that allow people to practice their reflexive capacity by building their own links of meaning was a recurring theme during our conversation with Jelger. Spanning a variety of subjects ranging from informational overload and extended cognition to guided meditation and note-taking apps, our discussion was a contemplation on what it means to design for autonomy.

 

Key takeaways from our conversation with Jelger:

• The challenge of information overload in fast-changing societies and the need to move away from categorizing the world as either noise or information.
• The central role of emotions and mental wellbeing in our perception of the world. Emotional resilience and development as key for individuals to practice agency in the digital world.
• Designing digital tools allowing users to create their own signals and build their own wisdom. Tools such as Obsidian support creating new connections between data points instead of simple collection for accumulation
• Reframing data gathering and Digital Twins to focus on users’ subjective experience and creativity rather than on profiling for behavior prediction. Digital Twins can be a medium guiding people into perceiving themselves in open-ended ways that take into consideration mind, body, and community.

 

Our Interview:

Inner Voice Team: Could you tell us more about your approach to tackle issues of mental health and emotional development when working with digital media?

Jelger Kroese: I think it’s incredibly valuable to work on this topic because, even if we might regard mental and emotional development as an issue exclusively related to health, I see it as quite a broader subject. Mental and emotional development is relevant to all sorts of people and not only to those who have serious mental health issues.

I’m currently looking into aspects more related to cognition, which has to do with information processing. There’s a quote by Edward O. Wilson that says something like: “We are drowning in the age of information while starving for wisdom”. Maybe he used slightly different words, but what he meant is that we have so much information all around us, but there’s also a lot of noise we’re unable to process. I think this is an important question nowadays. The people who are effective in processing all this information in meaningful ways and synthesize wisdom from it, will be the people capable of holding power.

In terms of well-being itself, I think that if people are constantly overloaded with stimuli, they will not feel well either. So, this and the previous aspect are closely related. Furthermore, there is also another aspect about emotional intelligence and development, which also feels very relevant now. Given that more and more science is showing how important our emotions are in our everyday experience —how they play a big role in shaping the way we perceive reality—, I believe that it’s incredibly valuable to develop them. Especially now that we live in a society that’s changing so quickly. Social change is necessary, but that also requires people to get out of their comfort zones. Get out of the things they know. And to do this, people require emotional resilience and development.

The process of learning to perceive emotions seems to be not too different from learning to perceive other stimuli, such as sounds and visuals. With training emotional intelligence you basically enhance this part of your perception, allowing you to better find a meaningful signal in the flow of information. Similar to how you can, through conscious attention and practice, train your vision, smell and taste to quickly distinguish ripe from unripe fruits in your garden.

Inner Voice Team: We have also pondered about the fact that we live in a society oversaturated with information and, yet, we do not have effective metaphors to synthesize knowledge from it.

However, we are not so sure that the distinction between noise and signal is that clear. This distinction would state that there are a specific kind of stimuli from which we can obtain knowledge or information, and then there are other stimuli from which nothing can be obtained. Translated into political terms, this might imply that there’s a clear-cut way to determine (and actors with the power to do so) which stimulus falls on each side. Perhaps what is happening in this information age is not so much that we are overloaded with noise, but that the distinction between noise and relevant information is not useful anymore. Not only relevant information can come from what is apparently noise, but maybe what this informational overload does is push us to think of knowledge not as a specific content extracted from favored stimuli, but as the relations established between them.

So, if we want to coin new metaphors or new ways to synthesize all these stimuli, maybe we should try to deviate from those dualities that categorize the world as either noise or information.

Jelger Kroese: Definitely. I agree and can imagine that that sort of talking about signal versus noise sounds like there might be one governance structure that states what is relevant. It is not like there is this top-down governing structure that determines what’s good to know for all the people. Indeed, there is a lot of valuable knowledge coming from unexpected places. The importance of context is key here. What is noise in one context can be signal in another.

In my view, it is important to acknowledge that everyone is part of a unique context and  to give people the autonomy to build wisdom relevant to their context. I don’t think everyone’s unique context exists in a vacuum. People are in constant interaction with each other, thereby shaping each other’s contexts. And I believe that there are fundamental laws of nature shaping our context (although they might be different from how we, or modern science, currently thinks of them). Therefore I think that there are general principles that are relevant to all or a range of different contexts, and that there are ways of looking at the world that are more in line with the laws of nature than others.

So, I do think that there can be a form of hierarchy in the usefulness of different points of information, its just not clearly top-down structured. Some piece of information can be extremely valuable to a very specific context. Another piece of information might be less valuable to that specific context, but useful across a big range of different contexts. Perhaps, would it be better to speak about the generalizability of a piece of information, instead of signal vs noise?

I think that, through a combination of having autonomy in building wisdom for their own context, and actively connecting and exchanging with others, people can get insight into generalizable principles and concepts. Interestingly, giving people autonomy might enable us as a species to explore the validity of a piece of information in as many unique contexts as possible, thereby doing the most rigorous testing of their generalizability.

There are all these different tools emerging for maintaining your bookmarks that are also connected to note taking. I’m thinking about tools like Obsidian. These are very interesting to me because, in the case of Obsidian, it’s basically a note-taking tool designed to build your own Wiki. It has a bit of a learning curve, but —at least for me— once I got the hang of it, it motivates me to create my own signal.

It gives me the autonomy to explore my own thoughts and ideas, but it also stimulates me to think about them and develop them in deeper ways. Before, I used different systems based more around the idea of collecting. There was no system motivating me to reflect upon what I had collected and then connect it with other ideas I also find interesting. By actively working with information in a way that lets me re-synthesize things, it is way easier for me to find relations between pieces of information and realize that many of the points and ideas collected are actually convergent.

Inner Voice Team: The go-to solution for informational overload tends to be a strict differentiation between relevant and non-relevant, facts and fake news. Yet digital media don’t seem to work this way. Unlike one-sided media such as newspapers and T.V., the internet (or data-driven media) gives primacy to user generated content and personalized experiences. The problem is not that there is a lot of  information or that this information does not come from the right sources, but the way in which this information is framed as a resource to be endlessly produced and exploited. So, instead of trying to neatly sort out overflowing data, perhaps we should try to produce higher orders of meaning around data-driven media. This would mean to gather and use data purposefully: focusing on conveying ideas or emotions instead of on producing value by incentivizing exploitable behaviors.

For our project, we are trying to reframe data gathering as an interactive narrative experience — in this case, one that fosters reflexivity. We are trying to distance ourselves from more traditional ways of giving and receiving data, such as filling out forms, or more exploitative ones, such as liking, sharing, and scrolling on social media. We would like to place the emphasis on the potential of data being something more than a set of discrete points to be categorized; more specifically, on data collection and reception as an embodied experience. In this way, the focus is placed not so much on the gathered results, but on what the user is doing and going through (what role is she playing?) during the process of these data points being generated, analyzed, and displayed.

Data collection thus becomes something more than a means to an end (accumulation for prediction and control) and starts resembling a communal participatory experience; a conduit of coordination between diverse subjects and the contexts surrounding them, rather than a mechanism to control or predict them. An experience more akin to dancing at a party or watching a movie in a theater than filling out a form. Seen this way, data gathering resembles an art form: rhetoric, storytelling, and stylistic decisions —how you present the context you are embedding the user in— become central and not just accessories for an ulterior end.

In similar ways to artistic creation, generating your own signal is, in a sense, not about seeking a static objective truth, but it rather asks you to go within and delve deeper to finally go out, in an attempt to express your own subjective worldview in objective terms. Exploring within and expressing out is a methodology borrowed from certain brands of psychology and phenomenology that might come handy for designing a conception of truth and autonomy that can be fit for the digital age.

Jelger Kroese: The interplay between the subjective experience and objective world is very relevant. Creating your own signal might be a way to describe it, but this interplay also recalls the idea of filter bubbles. In some ways everybody lives in their filter bubble, but I do want to clarify that creating your own signal is different from a filter bubble. The latter are very much controlled by algorithms we don’t have power over. While, on the other hand, creating your own signal is about motivating people to make their own wisdom, through critical self reflection, comparing and exchanging with opposing views, and building upon it. So expressing your subjective experience in objective terms is not about shouting into the world. It is a means to critically examine your experience and contextualize it within the objective world (or other people’s experiences) in meaningful ways.

Inner Voice Team: Would you call this, or would you find it useful to call this act of articulating your own experience in objective terms, creating your own signal, a Digital Twin?

Jelger Kroese: In a way, right? If you look at the terminology of “building a second brain”, that’s in a way what you are doing in this scenario. This is a way to be in constant interaction with your digital counterpart.

Claiming to “build a second brain” or “make a Digital Twin of the brain” is already very confusing terminology because this raises the question: is it really only our brain that we are copying? I believe that knowledge is also embedded in other parts of our body; there is not necessarily a strict distinction between the brain and the body.

This approach would not start from the objective or physical reality. Instead of trying to make a computer model with exactly the same connections as your own physical brain and trying to have a copy of your already existing self, it is more about writing your thoughts and experiences in a computer system and interconnecting them.

Through that process, you reflect on your own thoughts and create different ones as you interact with this system that you create yourself. So, it’s not really creating a copy of your brain, I would say, but rather it’s more about creating a sparring partner for yourself.

Inner Voice Team: The primary use of Digital Twin technology is to create a model of industrial processes in order to optimize them and/or prevent any possible malfunctioning. However, for our project we are working with people who are not necessarily driven by cost-optimization, especially when talking about their emotional development and mental health. How is this technology and terminology from industrial contexts relevant to mental health? How do we make sense of this jump from one context to another?

 

Jelger Kroese: Indeed. These two contexts are very different from each other. In a factory setting you have clear measurements and outcomes to optimize for. The technology used for these purposes can be quite high-tech, but its goal is, in a way, simple and straightforward. Within the context of emotional development and mental health, the outcome feels more open-ended. And I think that’s because people have different goals regarding these topics. So, the question becomes a more philosophical one, which is: what motivates you? What is the goal that people want to and should optimize for? Is it being happy? Or are personal growth and development goals in themselves?

However, the idea of optimization is not completely out of the picture when talking about wellbeing. I feel that all of these tools, all these calendar or note-taking applications, emerge and become very popular because professional users make use of them within contexts where they need to become more efficient. This leads to people wanting to make their way of thinking and planning more efficient too. And I think that, when used with the right mindsets, goals and wisdom, these tools can enable more efficient self development.

Inner Voice Team: Going back to the idea of regarding mental activities as processes in which factors beyond the brain are involved, the philosophical question about the teleology or end goals of the mind can be opened to other bodily and even social processes[1]. So, in as much as the mind is the result of networks of relationships (both, organic and artificial, individual and social), which in this case are contexts of value production and optimization, this “human-arte-fact coalition”[2] does aspire to become more efficient.

But, then again, there’s this evolutionary biology commonplace for which the brain, as a biological organ seeks to act in the most optimal or cost-effective way. According to it, humans detect patterns and create metaphors to synthesize (and therefore simplify) the variety of phenomena in the world as a method of cost reduction. Yet, this view paradoxically ends up describing biological mental activities as external, economic, social processes.

So, in this last sense, comparing the biological brain to industrial settings is indeed a risky categorical leap. It assumes that, both, brains and technical systems work as self-regulating feedback loops. Not as a result of participating in the same extended cognitive processes, but because the brain happens to be ontologically analogous to the artificial world that surrounds it. This analogy is not casual, it is one of the main proposals of cybernetics and behavioral psychology — disciplines to which many of the contemporary commonplaces around the brain are deeply indebted[3].

In brief, many of the preconceptions we have about the mind and brain entail describing them as external processes, be it informational, economical, or political. These descriptions already imply an answer to the question concerning our goals and purposes. As designers, we are implicitly deciding on these preconceptions and play a huge role in determining how people understand their mind and its wellbeing. Or, in other words, on how they engage in these technological-organic coalitions.

Linking this to the aspiration of going beyond data collection, perhaps what a Digital Twin can do as a sparring partner is to enable connections. It would reframe the act of self-reflection as an assisted navigation through an externalized digital mind — connecting emotions to a network of body functions, situations, contexts, or ideas. Self-knowledge becomes in this scenario the technical mapping and navigation of yourself among others.

Jelger Kroese: Yes, and I think that for people to themselves engage in the act of connecting is important here.  Through this process, people become aware, on an experiential level, how interconnected they are. I think this might stimulate a nondual perspective on one’s experience, where there is no clear distinction between self and environment. This state of nondual awareness is actually central to some ancient contemplative wisdom traditions, from for example Tibetan Buddhism. These ancient practices are now becoming quite popular through some meditation apps. I think this indicates the value that this brings to people’s well-being. Another interesting thing I find about meditation apps is how they incorporate autonomy. They are basically a medium instructing and guiding people through an experience only through speech, while users retain quite a lot of autonomy to shape the experience themselves. They are only guided by a voice, but whatever they focus on or notice exactly is up to them. I can see this as having a similar function to journaling but prompted and guided by specific questions. This might seem simple but feels powerful to me.

 

Written by Jordi Viader Guerrero: I am a practice-based researcher on philosophy of technology, internet culture, and new media. My work is chiefly focused on articulating digital culture within wider cultural, political, and epistemic logics. By conceiving technology as a tool of production, destruction, and expression of identities, I am drawn to issues raised by our inescapably “technological nature”. Most recently, I attempted to use TikTok’s scrolling UI to formulate a general theory of scrolling itself. At DSS, I am currently creating digital doppelgangers to allow us to experience and understand our emotions in new ways.

The Inner Voice Team:

• Nikta: Tech-savvy product manager interested in human-centered design and innovative high-tech companies.
• Reza: A solution architect.
• Jin: New media student who is in media art, music culture, and every creative industry.
• Kofi: Geological Engineer and data enthusiast, focused on geological data analysis and map generation.

*Coach: Mariana Pinheiro – Industrial Designer.

Read part 2 here.

See our project page here.

References

[1] For the canonical presentation of the Extended Cognition hypothesis see: Clark, Andy, y David J. Chalmers. “The Extended Mind.” Analysis 58, nº 1 (1998): 7–19.

[2] Michael Wheeler formulates the extended mind as “dynamically assembled, organic-technological systems’’. Wheeler, Michael. “Thinking Beyond the Brain: Educating and Building from the Standpoint of Extended Cognition.” In Alleys of Your Mind: Augmented Intelligence and Its Traumas, by Matteo Pasquinelli (Ed.), 85–104. Lüneburg: meson press, 2015.

[3] See the second part of this post for further elaboration on the relationship between cybernetics and behavioral psychology.