Project

Data Shoes

DESIGN CHALLENGE

How can we track shoe use through artistic & digital methods of wear and tear?

We know that like our fingerprint, the way we walk and the shape of our feets are different and unique to each individual and that these differences can be seen in the ways our shoes wear down over time. But what can we learn about the user from these wears and tears, and how can we use those learnings to create a self-improving system that helps to better deepen the relationship between the user and their shoes as well as improve the quality and sustainability of existing shoes and its production methods.

Research Questions

We converged our understanding from this to three important questions which we expect will serve as guides and enable us to proffer answers or if you will; solutions to this idea. These questions include:

  • How can we capture and analyze data of the shoe while it is being worn (co-analysis)?
  • What model can we create with which to predict further use (profiling)? and
  • How can we create the data flows that will allow the designers to improve their designs (co-design)?

Sustainable Development Goals

The project focuses on the following SDG goals: Industry, Innovation & Infrastructure and Responsible Consumption and Production. This is because the project aims to transform industry to become more circular through domain specific innovation. At the same time the”responsible” aspect of goal 12 can also refer to the sovereignty and protection of users/consumers/citizens versus a disproportionately powerful industry.

  

Research

Materials – wear and tear

We went through the materials and processes that go into making different parts of the shoes to try to understand which and to what extent the materials are recyclable or non-recyclable and to understand what sustainability looked like in the whole system. Our findings showed that the majority of the materials that go into shoe manufacturing are non-recyclable meaning that it is almost impossible to break down a shoe to its part and rather than recycle, most of the shoes at the end of use, are incinerated thus showing the unsustainable nature of the shoes ecosystem.

Technologies – data collection methods

Following the maker’s sprint, we engaged the project objectives, firstly to find ways we can capture and analyze data of the shoe while it is being worn (co-analysis). To achieve this objective we began by reviewing the current state of the art of the technology used in shoes. The aim was to find out what technology used during design and manufacturing of shoes made it possible to track their wear and tear. We also looked at what technology by means of apps/softwares or artistic design we could use to collect data about wear and tear and what ubiquitous technologies we could repurpose to achieve the same goal while the shoes are in use. 

And secondly to create a model with which to predict further use (profiling). For this we looked at methods/models with which we can analyze the data we get and at the same time make predictions about wear and tear.

Playing the Loopholes Game

From our research, our understanding of the shoe manufacturing industry was that of an open system in which the materials, the production methods as well as the disposal methods currently being used does not present an ideal sustainable ecosystem. In order for an ecosystem to be sustainable it needs to be a circular system where improving longevity of use and clean recycling at end-of-use is an important part.  Following our research, bringing all these ideas together became really important to us, so we tried to brainstorm various interventions within a system framework that brings to life what a sustainable shoe ecosystem looks like through identifying wear and tear to improve longevity of use and enable a sustainable end-of-use practice. And what better way is there to do that than playing the Loopholes game. We found out that our multiple ideas could fit as various interventions within a system framework. This understanding led us to creating a systemic solution to help us track the wear and tear on shoes.

   

Re-imagining the scope

Our research further pointed out that of the many shoes and shoe manufacturing industries out there, the sneaker shoes industry, despite being the most popular and universally used, is the most unsustainable in terms of the materials used, manufacturing process and end-of-use practices. Since sneakers are the most commonly used shoes as well as the ones creating the maximum carbon footprint on this planet, we decided for the time being to narrow down our scope and focus on them.

Ideation

SYSTEM 1.0

To help us ideate the right system for tracking shoe wear and tear, we looked to answer three important questions including, what type of data we need, how do we collect this data and finally how do we intend to use the data collected? We used these questions to serve as a guide to help us create not just an ideal system but one that is usable and testable.

The system we developed to track wear and tear on shoes can be broken down into 3 different categories taking into consideration the different aspects of a systemic solution. Our visualization of the system in the form of a system flowchart can be broken into three steps which includes the following:

  • Data Collection System – During our brainstorming session with the loophole game we identified that tracking shoe wear and tear can be done either on universal shoes or data enabling shoes. Universal shoes are the sneakers and as we have previously found are the most commonly available shoes while the data enabling are shoes that by design enable data collection for the wear and tear,  for instance, spiky shoes, layered colored sole etc.. Since very few people tend to use the data enabling shoes, we wondered what a system made to collect wear and tear data on universal shoes would look like. Firstly, we also looked at the type of data and format we could collect as well as the already available ubiquitous technology that we can use to collect said data. We identified that we can collect data in the form of  image capture either in 2D or 3D using many readily available and ubiquitous systems such as the phone camera (2D) or scanning using the Lidar software (3D). However, despite the quality and depth of information possible with the Lidar scanning, we quickly realize the difficulty associated with analyzing 3D data. In addition to that, we came to understand that for 2D image capturing, we could use scanning and photography but it could be challenging to get an accurate measurement of the shoe wear and tear (which could be in millimeters) using either data collection method in an uncontrolled environment. To create and test out what a controlled environment would look like, we prototyped a device that gives us consistent and comparable data.

  • Data Processing – current computer vision technology to train an object detection model to detect the wear & tear on the shoes, e.g. staining, tearing, wearing, etc. We explored different image processing models that we can use for object detection and image segmentation. Image differentiation is another technique that we found could show the changing process of wear & tear in a chronicle perspective. To create a usable data processing system, we need to process the raw image data that we get from the data collection and create annotation labels e.g. fabric wear, staining, tear to train our machine learning model to detect wear and tear. After training, the model should be able to do the automatic annotation and detect wear & tear such that we can process more images and get insight from the data.

  • Insightful Results – After having the image processing results, we can get the wear and tear patterns for users. We analyze these patterns using machine learning and combine it with the user’s autoethnography to create a personalized profile. The insightful result is what we termed “Mary’s Shoe Box” and her shoes’ stories. The idea of the Mary’s Shoebox is a chatbot that generates questions from the personalized profile to interact with the user and update the database to not only improve the accuracy of the model’s insight but also gain more information that would be useful for the production of shoes personalized for the user.

To build this model, we need to look at a system that collects image data from shoe users, uses machine learning to annotate these images for different kinds of wears and tears and create a user profile for each unique use-case. In this sprint review our partner nudged us to look at the users motivation to click and upload these pictures on a regular basis. So we ended this sprint with the following question:

How might we motivate users to click and upload pictures of their shoes on a regular basis?

AUTO-ETHNOGRAPHY

To dive deeper into understanding the user behaviour, as researchers, we went on a journey to discover our relationships with our shoes by ways of tracking what shoe wear and tear means to us. We realised that all of us had different relationships with our shoes which led us to try different ways of tracking our shoe use.Using autoethnography as a common research method we explored our feelings, emotional connections and memories of our shoe use. We also detailed the intended and unintended uses of our shoes to understand the kinds of relationship we have with our shoes.

The influence of my shoes on my fashion choices – Osaze

Through the autoethnographic process, Osaze discovered that he has been wearing the same pair of shoes for the past seven months which made him wonder if there could be a deeper reason that made him wear this specific pair of shoes. Documenting this journey with his shoes led him to take pictures of his current shoes where he noticed that the colour of his pants had been along the colours on his shoes. To find out if this trend/behaviour has occurred in the past or if this was a recent occurrence,he went through his old photographs to find images where he had his shoes visible. He saw a trend of his wardrobe slowly shift as he changed his shoes. The colours of his pants and t-shirts slowly merged with the colours of the shoes. This led to an interesting revelation on how his shoes after long term use influences his fashion choice and how that affects his idea of wear and tear and the decision to buy new shoes.

How does my shoe read a city? – Geetanjali

Using autoethnography, Geetanjali traced and tracked her relationship with one specific shoe, a pair of Clarks black leather derbys. Moving from one city to another, she wondered if her shoes were wearing out differently. How does her shoe read a city? Can her shoes tell if they are being used in Delhi or Amsterdam? She mapped the journey her shoes were currently taking in Amsterdam and documented the “shoe-city” interaction by observing the ground while walking and taking pictures of every new surface her shoe came in contact with. She wondered if these patterns would be remembered by her shoes. She explored  data visualisation methods by extracting the patterns of the surfaces and how these are imprinted on the shoe sole and analysed how these patterns change over the course of a walk through the city.

What to do with lovely but useless shoes? – Maryam

Maryam started her auto-ethnographic journey by taking out an old pair of Reebok shoes with which she had a deep emotional attachment as they were a gift. As the shoes were used by her in a leather workshop, they had many stains on them which led them to be in the closet for three years. She had an urge to find a way to wear them. She tried washing them, using DIY techniques to clean them and even looked at ways of  upcycling them. She went on an introspective quest about looking at how much damage is too much. At what point does she stop using the shoe, and at what point does she throw it away?

How to build a Shoe Network from texts using NLP tools to reflect the journey with your shoes?- Wendy

Wendy began her auto-ethnographic journey by journaling about her shoes, writing down special events and memories associated with each pair. She was curious to process the journal text using NLP tools and build a unique shoe profile and visualise the sentimental meaning of her shoes in a “Shoe Network”. Each shoe profile embeds the memories and life journey between her and her shoes. She further explored how the creation of a shoe network helped her re-establish her relationship with her shoes.

How can I bring my old shoes or their memories back to life?- Sahil

For Sahil, the auto-ethnographic process began with introspecting his personal experience with shoes that have reached the end of their life but he is not ready to let them go. He describes a state of limbo for 2 pairs of sneakers that are not being used but are also not being disposed of. This leads him to explore using text-to-image AI models to generate visuals of how these shoes can be repaired. To explore this further, he used Dall-e to visualise the memories and generate visuals of how his shoes can be re-imagined.

Through this self-analysis step (blog writing), we reflect on challenges with our shoes based on different environments and life experiences. Each of us implemented different tools to present our shoe data, such as AI image generation tools, NLP sentiment analysis, decision tree, ground texture extraction, and wear & tear timeline.

SYSTEM 2.0

Looking at how much our relationships with our shoes improved from embarking on this autoethnographic journey we wondered how we can get others to do this (autoethnography) in an easier and more consistent way. This led us to ideate an AI-assisted tool for autoethnography useful for data collection in our partner’s research. This tool is designed keeping in mind the user motivation of a research participant.

We created the following pipeline of data collection system :

  1. collect shoe journal from participants
  2. use NLP tools to generated summary of the journal
  3. use the summary as the prompt to generate AI image/comics

We prototyped Geetanjali’s blog using the above method and the results are as follows : 

This led us to create a web demo for the Digital Society School showcase. We will use the showcase as an opportunity to test our web prototype. Come join us for the showcase! Here’s a link to our teaser! And here is the link to our final prototype : https://datashoes.dss.cloud/

 

Conclusion

Through our research and the creation of the Memories tool, we hope to encourage individuals to reflect on their own shoe usage and the emotional connections they have with their shoes. By fostering a deeper understanding of our relationships with our shoes, we believe that we can make more mindful decisions about when to keep our shoes and when to let them go.

In addition to promoting more sustainable shoe consumption, the Memories tool also has the potential to serve as a platform for personal reflection and storytelling. By sharing our personal comics, we can learn more about the different ways that individuals interact with their shoes, and connect with others who have similar experiences.

We hope that the Memories tool will inspire individuals to think critically about their own shoe consumption and find ways to reduce waste and minimize our environmental footprint. Overall, we encourage people to continue the journey of tracking our relationships with our shoes and discovering ways to preserve them or bid them goodbye in an eco-friendly way.

Project Team