Rebecca Carlson, Ruth Dorothea Eggel, Lina Franken, Sarah Thanner, Libuše Hannah Vepřek

Approaching code as process:

Prototyping ethnographic methodologies 

Some research fields force us to look at computer code^[1], for example, when we study coding practices, programmers or software engineers. Yet in many research fields we encounter code in a more subtle form, as it is the ground for every digital phenomenon. Code is not a defined category but rather a methodological perspective that should be considered in every anthropology of the digital. It is a perspective we deliberately include for a necessary understanding of digital practices. But how can we approach code methodologically?

In October 2020, the authors, together with 10 other participants, came together to discuss these perspectives in the workshop “Approaching Code Methodologically”^[2]. In cultural anthropology and related disciplines, a general understanding of code as being entangled in socio-material networks (Haraway 1991) has been established and there exist several theoretical conceptualizations of source code and algorithms (a.o. Gillespie 2014, Seaver 2017, Introna 2016). Nevertheless, many studies turn a blind eye to the processual nature of code itself, which, in its meaning-making processes and practices, form the focus of ethnographic approaches. As of yet, there is no general methodology for including code in ethnographic analysis. This article, which forms the first result of a subsequent and ongoing collaboration to develop a methodological approach for the analysis of code, explores how we can grasp code theoretically and empirically in ethnographic research.

Our individual perspectives are connected by a common theoretical understanding: We neither accept code as a black box which cannot be interrogated, nor do we see computer algorithms as neutral, objectively identifiable objects (Gillespie 2014). Code is an integral part of all software; however, it remains largely invisible (Kitchin & Lauriault 2018). Everyday practices mostly accept software as complete as it is. Also, we researchers often consider digital technologies to be a complex matter, an unknowable depth, unless we have expert programming knowledge. But we are surrounded by software that shapes our practices and perceptions of the world. As such it is an important part of the everyday life we study in cultural anthropology.

Code comes with its peculiar implications. Even the term software does not refer to one item in a particular form. Instead, code is an assemblage of multiple layers in polymorph existence; of parallel aspects that are mutually dependent and interlinked. We, thus, build on the theoretical concept of assemblage (Deleuze & Guattari 2013; Collier & Ong 2005) to grasp code in its heterogeneity, its multiple dimensions and layers, distributed agencies, as well as interconnections and mutual dependencies. Assemblages are neither static nor given but constantly (re)produced, performatively constituting phenomenon in ways that stay unstructured, messy and therefore harder to catch. With a focus on social, material and semiotic aspects, ethnography allows us to apprehend the interplay of different human and non-human entities that together form the assemblage of code. Moreover, if we take on the theoretical assumption that subjects and objects come to be through practice, we can see that coding and the doings of algorithms (Introna 2016) performatively take part in the constitution and (re)negotiation of subjectivities and objectivities.

Code is not a product, but a non-human meaning-making agent with their own history. We can neither limit our analysis to its material form nor to the human agents dealing with it, but instead must focus on the dynamics, interdependencies and interconnections between them (Reichert & Richterich 2015). To stress the processual character of these assemblages, the (re)production and forming of code through different intra-actions (Barad 2007) are seen as agents that “emerge through and as part of their entangled intra-relating” (ibid.: ix). In a similar shift, Timothy Ingold suggests that where we see making, we should consider instead the act of weaving; as objects are neither wholly made nor grown, in this dualistic sense, weaving is the embodiment of process (Ingold 2000). With regard to code and coding, this focus allows us to study the entanglements that code performs within different fields.

These theoretical considerations form the common entry point of our individual approaches, which cover a variety of fields: bioinformatics in a bioscience laboratory, the role of source code within digital humanities, new forms of human-code collaboration in human computation systems, the development of digital augmentation of physical artefacts, and everyday life encounters with code when surfing the web. Throughout, we show how the assemblage and acts of weaving as theoretical perspectives allow us to encounter code in our different fields of study, in effect, prototyping ethnographic methodologies.

The Circulatory System of Coding (Rebecca)

Bioinformatic data starts in the body: in tissue, blood cells, a snippet of DNA. In a method that still seems like magic to me, these pass from physical state and are digitized in biological experiments that transform liquid containing DNA into endlessly manipulatable computer data. The body, however, is no easy metaphor for a computer. That is, unless we look at coding, and not only code: coding as a moment of translation, a liminal state, a point along a rhizomatic path; a distributed process of intersections that, like the body, has integrated networks and systems of vessels, and even a kind of bone, working in service of transformation and pushing of objects and information across space.

In my field site, a bioscience laboratory near Tokyo, the body is more than a metaphor. Bodily materials arrive in the lab and are manipulated by scientists – often, but not always, separated in time and space from their more meaty context. For example, in a cell line that is grown, almost as endlessly, and with just as much trial-and-error effort, as a string of computer code in a state of debugging. The assemblage of the body then is more than a theoretical musing. It is a directive. It is a demand to attend to the paths and the vessels. To the comings and the goings. An ethnographic methodological assemblage for coding then demands attention to the circulatory system.

In bioscience, however, sometimes it's easier to see the outcomes – data and the objects produced by experiments and code – as stable facts; to get tripped up on things appearing as if whole and unaided, like phylogenetic trees (lineages of genetic divergence) emerging from the programming packages and the scientists that compute them. After all, for ethnography, a phylogenetic tree can say just as much about the programmers who coded it, about the way they might expect the world to be logically ordered, as it does about the evolutionary processes that coding, in this case, is meant to unravel.      Although I know a bioscientist who disagrees with this statement.    

But I am finding in my research that bioscience is just as unstable as code, just as open, just as much about closing down and just as incremental. For example, I am learning that bioinformatic scientists don’t really arrive at or depart from code as a settled thing, even if disciplinary habit considers it finished and fixed upon academic publication. Instead, scientists keep working, not only to make a better coding tool, but to make a tool that can open new ways for scientists to “see” the body. Along the way, scientists negotiate consistently unstable but tightly networked material and social relationships: international consortiums which hold vast genomic data captive, month-long data downloads, access to supercomputers, “research environments” hosted on for-profit platforms, an auteur idealization of the self-taught coder and a hesitancy, as for any maverick, to share pipelines in development.

To capture the elements of this indeterminacy requires a kind of system mapping, a piecing together of ethnographic data from diverse and seemingly disparate loci. What’s needed then isn’t really new ethnographic methods, but a vision for this linking. Afterall, I attend mostly online lab meetings and the “free” bioinformatic courses I enrolled in online proved more useful for understanding how code is imagined by scientists than for learning Python. Coding as circulating assemblage, as weaving then, uses ethnographic methods to see the cell in the body, the experiment, the bench, the long-read DNA sequencing tool, the scientist, their performance of expertise at an international conference, the cultural value of scientific truth-making, and then demands we look at all those moving as well in reverse – not as foregone conclusions, not as finished through code, but in an ever state of reciprocal becoming.

Researching the entanglements of code in a field where everybody seems to learn coding (Lina)

Contemporary research is nearly impossible without taking digital methods or computational assistance into account. Researchers use computers and software, whether for data capturing or writing. Digital Humanities form an interdisciplinary forum for the unfolding of reworking existing methods with computational support and developing new methodological approaches (Schreibman et al. 2016). Within this field, the focus lies on the development of new scripts that enable research to hand over specific parts of the work to a machine, e.g., tagging specific concepts or image labeling. Within my research, I focus on how the development of methodological approaches and digital possibilities are interwoven.

During my fieldwork I participated in a research project on the implementation of Digital Humanities, experimenting with different approaches in an interdisciplinary setting (Gaidys et al. 2017). Moreover, I immersed myself into the field of Digital Humanities by attending conferences, smaller meetings and workshops, joining mailing lists and twitter discussions as well as bilateral cooperations. Looking at the practices, the talking and writing about (digital) methods and tools, I take a somewhat classical ethnographic point of view, combining fieldwork with interviews and discourse analysis to gain a deeper understanding of how research practices are sought to be questioned and enhanced.

In Digital Humanities, in contradiction to “classic” humanities, coding is part of everyday research or at least, this is what the community’s narrative promises. When confronted with methods applied here, I immediately stumbled upon coded scripts in Python, R and Java. Many of the researchers I got to know described their initiation within the community as being closely connected to learning to code. I as well took a course offered as “Python for Humanists” where basic steps such as counting and sorting the words of different documents were taught. But when completing the course, I had more questions than answers since I was able only to perform a bit more than I did with spreadsheets beforehand.

Surprisingly, reflections on how code behind these software systems determines the way research is being done are largely missing. However, I can collaboratively construct the form code might need to have when it should help to solve research questions, but this assemblage is incomplete without taking the agency of code into account, weaving its way into the research practices. When talking to colleagues and collaborators, they stressed the agency of code within their narrative, it was “doing” something or preventing certain research aspects by not performing the way it should. At the same time, code written by third parties plays an active role without the human agents who wrote it even knowing that the code entailed an important role within workflow setups, bringing itself into the assemblage through internet search queries and specific platforms for sharing open-source code such as GitHub.

When giving code its own agentic capacity within the intra-actions observed, multi-sited and multi-perspective ethnography is of enormous relevance, since otherwise we cannot capture the assemblages code is interwoven within. I do not need to understand the code in detail or construct it myself in order to get an insight in what it means to researchers and what active role this digital materialism plays within the assemblage of human and non-human actors. I need a basic understanding of the forms and structures code as coding can take, in order to collaboratively engage with it, since I am at the same time weaving in a specific perspective with my own agentic cuts (Barad 2007). With this, I can dive into a legitimate peripheral participation (Lave & Wenger 1991) within the assemblage Digital Humanities form.

Analyzing the human-code interplay in human computation systems (Libuše)

Within the realm of human computation, humans can now work alongside algorithms to, e.g., help advance research on Alzheimer’s disease with Stall Catchers^[3] or contribute to COVID-19 or HIV research with Foldit^[4]. Human Computation systems (HCS) combine the complementary strengths of humans and computers to address problems that neither of them can currently solve on their own (Michelucci & Dickinson 2016). These new forms of human-computer collaborations are often designed as games where “players perform a useful computation as a side effect of enjoyable game play“ (von Ahn & Dabbish 2008: 61). Whereas the technical mechanism of outsourcing computational steps to humans can be defined rather concisely, from a cultural anthropological perspective it remains unclear how such collaboration between humans and code unfolds and how the different actors ascribe meaning to these systems. In my research project, I investigate these new forms of collaboration in human computation-based citizen science projects.

Stall Catchers, for example, is an online video game developed by the Human Computation Institute in which players help advance Alzheimer’s research by simply playing the game. The human’s task consists of annotating short movies to identify whether blood vessels are blocked, which is related to the disease. Identifying these stalls is outsourced to humans as computers currently cannot solve this task in a satisfying way for scientific paradigms.

The participant’s task appears to be straight-forward. However, on closer investigation of the HCS as a social, semiotic, and material assemblage, a complex interplay between code, interface, servers, developers, designers, participants (and others) unfolds. To analyze this interplay, I approach the HCS ethnographically from different methodological angles: Aside from contributing to the project as a participant, analyzing chat data from the game and conducting qualitative interviews with participants, designers, and developers, I “co-laboratively” (Niewöhner 2016) research and work together with the Human Computation Institute as an intern.

Including the source code in this approach has proven to be essential. In what I call a “focused code analysis” I analyze specific sections of human-algorithm interplay in the Stall Catchers source code to gain a deeper understanding of the intra-actions at play. These sections are especially insightful to better understand the very human-code collaboration, which, together with materialities and the designer’s input, constantly form the HCS during gameplay. The importance of each element for the enactment of source code becomes apparent in moments of failure. When materiality breaks down, e.g., when a server is down, collaboration becomes impossible.

By including all layers and actors in their intra-actions into the analysis, we can analyze how algorithms invite user actions and how they control user input but also how users can subversively change the intended meaning of the interplay. The scoring processes of Stall Catchers, for instance, provide an interesting example here: Designed by the team of the Human Computation Institute, the scoring algorithm doesn’t simply reward participants with points by counting correct answers. Instead, each player’s answers are continuously statistically analyzed to rate their overall performance, sometimes with imperfect information (since not all correct answers are known). Moreover, this algorithm also realizes the designers’ aim of preventing harm caused by players submitting incorrect responses and further incorporates the player’s skill in weighting their answers in the final analyzed dataset. This way, players and algorithms together constantly form something new, a system tackling a problem that neither of them could solve on their own. Understanding this complex interplay requires analysis of the acts of weaving between various aspects, entities and layers, of which code turns out to be particularly crucial.

(Re)negotiating subjects and objects in projected augmented reality technology (Sarah)

In his essay on the mediating role of tables and chairs, Mark Kingwell (2011) describes tables as “machines for thinking” when reflecting on the characteristics of furniture, and, sitting at my desk, my forearms resting on its surface, my hands touching the computer keyboard, I can intuitively grasp his idea. Yet, in my own research, this relation seems to be a bit more puzzling, as the research group in which I am conducting ethnographic fieldwork, is turning tables  into “thinking machines”, or more precisely, equipping them with “artificial intelligence”.

When the first prototypes of interactive table surface technology currently developed in the project VIGITIA^[5] were shown to me, I no longer sat, but stood in front of tables, above each of which a camera-projector-system had been mounted. One prototype already recognizes oranges, carrots and smartphones, I was told, but sometimes still confuses bananas with surfboards. As stated in an early position paper, the project aims to develop hardware and software to support and enrich everyday life (inter-)actions at and around tables by interactively aligning physical and digital content on the surface (Wimmer & Echtler 2019).

The human computer interaction technology at stake here is called projected augmented reality (projected AR). For this purpose, the tabletop is both recorded and projected onto, with a depth camera providing measurements used for reconstructing a 3D representation of the scene. In this way, the hand of a user or an object lying on the table may serve as both, interface and input device through which real time interaction with the software, and thus, the multitude of processes constantly being (re)produced through code, is enabled. A human hand, for example, hence may be part of a subjective embodied perceptual experience of a human agent and at the same time be just as much an object for algorithmic comparison and measurement as the cup of coffee the hand might be reaching for.

In my research, I work towards an ethnographic account of the process of digital-material innovation in the project VIGITIA in terms of a socio-material assemblage and in order to do so it seems crucial to me to pay attention to the multiple interweavings of code with human and non-human practices.

After being involved in the research project as a project employee and conducting participant observation for around a year now, I listened to computer scientists talk about code in different situational contexts, I was shown lines of code, I observed people coding as part of their everyday work practices, and different metaphors have been used to explain to me how code works.

Moreover, I try to gain information on the measurement information data involved in the interactive table and on how they are constantly (re)contextualized and (re)translated by various actors. Furthermore, I observe people interacting with the prototypes and conduct interviews with different people involved in the project.

Currently, I investigate how the performative agency of code partakes in the enactment and dynamic (re)negotiation of subjectivities and objectivities in the context of projected AR technology and as such is shaping the relational intertwining of analogue and digital practices in everyday life (Koch 2017; Beck 2013). Hence, I am trying to take a look at the mediation of human and non-human entities and its relatedness to design decisions, materialities, measurement procedures and usage patterns as situated practices. This also includes investigating epistemic modes of knowledge production in applied media informatics and its relation to the affordances of code. Since I am actively participating in the design process and the project’s public outreach, e.g., in co-organizing exhibitions and deployments, I am as well      able to help shaping digital technologies myself.

Architectures of code in everyday practices (Ruth)

It is not necessary to understand computers to use them. But it is extremely helpful (Crichton 1983). This is true for code – as an integral part of computers – too, even more so in research. Code is helpful to understand and contextualize all digital vernaculars (Coleman 2010: 492) and technosocial practices (Raffl et al. 2008). In my research of everyday digital life as spatial practices, I understand code as an architecture of the internet (Lessig 2006). What buildings are to a city, is code to digital spaces. It brings history, socialites and power relations into manifested forms and shapes our practices therein. Therefore, code shows us affordances, but it also points out missed chances and impossibilities, as even a mundane action, like accessing a website shows:

A retired Lady in her sixties opens her internet-browser with a Google start page, types in the name of a website and does not find the result she is looking for. The search engine serves as receptionist and gatekeeper to “the internet”, and architectures of code shape what it encompasses for users and what not. The algorithmic results are shaped by advertisements and economic interests, technological knowledge and resources (e.g., for website optimization), legal decisions and personal data provided. In return, everyday practices of accessing a website via a search engine, influence law making and media coverage. Since 2016 an EU-law allows requests to erase personal data from search engine results, rather than deleting original content. Google’s code becomes an official watchman for the World Wide Web.

A woman in her twenties accesses the website by typing its URL in the address line, sometimes using a VPN client to mask the nationality of her own IP address. Practices to circumvent regulations become possible by understanding their enforcement through code. A different text input box on the interface level can decide on the (im)possibilities to access digital spaces. Website blocks e.g., for copyright infringements are often search engine blocks rather than URL-blocks. If URLs are blocked, it is frequently enforced by ISPs locally restricting access, e.g., from a certain country. A VPN client uses the same architecture of code (localized IP information) to mask the user’s actual location with the help of code.

A mid-thirties programmer types the IP address of the website, as a list of numbers into the address line. The many layers of translation of code become relevant, as an easily readable URL is translated into a sequence of numbers. The history of the internet is profoundly shaped by the history of TCP/IP protocols (Barloewen 1998:14). IP addresses as a numerical label become a key to a blocked URL for those with an expertise on the architecture of code.

By including code and knowledge of code into our analysis, the mundane act of web surfing reveals an assemblage of different people, institutions, soft- and hardware processes involved. Code helps us to understand the implications and limitations of digital practices in both visible and invisible processes. It allows us to see all the possibilities of actions as well as the impossibilities that are included therein. As such, it is only through a perspective of code, that we can understand the cultural embedded in the digital. Because everyday digital practices – even the smallest acts like accessing a website – are practices (inter)woven with and through code.

Code is never one – it is many. A multitude of processes

Code exemplifies not just contemporary digitally embedded experiences, but anthropological paradigms that have always attended to the processual nature of human life; attention to the structuring aspect of the materials, social systems, and ideological pursuits – webs in which we are bound. Coding is an example of structuring par excellence; it frames but does not predetermine human action. As a research topic, it helps us arrive at classic ethnographic questions, as our examples begin to demonstrate: the weaving of human bodies into code (Rebecca), the performance of expertise inside human/nonhuman networks (Lina), anticipated and unanticipated couplings and even subversive outcomes in forms of serious play (Libuše), dichotomous subject/object distinctions which begin to unravel (Sarah), and mundane, everyday forms of electronic resistance (Ruth).

Methodologically, there is no easy way or “how-to” solution to approach code. Because coding is a constant weaving of human and non-human entities, entangled in socio-material networks, it can only be studied with agentic cuts performed by researchers as well as by the subjects and infrastructures that make up the field. Code therefore is a methodological perspective we should consider in every anthropology of digital life. Obstacles such as lack of access to a certain part of the field or a lack of understanding of the assemblage and specific algorithms implemented can limit our ethnographic research. But when focusing on meaning-making processes and practices, “classical” approaches to the field can nevertheless provide deep insights. The focus on process and the perspective of weaving allows us to capture the different aspects and perform boundary work in researching code and its contexts within the assemblage studied. We therefore study the entanglements that code performs within our field sites.

Building on thinking that overcomes the dichotomy between separate entities and subjects, the theoretical understanding of code developed in this article can inform a practical strategy on what it means to weave code into methodological approaches. Ethnographic analysis is especially well suited to capture the rapid changes and dynamics in the codified world, including developments in quantum computing. It can also contribute a perspective on code that enables us to search for not only existing frictions and ruptures, but to develop a sense for what is missing.

For such complex interlocking systems ethnography is an essential research method. It helps us “see” code and its messy, uneven assemblage. Ethnography doesn’t just trace the outline of a somehow fixed and settled object, things that are finished, or made to stand in for. As assemblages are always shifting, they remind us that any ethnographic subject is inherently slippery. It is precisely about a tracking of that very knitting up, that human activity which objects merely come to replace and represent.