Dance Becoming Data: Part One Software for Dancers

The starting point for this contribution to the special section of Computational Culture on Computing the Corporeal is a relatively small cluster of research projects starting in 2000, which explored various roles that software and software development might play in the context of contemporary dance creation and performance. The inaugural project for which four choreographers, four software artists/ developers and additional guests were invited to take part was titled Software for Dancers.¹ The motivation for the project was the need to question more extensively the historical and cultural accumulation of bodily skills, sensory knowledge and tacit understanding, which would be the domain of dance artists, as a critical precursor to engaging with software. The title of the project came from the book Software for People: collected writings 1963-80 of the American composer Pauline Oliveros. Software for People was originally the title Oliveros had given to a paper she presented at an international seminar in 1978.² In the third part of the paper she presents some of the “theory concerning my ‘software for people’”³ in which she aligns ideas from psychology and information processing⁴ to explain her model for the organisation of sensory attention. Oliveros follows this by leading the audience in an exercise to help them “experience directly some of the theory I have been talking about.”⁵ Referencing a project that had taken place twenty years earlier helped establish the framework for a conversation about practices and their histories for this group of choreographers and software artists/ developers meeting in the early 2000s.⁶ Not only to deepen this conversation from the perspective of the dance artists, as already mentioned, but equally to engage in a dialogue with the invited software artists about their processes.⁷ While the group ostensibly shared the task of developing concepts for rehearsal tool(s) for dance, there was the chance in this exchange to question assumptions on both sides, including the assumption that software has to be useful.⁸ Two pieces of writing emerged from the project, one is by dance critic Sanjoy Roy, who was invited to join the project and report on its outcomes. Roy’s article, published in Dance Theatre Journal, explores the results of the shared task of developing a rehearsal tool together, to make something functional for choreographers. The other writing, titled “Software for Dancers: Coding Forms” (deLahunta) takes another perspective, contemplating the implications of software as a material and coding as a practice. This is more aligned with the approach of Software Studies as articulated in the introduction of Matthew Fuller’s edited book, published in 2008, where he writes: “programming is also a result of a live process of engagement between thinking with and working on materials… ”.⁹

CODING AS A PRACTICE

The Software for Dancers project made possible a conversation about creative process in dance to take place between choreographers and coders; a conversation focused on methods, approaches, histories and contexts. This conversation remained the central feature in a cluster of four research projects that would emerge over the next decade. Each of these took their starting point from a particular dance artist: Wayne McGregor and Siobhan Davies (two London-based choreographers who took part in Software for Dancers) Emio Greco|PC from Amsterdam and William Forsythe working out of Frankfurt, Germany. In various ways each artist and the interdisciplinary research teams coming together around them were embracing digital technology as a means for furthering their particular research enquiry, whether on how to create an artificially intelligent choreographic agent (McGregor); how to notate and transmit inner intention (Greco); how to help audiences see complex choreographic organisation (Forsythe); or create a digital archive that shares the raw material of dance making (Davies). Between 2005 and 2010, these projects were all under intense development as captured in two articles published in Performance Research (deLahunta and Shaw). The first published in 2006 opened with a discussion of the potential of dance documentation as exemplified by these four projects; the second was published in 2008 as an update on the developments of the last two years. In 2008-2009, selected researchers involved in these projects came together for a series of three workshops titled Choreographic objects: traces and artefacts of physical intelligence¹⁰ centring on the output of these four research teams bringing “choreographic ideas and processes into newly productive exchanges with both general audiences and other specialist knowledge areas.”

TWO CASE STUDIES FROM CHOREOGRAPHIC OBJECTS

As mentioned at the outset, Software for Dancers grounded its initial reflexive questions and concerns in deepening understanding of the embodied practices of dance artists and bringing this into a relationship with coding knowledge and practices. This project laid the ground for the Choreographic Objects projects to follow, two of which are described here in some depth.

CHOREOGRAPHIC AGENTS

The departure point for the project with London based choreographer Wayne McGregor was the concept of building an artificially intelligent choreographic agent, to “generate unique solutions to choreographic problems and augment McGregor’s creative decision-making processes in the studio”.¹¹ These developments were supported by ongoing interdisciplinary research with cognitive psychologists focused on studying the ‘choreographic and physical thinking’ of McGregor and his company in the context of creation.¹² The Choreographic Agents project went through two main iterations involving a significant amount of software development by digital artist and developer Marc Downie/ Openended Group¹³ working in collaboration with digital artist composer Nick Rothwell/ Cassiel¹⁴. The first iteration was given the name Choreographic Language Agent (CLA). The CLA was completely implemented within FIELD, an “open-source software project initiated by OpenEndedGroup, for the creation of their digital artworks”.¹⁵ The CLA was designed for exploring variations in choreographic instruction, drawing inspiration from how McGregor and his dancers work with visual imagery to generate movement material.¹⁶ The CLA was itself considered a “programming language” that emphasised “transience, ambiguity and creative flow rather than the conventional requirements of (… ) software engineering contexts.” It was designed with the aim of bridging “the intellectual and embodied improvisation aspects”¹⁷ by using language, grammar and syntax to build a complex 3D geometric form whose behaviour might not be entirely predictable. In this sense, the CLA functioned like a sketching tool, generating dynamic geometries as moving ideas for the dancers to work with in the studio.

The second iteration of the artificially intelligent choreographic agent was given the title Becoming.¹⁸ Becoming was created in close collaboration with social anthropologist James Leach who was the Principal Investigator on the previous Choreographic Objects workshops. Marc Downie and Nick Rothwell were also invited to work on this version, which again was implemented entirely in the FIELD environment. Similarly to the CLA, Becoming was to rely on the dancers’ abilities to work with moving images as inspiration for the creation of movement material in the studio. However, with this second iteration of the choreographic agent, the focus shifted to the creation of something that would have a physical presence in the studio and generate moving images autonomously. Becoming was again built around the manipulation of geometric shapes composed of points, lines, and planes (similar to the CLA). But rather than being programmed by the dancers and viewed on small computer screens at the side of the studio, Becoming had a virtual body the same scale as a human body displayed in portrait mode on a 6 foot 3D screen. Its The movement stimulus for this virtual body came from iconic 1980s science fiction film¹⁹, parsed into its 1240 shots, each of these analysed using computer vision to extract geometry, colour and movement. Downie describes the actions of the virtual body as follows: “The abstract agent then enacts an heuristic search through the space of all the configurations and muscle activations of its own peculiar body to match the movement of each shot. It works out its approximations through a series of iterations, stopping only when satisfied that it has come as close as it can.”²⁰ This description gives a sense of the autonomy of this particular iteration of the choreographic agent concept.

In summary, these two software iterations (both programmed in the same FIELD environment by the same digital artists) made manifest two distinctly different approaches to the idea of the thinking dancer’s body. The first was built on the concept of the thinking body as an instrument of cognition and the value of deliberative thought integrated with intuition as a means of perturbation, of shaking up habits of working. The second relied on an entirely different idea of the thinking body, emphasizing empathic relations between dancers (thinking bodies) in the space and the role that sensing and presence play in achieving social connection, less about breaking habits (deliberatively) and more about an elicitation of a kinesthetic response, through the sensation of movement. While it is interesting to consider how the two software versions uniquely reflect radically different modes of thinking with the body, it is significant and important to emphasize that these different approaches were in part the outcome of engaging researchers with varying conceptual/ intellectual goals. The CLA evolved in collaboration with computer scientist/ psychologist Alan Blackwell and, as mentioned previously, Becoming emerged in a close collaboration with a Social Anthropologist, James Leach.²¹

CAPTURING INTENTION

The research project with the Amsterdam based dance company Emio Greco|PC (Pieter C. Scholten) emerged from a background of questions the artists had regarding the documentation and transmission of their repertoire which at the time comprised six or seven major works. This was a process of searching for alternatives to existing approaches to the documentation and in particular notation of dance, and digital technology was thought to be one of the ways forward. A key driving question was: what kind of notation system²² can “capture inner intention as well as the outer shape of gestures and phrases?”²³ This inspired the title of the research project Capturing Intention which began in 2004 and continued in its first phase through the launch of an interactive installation, book, film and DVD-Rom (deLahunta, editor). The central line of enquiry involved the close analysis and articulation of a physical/ mental training system Greco and Scholten had developed called Double Skin/ Double Mind (DSDM).²⁴ The analysis of the workshop broke its structure down to several themes (e.g. breathing, jumping, expanding) and sub-chapters within each theme, and this was the basis for the development of an interactive installation that would communicate the principles of the DSDM training. Leading this enquiry was Research Coordinator Bertha Bermudez, who had been a former dancer with Emio Greco|PC. Bermudez gathered a group of specialists in notation systems, cinematography, interactive media design, cognitive linguistics and computer-based gesture analysis to work on the project. This included a close collaboration with Frédéric Bevilacqua²⁵, a member of the team at IRCAM²⁶ researching gesture analysis and interactive music systems. For the purpose of the discussion about ‘dance becoming data’, the role Bevilaqua played in the research with Emio Greco|PC on Capturing Intention will be the focus of the following brief exposition.

Since he joined IRCAM in 2003, Bevilacqua’s main focus has been on gesture analysis for the performing arts with the aim of being able to “compute from gesture data ‘high-level parameters’ of movements (…) that could refer for example to ‘movement qualities’ and would be thus more graspable by artists.”²⁷ When he was invited to take part in the creation of the DSDM interactive installation, Bevilacqua brought his work on the so called ‘gesture follower’²⁸ to the project. The ‘gesture follower’ uses a recognition scheme “based on a set of labelled examples that allows the computer to ‘learn’”.²⁹ Following this approach, selected movement phrases from the DSDM workshop were recorded on video and combined with sensor data simultaneously collected from accelerometers attached to the dancer. These were used along with manual annotation to train the gesture follower in a series of experiments that generated “interaction paradigms”, which were incorporated into the DSDM Interactive Installation.

Bevilacqua covers his motivations and methods of research in a chapter for the Capturing Intention book. This chapter gives insights into the continuity of the research supported by IRCAM and related communities into gesture analysis for the fundamental purpose of carrying out “research and development on interactive systems dedicated to music and performance”.³⁰ Sarah Alaoui, another specialist in human computer interaction, joined the research team in 2008 for a new phase of research (Inside Movement Knowledge³¹). Working within a more scientific paradigm than the CLA and Becoming Coding projects, both Bevilacqua and Alaoui’s efforts were motivated by an interest in how “careful case studies will eventually produce general results in the field”.³² Bevilacqua and Alaoui were also founding members of the International Symposium on Movement and Computing. “The symposium references the challenge of representing embodied movement knowledge within computational models, yet it also celebrates the inherent expression available within movement as a language”. The symposium also “seeks to explore an equal and richly nuanced epistemological partnership between movement experience and movement cognition and computational representation.”³³

This interest in seeking a new “epistemological partnership”, a key driver within the International Movement and Computing community (where scientific and engineering goals generate interesting friction with artistic ones) is similar to the goals of Software for Dancers. What has been distinctive about the Capturing Intention and Choreographic Agents research projects is how they reflected a keen motivation on the part of these particular dance artists for a wide research landscape to pursue questions related to movement and meaning, writing and dance, documentation and notation, transmission and dissemination of dance knowledge. Both projects drew attention to the idea that the complex embodied creative process in dance is available to systematic interdisciplinary investigation and that collaboration with coding practitioners can be a part of this research. In this sense, both software and dance artists are contributing to and learning from the same research environment in ways that can be understood to be collaborative, but also distinct. It is this relationship underpinning the latest phase of research, as detailed below and reflected in the conversation with Anton Koch in part two.

EVERYTHING IS DATA

For these two projects in particular (Capturing Intention & Choreographic Agents), the management and storage of dance data for the long term and/ or access to dance-related databases was not such a concern for the artists and scientists involved. The coding work was generally focussed on direct implementation, not on building frameworks; in some cases without much regard for future-proofing, data preservation³⁴ or digital obsolescence.³⁵ Synchronous Objects for One Flat Thing, reproduced, a web-based project led by William Forsythe, departed significantly from this approach.³⁶ The driving research question for Forsythe and his collaborators at The Ohio State University was, as mentioned before: how to help audiences see complex choreographic organisation? Their source material for the project was a performance of the dance, One Flat Thing, reproduced, filmed in high resolution from the front and above. In their essay titled “Dance, Data, Objects,” Norah Zuniga Shaw, William Forsythe and Maria Palazzi (co-creators of Synchronous Objects) explain how the dance was analyzed, decoded and quantified into the data to be used as material to generate the visual interpretations or what they refer to as “Objects” that exist on their website. Relying on manual annotation to apply coding schemes corresponding to three different types of choreographic structure (cues, alignments and thematic material) dancers and animators studied and processed the material into mainly two forms of data, Spatial (location coordinates of the dancers) and Attribute (built from the dancers’ firsthand accounts of the choreographic structure).

In parallel with the development of Synchronous Objects and following the launch of its website in 2009, William Forsythe and others began work on a project with the title Motion Bank.³⁷ The aim of this project was to explore how computer-aided design might aid in the explication (or publication³⁸) of choreographic ideas with a diverse range of dance artists, effectively requiring singular approaches for each. With funding from the German Federal Cultural Foundation and other sources, Motion Bank was able to begin its first phase in 2010.³⁹ Building on the approach of Synchronous Objects to the idea of developing and working with dance data, the Motion Bank team in Frankfurt emphasized digitization as an integral part of Motion Bank from the start and designed recording setups to ensure that everything captured could be available to computation. All recording situations were installed and calibrated to allow for as little ‘noise’ as possible, such that software algorithms might help extract features and recognize relevant patterns in the data. This was combined with the use of a video annotation tool titled Piecemaker, a software project developed by The Forsythe Company member David Kern to support the organization and recall of materials created by Forsythe and his performers in the rehearsal studio (in use from 2007 to 2013), making it possible to tag, annotate and search across the many video recordings generated during creation. In the context of Motion Bank, this software was reprogrammed for use in the development of the on-line digital scores and as a standalone tool for use in the studio. Renamed Piecemaker2 (PM2) it made it possible for annotation sets or markers to be easily related and provide access to multiple versions of the same event (e.g. video, audio, motion capture, scores, etc.). This enabled the building of connections that could generate useful visualizations or other representations both during and post-annotation. As with the Synchronous Objects project, the quantification of the dances of the Motion Bank guest artists into data involved a combination of computational and manual work. This often required many hours spent on computer-based video processing, for example subtracting the background of the image leaving only the silhouettes of the performers, alongside watching the same video for many hours in order to manually annotate and describe time- based events the computer would not be able to recognize on its own.⁴⁰

The work with guest choreographer Deborah Hay provides an important contrast with the descriptions of work with other choreographers (McGregor, Greco, Forsythe). Hay has a unique choreographic approach, which cannot be exposed by recording repeatable movement phrases, studying the way dancers generate such material or by analysing the choreographic structures in a single version of the work. The choreographer/ dancers who work with her know how to interpret the written scores she provides them, each score uniquely combining questions (referred to as ‘tools’ for the dancers) alongside images, reminders and instructions. Only very rarely is there something that might constitute a stage direction or body movement. These are all left up to the choreographer/ dancer to discover as they practice the score for a prescribed number of days, individually, eventually arriving at their own solo adaptation. Body movements and timing are rediscovered each time they perform their adaptation. This means that there is a lot of variability to be found across performances of the same written score, each cannot be seen to be a repeat of the previous – although Hay is clear that “the movement may change, but the choreography itself does not change”.⁴¹

Based on this choreographic approach (the structure of the written score remains the same, whereas performances vary in terms of movement and timing) as many versions of each adaptation were recorded as possible in order to compare them and look for other kinds of patterns in and across the performances. Five digital video cameras were used to record each performance (twenty one in total, seven times for each adaptation performed by three different artists Jeanine Durning, Ros Warby and Juliette Mapp). These recordings were then synchronised and annotated using PM2.⁴² The background of each recording was subtracted leaving only the silhouette of the solo artist. From this data the 3D pathway of each performer could be extracted. Thus, the dance data collected for Deborah Hay’s on-line score includes these 21 digital video recordings, extracted silhouettes, 3d pathways, the score text and the annotations. This material is used to publish research results on the Motion Bank score website for Deborah Hay, alongside extensive interview fragments organised in relation to six conceptual themes framing her choreographic methods. Motion Bank continues to probe and explore this dance dataset, a total of 4TB, for example in the context of Choreographic Coding Labs⁴³ and other research and creative contexts.⁴⁴

Motion Bank had as one of its goals the development of software that might be used by others to create their own on-line scores to add to the Motion Bank collection. This was achieved through the development of two systems. One of these is the reprogrammed version of Piecemaker, PM2, based on the original research of David Kern.⁴⁵ PM2 is currently in use by several organisations including the Pina Bausch Foundation; MA Contemporary Dance Education, Frankfurt University of Music and Performing Arts; Codarts, Rotterdam and the International Choreographic Arts (ICK) Amsterdam. The other software is MoSys, the publishing system developed for the publication of the on-line scores. MoSys consists of an editor to browse collections of recorded, analyzed and annotated material and arrange it into ‘views’ as sets and a frontend to view them. Each set comprises a grid-like system of cells that can interact with each other using a unique messaging system. Since 2013, an additional system, Piecemeta (PMa) has been in development. PMa is a platform for sharing and collaborating on dance-related data, e.g. the Deborah Hay dataset. It enables simplified data storage through a variety of import formats and recording tools and offers the possibility to play back, remix and extend the stored data sets through the services’ programming interface. These data sets can be made public to be further analysed, transformed and enhanced by other researchers and artists.⁴⁶ Currently, Motion Bank is developing a concept for a Dance Data Network, which will feature local, affordable data storage at each network location and sharing methods and systems.⁴⁷

The aim of drafting this account of Dance Becoming Data has been to draw attention to a relatively small number of high profile projects bringing together artists and researchers with different backgrounds, motivations and concerns. What linked all these projects was the sharing of certain questions emerging from and around the practical work of documenting and digitising the processes and products of dance making. These questions take on new meaning and added emphasis in the context of the latest developments in computational and networked media. The future trajectory for this research into dance becoming data (the work Motion Bank intends to continue) requires even more focus on the critical conversations occurring at the intersection of various disciplines, crossing the boundaries between artistic, scholarly and scientific practices.

REFERENCES

Bevilacqua, Frédéric. “Momentary notes on capturing gestures.” Capturing Intention: Documentation, analysis and notation research based on the work of Emio Greco|PC. Editor Scott deLahunta, 26-31, Amsterdam, NL: Emio Greco|PC and Amsterdam School of the Arts, 2007.

Church, Luke, Nick Rothwell, Marc Downie, Scott deLahunta, and Alan Blackwell. “Sketching by Programming in the Choreographic Language Agent.” Proceedings of the Psychology of Programming Interest Group Annual Conference (2012): 163-174.

deLahunta, Scott. “Wayne McGregor’s Choreographic Language Agent.” Transmission in Motion: The Technologizing of Dance. Editor Maaike Bleeker, 108-117. London: Routledge, 2016.

deLahunta, Scott. “Software for Dancers: Coding Forms” Performance Research 7, no. 2 (2002): 96-102.

deLahunta, Scott, and Norah Zuniga Shaw. “Constructing Memories: Creation of the choreographic resource.” Performance Research 11, no. 2 (2006): 53–62.

deLahunta, Scott, and Norah Zuniga Shaw. “Choreographic Resources Agents, Archives, Scores and Installations.” Performance Research 13, no. 1 (2008): 131-133.

deLahunta, Scott, editor. Capturing Intention: Documentation, analysis and notation research based on the work of Emio Greco|PC. Amsterdam, NL: Emio Greco|PC and Amsterdam School of the Arts. 2007, 42-55.

Downie, Marc. “Becoming (2013 Downie).” OpenEndedGroup Website. Accessed 9 October 2017, http://openendedgroup.com/artworks/becoming.html.

Fuller, Matthew, editor. Software Studies: A Lexicon. Cambridge, MA: The MIT Press, 2008.

Leach, James, and Scott deLahunta. “Dance Becoming Knowledge: Designing a Digital “Body”.” Leonardo 50, no. 5 (2017): 461-467.

Manovich, Lev. “New Media from Borges to HTML.” The New Media Reader. edited by Noah Wardrip-Fruin and Nick Montfort, 13-25. Cambridge, MA: The MIT Press, 2003.

Oliveros, Pauline. Software for People: Collected Writings 1963-80. Baltimore, MD: Smith Publications, 1984.

Roy, Sanjoy. “Technological Process.” Dance Theatre Journal 17, no. 4 (2001): 32-35, http://sanjoyroy.net/2002/01/technological-process-software-for-dancers/.

Shaw, Norah Zuniga, William Forsythe and Maria Palazzi. “Dance, Data, Objects.” Synchronous Objects. The Ohio State University, 2009, accessed 9 October 2017, https://synchronousobjects.osu.edu/assets/objects/introduction/danceDataObjectEssays.pdf.

Tribe, Mark, Reena Jana and Uta Grosenick, editors. New Media Art. Koln: Taschen, 2006.

 

Notes