From WIMP to ATLAS: Rhetorical Figures of Ubiquitous Computing

Introduction

“In the new millennium, and notably after 2005, as a citizen of more or less massively networked information societies, one has already been interacting enough beyond command-lines, menus, desktops, and GUIs to have realized that another set of models is operative, and that there is at this point an obvious need to pursue analyses and critical evaluations of these models.”
——Ulrik Ekman, “Interaction Designs for Ubicomp Cultures”

What unique analytical perspectives does rhetorical theory bring to bear on our era of mass computation and digital communication? Or, at the very least, which kinds of questions and focal points do rhetoricians tend to emphasize when they examine computational artifacts? The digital turn within rhetorical studies, especially widespread over the past decade, has indeed contributed several distinct lines of inquiry to the broader, interdisciplinary study of new media. Many rhetorical concepts have been with us for over 2,000 years, but this persistence may be attributed largely to their adaptability in the face of technocultural change. Digital rhetoricians variously appropriate classical terms, refashioning them as conceptual lenses for asking how, for example, credibility (i.e., ethos) is performed and assessed on Wikipedia.¹ Rhetorical analyses of software, hardware, and websites push scholars and practitioners to attend to the persuasive dimensions apparent both in user-end interactions and the lines of programming that underlie them. Additionally, owing to its institutional coupling with composition studies, rhetoricians often foreground matters of authorship and deliver insights into how emerging media authoring platforms afford, constrain, or otherwise shape acts of digital writing and online identity formation. One such study, “The Design of Web 2.0” by Kristin Arola, analyzes pivotal differences between MySpace and Facebook’s respective user-generated content features; Arola highlights how each website’s conventions governing the arrangement of users’ profiles also endorse particular ideologies of selfhood.² Scholars across the field continue to mine classical and modern traditions (e.g., the canons, the appeals, the rhetorical triangle) for hermeneutical and heuristic frameworks applicable to computational environments; in turn, others insist that websites, authoring software, and the like challenge rhetoricians to rethink print-bound assumptions.

Collin Brooke’s 2009 book Lingua Fracta: Towards a Rhetoric of New Media performs both of these modes of inquiry, and his initial thesis offers a suggestive point of departure for my essay’s impetus, which is to mobilize rhetorical theory toward critical analyses of ubiquitous computing (ubicomp). Brooke makes a compelling distinction between text and interface, arguing that the Internet occasions a fundamental change in the basic unit of analysis and composition that has hereto anchored rhetorical thinking.³ In noting the significance of Brooke’s claim, however, we must also recognize the need to extend and refine it. ‘Text’ and ‘interface’ are perhaps two of the slipperiest terms across the academy today, in the wake of postmodernism and the digital humanities. Furthermore, computer scientists and interaction designers often develop interfaces according to research agendas that are formulated in direct contrast to previous agendas. For instance, as I will note below, Mark Weiser’s vision for ubicomp (i.e., ad-hoc networks of smart objects/environments) is at once a defiant critique of personal computing (e.g., desktops, laptops, pocket PCs). The basic tenets of the latter serve, in Weiser’s writings, as a list of outcomes to avoid. While text and interface are quite enlightening as broad umbrella units, they are hardly substantial enough to support the kinds of rigorous distinctions needed when thinking deeply about differentiated modalities of computation in times of permanent innovation. The notion that digital media must be critically examined in terms of interface needs to be unpacked through rhetorical explorations into more particular interface-units that are forming heterogeneously across increasingly diverse mediascapes.

This task calls to mind a vital lineage of the rhetorical tradition, one which digital rhetoricians have not yet taken up in earnest: rhetorical figures (e.g., ‘figures of speech,’ ‘stylistic devices,’ ‘schemas and tropes,’). In what follows, my primary objective is to show how an understanding of the figures may inform the cultural criticism of post-desktop interfaces and the unique media ecologies they engender. Furthermore, just as traditional figures like metaphor and metonymy have proven useful to both poets and poeticians, I believe that the new set of rhetorical figures emerging in the wake of ubicomp—for which I propose the acronym “ATLAS” below—offers an inventive heuristic for multimedia producers creating content for such platforms, as well as (though to a lesser extent perhaps) a basic rubric for interface designers making decisions about user-end interactivity.

Rhetorical Figures and Computational Interfaces

The move to orient digital rhetoric around the figures is not without precedent in other humanities fields. Indeed, outside of rhetorical studies, leading historians, linguists, and philosophers have long been appealing to and appropriating specific figures of speech as essential terms for articulating new theories and analyzing various objects of study.⁴ Rhetoricians, meanwhile, are keen to assert the ways in which figures may clue us into recurring and emerging communicative structures, which in turn shape cultural and cognitive practices.⁵ A recent standout in this respect, Richard Lanham’s The Economics of Attention stipulates that rhetorical figures are ‘patterns of speech or writing that provide patterns for thought.’⁶ With this definition in place, Lanham insists that figures are among the most pertinent humanistic resources available to us in an age when data proliferates exponentially and attention is the scarcest asset. The figurative imagination is, in this sense, all about creating and analyzing attention structures, which are crafted to support higher cognitive processes like philosophical reflection, public deliberation, and creativity. Computational interfaces are patterns of digital media that provide for patterns of thought, structure attention, and support a multitude of cognitive processes. Bringing the figurative imagination to bear on contemporary interfaces stands to fashion theorists and designers with critical resources for identifying constitutive patterns and forms underlying various mediascapes—and for analyzing how such forms are (re)structuring attention, culture, communication, as well as the shape of digital networks themselves.

Viewed through a rhetorical lens, categories of hardware and software may be regarded as figures of the digital interface, and thus studied in concert with the classical rhetorical tradition of figures of speech. Sharing the notion of figures, both traditions revolve around cardinal units that serve to orient critical engagement; moreover, figures also act as inventive heuristics to inform communication and design decisions. The figurative imagination cuts across all modes of cultural expression, examining speech, writing, and multimedia practices in order to theorize distinct categories of production and analysis. My concern here is with computational interfaces, the figures apparent in them, and those yet to be articulated.

From the vantage point of this tradition, I move to address a conceptual and rhetorical lacuna apparent in the current state of computational culture. Namely, I am referring to the general effort to reimagine digital media beyond the desktop—both the device and the metaphor—which has gained considerable traction over the past decade, harking back to the groundbreaking work of computer scientist Mark Weiser. In 1991, Weiser ambitiously proposed that his new research agenda at Xerox PARC—what he called ubicomp—marked a third wave in the history of modern computation, succeeding mainframes and personal computers.⁷ Weiser’s first concern was to create hardware beyond the desktop; his team rapidly prototyped entirely new categories of mobile devices—‘tabs, pads, and boards’—that clearly inform today’s smartphones and tablets. On the level of software, however, Weiser’s vision lacked a comparable set of figures. Since the rise of 1980s personal computing, meanwhile, interface designers have been structuring user experiences around the WIMP framework: Windows, Icons, Menus, and Pointers. WIMP design undergirds the virtual reality of the desktop metaphor that Weiser so vehemently critiqued. In the absence of ubicomp-oriented frameworks, many mobile interfaces remediate WIMP design elements. Modes of interacting with desktops have thus become regular ways of behaving anytime-anywhere.

This lacuna should prompt us to seek answers to the following questions: Which emergent elements of interface design today might constitute a ubicomp-specific counterpart to WIMP? That is, what are the rhetorical figures of ubicomp? How might we begin to theorize their heuristic role in the composition of post-desktop mediascapes, as well as the implications this class of interface figures raise for humanistic studies of ubicomp culture?

Attempting to theorize such figures may also help address another ominous concern about ubicomp: its perennial (though not unchallenged) fondness for ‘invisibility’ and ‘calm computing,’ or the notion that user-end interactivity should integrate seamlessly into the fabric of everyday life, perhaps to a point where people hardly notice computers at all.⁸ This scenario is disconcerting to many critics in the humanities because, as Lori Emerson rightly points out, the development of so-called natural, transparent, and user-friendly interfaces generally makes computation harder to manipulate, hack, or critique.⁹ While advertisements sometimes tout post-desktop devices as being ‘interface-free,’ I contend that researchers may counter this ideology by accounting for the nascent set of interface figures at play in ubicomp cultures. After all, devices such as Google Glass and smartwatches, even smart environments, are far from invisible. But they do beckon us to articulate a new conceptual schema, a new critical lens for seeing configurations of digital media that may indeed be hard to piece together from a desktop-oriented perspective.

Latent connections in the writings of technologists, rhetoricians, and designers—which I assemble below—supply a target for figural invention. Since the turn of the century, computer scientists such as Andries van Dam have employed the term ‘post-WIMP’ to call for the development of interfaces and design principles that better reflect ubicomp researchers’ objectives.¹⁰ The WIMP design elements (Windows, Icons, Menus, Pointers) are neither inherent nor universal to computing in general. Granted, WIMP remains a particularly privileged set of figures within the desktop metaphor, fundamental for constructing, navigating, and critiquing an array of use cases synonymous with personal computing. Moreover, critics writing about WIMP in the 1990s provide a helpful precedent for grasping today’s post-WIMP figures. Jay Bolter and Steven Johnson each historicized specific figures from the WIMP framework; in doing so, both critics made convincing cases that these technical units are also vital cultural forms. In Bolter’s analysis, Icons come to be understood as an element of writing foundational to the graphical logic of desktops, and Johnson frames Menus and Windows as ‘basic building-block metaphors’ that punctuate the grammatical breakdown and transformation of the command-line.¹¹ We should also note how fundamental the WIMP framework has been to the evolving scope of digital rhetoric, new media studies, and multimedia composition. The software in which we locate cultural logics, the websites that we claim are challenging disciplinary assumptions, the authoring programs at the foundation of digital pedagogies—so many of these objects of study and tools for production were built by technologists working from the WIMP schema. As such, WIMP GUIs created the enabling conditions for much personal computing since the 1980s, helping software become a mass medium for communication, inquiry, and creativity. Given the centrality of WIMP to new media practices over the past three decades, the bourgeoning conversation surrounding post-WIMP interface design holds an acute potential to incite sweeping changes in digital culture.

The Rhetoric of Post-WIMP Interface Design

Ulrik Ekman, editor of the most substantial arts and humanities volume on ubicomp to date, aptly describes the general status of ubicomp development: ‘…in a number of ways we are already living in a ubicomp epoch and world, [but] it is also the case that ubicomp largely remains to come…a set of recognized paradigms for interaction design and multimodal exchanges has yet to emerge.’¹² This is precisely what I endeavor to introduce in the next section: an initial set of forms—rhetorical figures—to help orient cultural criticism and critical media production in the wake of ubicomp.

To that end, I first need to foreground some additional distinctions between the WIMP schema and the alternative design principles apparent in post-WIMP discourse. Quintilian’s conception of rhetorical figures provides an apt lens for examining this rift; he identified the figurative imagination to be at work whenever language was ‘poetically or rhetorically altered from the simple and obvious method of expression.’¹³ Over the past three decades, the initial innovations of the desktop metaphor have become commonplaces. WIMP interface design is now ‘the simple and obvious method of expression’ that self-proclaimed post-WIMP technologists set out to alter or bypass. Whereas ‘user-friendliness’ remains the highest value of personal computing enterprises—achieved largely via WIMP GUIs—the ubicomp research agenda aims after a further goal: ‘to minimize the mechanics of manipulation and the cognitive distance between intent and the execution of that intent.’¹⁴ The key objective, more simply put, is to impart greater degrees of synchronicity between computation and local action. Writing about the rise of ambient interfaces situated throughout build environments, Malcolm McCullough observes that ‘the role of technology shifts…away from a means to overcome the world toward a means to understand it.’¹⁵ Post-desktop initiatives are largely a matter of designing new ways to circulate data, out of cyberspace and into users’ everyday surroundings. A quasi-phenomenological principle driving this design philosophy is to anchor computing in the lifeworld by embedding it within things themselves, rather than in graphical simulations of things.¹⁶ The file-folder system on desktops may be quite simple to learn and master, but it requires users to manipulate virtual objects in spaces that are disconnected from (and incapable of registering) live, spontaneous extra-computational happenings. With some exceptions, WIMP GUIs generally mandate that users allocate their attention in the manner of a zero-sum game; to engage with the personal computer screen means disengaging from one’s surroundings, with varying degrees and durations.¹⁷

By contrast, ubicomp interactions may occur discreetly at the periphery of otherwise non-discursive action. We already see all sorts of scenarios, absolutely impractical if not impossible with PCs, in which people engage with computing while they remain engaged in another activity. A runner wearing a Nike+ sensor, for example, generates a digital data visualization as she runs; each stride produces an indexical trace recorded and stored on an online database. She runs, and her running is computing. Situated actions like these, occurring in traditionally non-computational environments, become interpreted by ubicomp systems as implicit ‘commands,’ which affect the course of computing and the flow of digital information. The task of rechanneling information from the center of our attention to the periphery, however, need not be done (only) in the service of a so-called ‘calm computing’ experience. Susan Kozel calls for theorists and designer ‘to see in ubiquitous computing the possibility for the unexpected.’¹⁸ Whereas use cases sketched by Weiser emphasize ubicomp’s basic utility around the home and the office, Kozel maintains that the same sensing systems and ambient feedback loops can, like writing or cameras before them, be reconfigured for the purposes of ‘enhancing creativity or facilitating expression’ rather than simply serving as a means to keep tabs on people, objects, and information.¹⁹ This expressive potential will be examined in two urban media installations I discuss in a section below.

Post-WIMP interfaces, moreover, are not (only) designed around a user, a willful individual whose every interaction with computers involves explicit commands. Much post-WIMP discourse holds contempt for anthropocentrism as an unnecessary source of technological limitation. Working in different fields but referencing similar projects, rhetorical theorist Thomas Rickert and interaction designer Jan Rod lend posthumanist perspectives to this erasure of the user, which implicitly complicates the human-centered undertones of Weiser’s manifestos. Principally, Rickert interprets ubicomp’s mandate to embed computation into everyday objects and architectural structures in terms of an ontological flattening variously signaled in philosophical discourse (e.g., Bruno Latour on actant-networks, N. Katherine Hayles on distributed cognition, Jane Bennett’s political ecology of things). As such, rhetorical activity exceeds intentional human subjects more audaciously than ever before. No longer regarded as an inert stage or mere tool to be taken up by human rhetors, ambient environments and smart objects become ‘an active player in their own right,’ registering fluctuations in the surroundings and generating ‘adaptive responses to evolving situations.’²⁰ Intentionality and agency become distributed to the point where subject/object dualities are conceptually untenable, if not imperceptible.²¹

Rod’s work complements Rickert’s Ambient Rhetoric by showing how ubicomp suggests parallel disruptions for design thinking. For Rod, smart objects incite a general erasure of the subject/object dualism on which popular user-centered perspectives on design and HCI have been based.²² User-centered frameworks prioritize humans as thinking subjects in opposition to things as passive objects; consequently, Rod argues that familiar user-centered approaches do not account for the fact that ‘subjectivity is being subsumed into objects and creates hybrid entities that break out of these dualistic categories.’²³ Infused with computation, smart objects exercise a heighten sensitivity to surrounding contexts, and they ‘relate to us through…mediated symbolic interactions, something we think of as being exclusive to humans.’²⁴ As a class, post-WIMP interfaces register a much wider array of variables and gestures than the mechanics of pointing and clicking associated with traditional inputs (i.e., a mouse and keyboard). Consequently, in ubicomp-saturated milieus, practically any actant (e.g., bodies in motion, inorganic matter, atmospheric effects) can impact the computational operations that regulate multimedia production and data flows.²⁵

Furthermore, unlike the desktop metaphor, post-WIMP interfaces favor metonymic design elements that do not block, detract, compete with, or stand-in for one’s live perceptions of the extra-computational lifeworld. It is about computing en plein air, in real-time.²⁶ Bernard Stiegler, the philosopher par excellence of the disorientation precipitated by contemporary technics, claims there is an urgent cultural demand for ‘new techniques to assist with orientation’ in the midst of ubicomp’s proliferation (qua geoinformatics).²⁷ Stiegler asserts, ‘These techniques are needed to help us navigate, no longer through past experience handed down by history, but through the real time of information events that occur on this planet, by the hundreds of millions, with every second that passes.’²⁸ The archive produces the event, but archives come in different forms. What Stiegler clues us into here is that the onset of ubicomp networks might mark the rise of a new kind of archive—one that we engage with amid the live contingencies of everyday life rather than in isolated, codified spaces through differed time (e.g., libraries, museums, theaters, and, arguably, even desktops and laptops).

Articulating the rhetorical figures of ubicomp is a first step toward orienting cultural criticism directed at this new archive and orienting critical analysis of the kinds of events it produces, or the ways in which it (re)casts horizons for social becoming. The first two decades of ubicomp research has provoked many related calls for conceptual invention by leading thinkers in a variety of fields, including media theory, digital aesthetics, interaction design, HCI, and rhetorical theory. Now, as we enter the third decade of ubicomp, this technocultural paradigm has ripened to the point where theorists may propose answers to these calls. Post-WIMP discourse amounts to a placeholder, beckoning the assemblage of a new schema that foregrounds units of analysis specific to ubicomp culture. Just as WIMP taught a critical mass of people to productively engage with computation (without having to face complicated arrays of programming languages and binary data), a new set of cultural forms has begun to emerge, and this schema is, conversely, optimized to reconfigure a critical mass of digital media—to transform the scope of its production and circulation beyond the desktop metaphor. The main concern here lies no longer in creating simulations of territories and behaviors for a virtual map, but in deconstructing the map to more profoundly integrate its components with lived territories and an infinitely wider range of behaviors.

ATLAS: Apps, Tags, Layers, Actuators, Sensors

First, a statement of the obvious: I did not invent these terms. The five figures that comprise my ATLAS acronym (Apps, Tags, Layers, Actuators, Sensors) already circulate as keywords in computer science, engineering, and tech journalism—just as the words ‘network,’ ‘database,’ and ‘link’ were in regular use before media theorists in the 1990s infused them with hermeneutical value. In order to close ‘the distance between the interface design community and that concerned with critical theory,’ Johanna Drucker advises humanities scholars to raid their own intellectual traditions for conceptual resources to describe contemporary interfaces and media effects.²⁹ I mean to imbue these five terms with rhetorical significance, in effort to critically inflect the modalities of communication, expression, and user-end interactivity that they bring to ubicomp cultures. To the extent that the circulation of digital media continues its post-desktop migration; Apps, Layers, and the like may become axiomatic cultural forms by which many of us read, write, design, learn, and think—in concert with our surroundings.

With his XEROX colleague John Seely Brown, Weiser proclaimed that, ‘when computers are all around, so that we want to compute while doing something else…we must radically rethink the goals, context, and technology of the computer.’³⁰ On the heels of the device categories Weiser initiated—tabs, pads, and boards—I propose the ATLAS heuristic as a set of rhetorical figures for analyzing and critically making ubicomp artifacts that are designed to inhabit this radically rethought mediascape, whereby people intentionally and/or habitually engage with computing ‘while doing something else.’ Whereas WIMP prompts users to withdraw from local action in favor of tele-action, the ATLAS figures forge links between local action and networked media. To manifest this design philosophy, each ATLAS element works in coordination with the others, as is the case with WIMP, which is why it makes sense to discuss them acronymically as a set.

At the same time, each ATLAS figure marks a unique mode of configuring bonds between multimedia and local actions, just as the classical figures of speech distinguish among different formalized structures of verbal expression that variously remix conventional usage. We can distinguish between Tags and Layers, for instance, in much the same way we differentiate alliteration from anaphora, or hyperbole from allegory. Related figures are grouped together when they appear to produce similar rhetorical effects; still, rhetoricians have always recognized the unique means by which each figure achieves its ends. ATLAS, the rhetorical figures of ubicomp, each emphasize a distinct way of breaking down the flux of the Web and/or a distinct way of transcoding actant-networks into networked traces. Apps, Layers, and Actuators break down the flux of the Web to suit experience economies specific to particular local actions. Tags and Sensors attach content around—and generate multimedia feeds from—events occurring in a given proximity, effectively assigning a variety of rhetorical and/or aesthetic capacities to the real-time movements of humans, non-humans, and even geographical flux. Each of these ATLAS figures constitutes an important site of rhetoricity beyond the desktop; they act as forms of expression and units of analysis that may orient content creation and new media criticism in ubicomp cultures. The following table indicates characteristics unique to each figure, as a basis for establishing initial definitions. While I lack space here to provide a thorough explication of each figure, below I will give a more detailed analysis of some rhetorical and cultural implications of Layers through case studies of several recent ubicomp projects.

ATLAS Basic Principle A Common Practice An Exemplary Project
Apps Streamline the arrangement, production, and circulation of networked media around a single embodied activity Crowdsourcing for ‘Citizen Science’ or ‘Government 2.0’ iSeahorse (2013)
Tags Transform vast arrays of objects and places into surfaces for inscribing and projecting digital content Place-Based or Object-Oriented Micronarratives Yellow Arrow (2003-07)
Layers Present fragments of pertinent texts and audiovisual media amid one’s current sensory perception Mobile Augmented Reality Browsing Manifest.AR @ ZERO1 (2012)
Actuators Perform computational operations in response to dynamic data gathered in-situ via embedded Sensors Public Interactive Installations David’s Way (2005)
Sensors Scan local surroundings for the presence and movements of specified variables; convert environmental ‘noise’ into meaningful signals to be actuated upon Tracking for Data Visualizations Trash|Track (2009)

In addition to providing a means to conceptualize rhetorical figures intrinsic to ubicomp, the figurative tradition also harbors a central distinction that critically attunes us to the poetics at play in post-desktop mediascapes.

Whereas WIMP designs remain rooted in metaphor, metonymy is the figurative logic that drives the formation and development of ATLAS media projects. In each iteration of the latter, a discrete part becomes detached from the desktop GUI, and that part stands in for the whole of post-desktop computing. Note Hugh Bredin’s distinction between these two important rhetorical figures: ‘Metaphor creates the relation between its objects, while metonymy presupposes that relation.’³¹ As a metonymy-oriented enterprise, ubicomp presupposes our familiarity with the virtual space of the personal computer and its core operations. ATLAS mark new configurations of multimedia that refer to—but at once transform—those of the PC era. Software applications comprised of intricate Menus and multiple Windows become streamlined down to Apps, which typically fragment larger programs, parsing out small doses of functionality that are highly specific to one task. Tags are lifted from their backstage function in computer code (e.g., HTML) and promoted from their supporting role in Web 2.0 (e.g., tag clouds, hashtags), such that Web 3.0 Tags (e.g., QR, RFID) now orient the delivery, circulation, and, sometimes, the very composition of digital media.

Image 1: Evolution of Digital Tags (from Web 1.0 to Web 2.0 to Web 3.0)

In each case, the ATLAS figures are becoming vehicles for inhabiting, interacting with, and making sense of a multitude of built environments. If the desktop metaphor brought the office into the machine, then ubicomp metonymies transcode the entire networked world in the machine’s image.

Consider, for example, the evolution of Layers over the past thirty years, from the PC era to the rise of ubicomp. One way to comprehend the post-desktop life of Layers is to trace the lines of flight by which they transgress the WIMP framework. While Layers have existed since the PC era in many guises, desktop end users interact with them quite regularly in two forms: (1) as dynamic Menus constitutive of and subordinate to WIMP interface logic; and (2) as modular components of audiovisual media files produced and edited within GUI authoring software. In his recent discussion of digital imaging in ‘Inside Photoshop,’ Lev Manovich explains that Layers have long been a commonplace form essential to ‘pre-digital’ media production techniques including multitrack audio recording and cell animation in filmmaking.³² In Photoshop, as in these production techniques that precede it, media are comprised of discrete ‘Layers’ (or ‘tracks’) that can be manipulated independently. In each case, we encounter ‘[t]he same idea of treating an image [or audio composition] as a collection of elements that can be changed independently and re-assembled into new images [or new audio compositions].’³³ As such, WIMP-based iterations of Layers presented transformative implications for digital rhetors and designers: ‘what used to be an indivisible whole becomes a composite of separate parts.’³⁴ Whereas a painter’s every brushstroke alters the appearance of their composition as a whole, the multimedia producer working with layered files can experiment with, oscillate among, and indefinitely bracket an immense variety of audio/visual properties and file combinations. With the ongoing proliferations of professional and ‘freemium’ WYSIWYG authoring software (WIMP design par excellence), the modularity and variability of computation continue to transform the rhetorical and aesthetic processes by which writers and designers invent, arrange, and stylize all manner of texts, images, and audiovisual media. These programs, in short, make for compositional environments that differ profoundly from more archetypical scenarios, wherein one puts pen or brush to a blank white surface—without a database of media effects standing in reserve at the margins, courtesy of WIMP design.

What becomes of Layers, then, as they migrate into the technocultural conditions of ubicomp? First, they are no longer subordinate to WIMPs; rather, they operate in conjunction with Apps, Tags, Actuators, and Sensors to orchestrate a new relationship between multimedia and local, live action. Beyond the desktop, Layers are media files (texts, images, audio, video) that reside, appear, and function amidst the contingencies of everyday life, the spontaneous activities unfolding in one’s proximity. While ubicomp Layers are typically still composed via WIMP-based authoring software like Photoshop, they are designed to circulate on post-WIMP platforms—on head-mounted wearable displays or mobile augmented reality Apps, for example, rather than the browser-based Web. The transformation of Layers beyond the desktop is also conditioned by the evolution of Tags. Whereas HTML Tags often stipulate the arrangement of media files within conventional websites, Web 3.0 Tags (as the next section illustrates) sync digital content with the movement of (non)human bodies, geographical coordinates, and architectural structures. Here, one no longer designs a Menu-Layer for navigating media on the Web; digital content exists as networked Layers outside of conventional browsers. And these Layers may, at least in some cases (e.g., augmented reality), obsolesce websites and become a chief unit of delivery and display in emerging media ecologies.

ATLAS in Emerging Media Ecologies

Having outlined basic principles that distinguish each ATLAS figure, and having noted the metonymic nature of their emergence, I now move to perform closer readings of three ATLAS artifacts and the ubicomp media ecologies they inhabit. In doing so, I will specify some theoretical and pragmatic implications to be gleaned from the post-WIMP circulation of digital media, whereby audio-visual-textual Layers enmesh with local action by virtue of their rhetorical and computational relations with Apps, Tags, Actuators, and Sensors. First, I perform a technical breakdown of ATLAS rhetoricity at play in mobile augmented reality (AR), and then I turn to consider two recent projects by artists and researchers working in the field of urban informatics.

Image 2: Menu-Layers vs. Ubicomp Layers

When examined as a new kind of media ecology, mobile AR provides a clear picture of how and why Layers are becoming metonymically detached from the WIMP interfaces that still govern their production in many other mobile use cases. AR Layers inhabit a screen-camera; resolutely permeable, the visual space for reading and writing always admits the surrounding context as dynamic semiotic occupants within the textual field. In fact, the ‘context’ constitutes the text to such an extent that we may no longer hold to traditional text/context distinctions when composing and analyzing Layers designed to circulate beyond the desktop. This scenario suggests further challenges to basic assumptions concerning arrangement, delivery, and perception when we account for the rhetoricity of the other ATLAS figures. AR Layers—accessed via smartphones, tablets, and headmounted displays—reveal themselves to the human sensorium only after being registered and processed by a device’s Sensors and Actuators. Optical Sensors, for instance, convert images or other perceptual data into electronic signals that, if recognized by the App, become interpreted as a command, thereby prompting a software-based Actuator to execute a programmed action in direct response to the dynamic data gathered by the Sensor. This series of action only occurs, however, after AR media producers have arranged content for delivery via quintessential Web 3.0 Tags.

In the case of mobile AR, media producers integrate digital Layers into environments in one of two ways, depending on which type of Tag they choose. Geo-tags enable us to host networked multimedia at precise longitudes and latitudes; in this case, the Sensors on our devices survey our surroundings for electronic signals via GPS. The other technique for tagging, computer vision, requires AR designers to scan images into browser-based software and to embed their digital content into specific scanned images. In this case, when an App encounters these images amid the built environment (in the form of signs, posters, pages, etc.), the Sensors on the device will read the surroundings in order to detect (and convert) any electronic signals that have been embedded or assigned to an image. In short, Sensors constantly scour the surrounding environment via GPS and computer vision; if the Sensor perceives X phenomenon, then the Actuator will perform any actions associated with X; if the Sensor perceives Y phenomenon, then the Actuator will perform actions associated with Y, and so on. Together, they constitute the machinic reading and writing processes always active at the margins of one’s attention, whenever AR Apps or head-mounted displays are activated. The underlying role played by Sensors and Actuators in such emerging media ecologies significantly extends the if/then logic of computer programming to our engagement with the world at large. To see with computer vision is to perceive the world through the lens of if/then logic. Digital rhetors and designers who create Layers for mobile AR, therefore, must arrange and deliver their compositions on the basis of embedded Tags that address the computer’s eye and mind (Sensors and Actuators) in route to the human sensorium.

ATLAS projects, in this sense, create conditions whereby an increasing array of objects, beings, and surfaces may operate in the manner of a hyperlink. This not only multiplies the stages on which media content and online networks can perform, but also introduces new modalities of interaction among humans, computers, and environments. As illustrated above, the Layers designed for ubicomp platforms are often managed through mobile Actuators and Sensors, which supplant pointing and clicking with an expansive general economy of gestures and activities. Mobile AR aside, other forms of post-WIMP Layers have been cropping up in installation-based projects built in cities around the world over the past decade. David’s Way (2005) in Dallas—described as an ‘urban musical instrument’—is a network of embedded Actuators and Sensors located at a high-traffic intersection along the city’s most popular running/cycling trail. David’s Way is a representative ubicomp installation in that it plays or actuates audiovisual Layers in direct response to the movements of pedestrians who pass within the range of its Sensors, which are in metal poles that line the trail.

Image 3: Artist’s Rendering of David’s Way

The poles themselves thus function in the capacity of Tags, as they comprise the links whereby digital content gets inscribed and projected. In his proposal, artistic director Christopher Janney describes audio Layers generated from the project’s database as containing ‘a mix of melodic tones and environmental sounds, possibly also texts spoken or whispered.’³⁵ The visual Layers, more visible at night, are simply programmed sequences of LED lights that punctuate the audio Layer. In addition to its aesthetic dimension, the project serves a utilitarian interest: the sonic character of the audio Layers in particular alert the trail’s runners and cyclists to the busy street they are about to approach. The audio Layer, which in most cases evokes a jungle vibe, delivers a gentle shock that—through cognitive dissonance (‘Why I am hearing jungle noises in the middle of Dallas?’ )—can snap people out of a ‘runner’s high’ and provoke them, almost instinctually, to inspect the surrounding environment ahead. Most crucially, from the standpoint of the ATLAS framework, ubicomp projects of this sort configure multimedia that sync with one’s live (otherwise non-discursive) actions and strive to engender a certain attunement to the activity of the actant-network in which s/he participates at that moment.

The examples described above have emphasized how ATLAS projects break down the flux of the browser-based Web (and WIMP GUIs) in order to project and display networked multimedia in the periphery of the real-time movements of (non)human bodies, via mobile, wearable, and embedded computers. The final example I consider here illustrates a different rhetorical affordance. Public interactive installations like David’s Way deliver already-composed Layers to a scene according to the rhythms of actions occurring there—doing so in order to engender (new) patterns of social organization through prompted environmental attunements. Other kinds of ubicomp artifacts, however, are designed to provoke critical reflection about deeply entrenched cultural practices on a much grander scale than the here and now; through rigorous tracking and visualization, they trace the actant-networks that underwrite some of the most intricately ecological processes that societies rely upon but also take for granted.

The Trash|Track project, launched by MIT’s SENSEable City Lab in 2009, aims to raise public awareness about the circulation of garbage and its environmental impacts. Essentially, it endeavors to write the actant-networks that condition acts of recycling in America. Researchers attached traceable microchips (i.e., Tags) to ‘different types of trash so that these items c[ould] be followed through the city’s waste management system, revealing the final journey of our everyday objects in a series of real time visualizations.’³⁶ Items of trash are pivotal actants in the environmental issues about which we propose public policies. And yet, we know very little about the afterlife of our things, after we have thrown them away. Where exactly does each piece of trash end up and how exactly does each item reach its final destination?

Translating the movement of trash into a real-time data visualization, Trash|Track traces the actant-networks that actually circulate particular kinds of trash around North America. As such, the ATLAS figures in this case are mobilized to generate/invent networked multimedia from trash-things, which each become cast in the role of a rhetor. Every pixel that is composed on the screen has been generated via the actions of various actants as they intersect with and propel the movement of the trash. These actants include the machines and employees at waste management facilities, the vehicles that transport trash, the highways and airports that structure their journey, etc. Each one of these actants, consciously or not, becomes embellished with a co-authorial function by which they shape the (re)composition of the real-time data visualization. The visualization is the writing of the actant-network. More than a laboratory experiment, Trash|Track visualizations have been set up as installations at museums and libraries to address public audiences.

Image 4: Trash|Track Visualization

Commenting on the results of the project’s data visualizations, Team Leader Dietmar Offenhuber noted, ‘The extent and complexity of the network of waste trajectories was quite unexpected.’³⁷ For instance, the most surprising (and disconcerting) discovery came when his team traced a printer cartridge that traveled 3,823 miles in order to reach the recycling facility to which it had been sent. Traces like these seriously complicate our societal inclination to regard recycling as an inherently eco-friendly practice. Sensing and mapping the contradictory outcomes of existing actant-networks—which are simply illegible without ATLAS-oriented tracking techniques—encourages us to revise those actant-networks and, more broadly, to reflect on the (dis)connections between apparent values and evident practices.

Furthermore, whereas critics tend to see pervasive tracking only in terms of its hegemonic potential for surveillance, Trash|Track demonstrates promising affordances of Sensors and Actuators that encourage heightened collective awareness and robust forms of deliberative rhetoric by ‘allow[ing] individuals to monitor and describe their environment, while also providing an insight into the impact of their own actions.’³⁸ As an innovative case of ATLAS media production, Trash|Track orchestrates a rhetorical-aesthetic-political performance on the basis of the real-time actions of actant-networks. Here, non-human actants are not only the subjects of texts composed to draw attention to the vibrancy of inorganic matter; rather, things are themselves made to double as rhetorical agents. Trash|Track ultimately suggests a modality of digital rhetoric that affords the parliament of things a more prominent place in deliberative democracy. In the case of ATLAS visualizations like Trash|Track, the traces of an actant-network become transcoded to have a political voice in public policy issues.

Conclusion

In describing the ubicomp media ecologies above, I have hoped to clarify how ATLAS may serve as a figurative framework for analyzing the rhetoricity of post-desktop mediascapes. Moreover, I have hoped to conjure an initial sense of how one might start to think, compose, and design via these figures when creating ubicomp artifacts. While canon-based inquiry continues to prove vital as a method for rethinking rhetorical concepts in light of computational media, theoretical initiatives rooted in the figurative imagination appear to forecast a different, complementary agenda. The figures bring rhetorical thinking to bear on digital studies in a manner that stresses the emergence of new cultural forms, as well as the persistence of classical tropes in contemporary design thinking. Taking the shift from WIMP to ATLAS as a pivotal moment in the history of user-end interactivity, we have noted some ways that metaphor and metonymy lend critical insight into the contrasting design philosophies/poetics that inform different interfaces along the spectrum from personal computing to ubicomp. Additionally, in the wake of urgent calls for conceptual categories to orient digital studies, turning to the figurative imagination positions scholars to inhabit a vibrant tradition of concept creation, wherein a long list of thinkers have proposed figures addressed to technocultural conditions associated with a variety of media effects and practices. Amid rapid innovation, HCI researcher Jonathan Grudin insists, ‘Our best chance to anticipate change is to find trajectories that extend from the past to the present.’³⁹ Rhetorical figures offer a pliable source of theoretical scaffolding for assembling such trajectories. By delving further into the multitude of comparative precedents and analogous circumstances associated with the figurative tradition; rhetoricians, designers, and media theorists may be able to better sense the stakes, challenges, and opportunities arising with each new wave of technical invention.

References

Arola, Kristin. “The Design of Web 2.0: The Rise of the Template, The Fall of Design.” Computers and Composition 27 (2010): 4-14.

Barthes, Roland. Camera Lucida: Reflections on Photography. New York: Hill and Wang, 1981.

Baxandall, Michael. Giotto and the Orators: Humanist Observers of Painting in Italy and the Discovery of Pictorial Composition. New York: Oxford University Press USA, 1986.

Bennett, Jill. Empathic Vision: Affect, Trauma, and Contemporary Art. Stanford: Stanford University Press, 2005.

Bolter, Jay David. Writing Space: Computers, Hypertext, and the Remediation of Print. 2nd Ed. New York: Routledge, 2001.

Bredin, Hugh. “Metonymy.” Poetics Today 5.1 (1984): 45-58.

Brooke, Collin. Lingua Fracta: Towards a Rhetoric of New Media. Cresskill, NJ: Hampton, 2009.

Brown, James, Jr. “Essjay’s Ethos: Rethinking Textual Origins and Intellectual Property.” College Composition and Communication 61.1 (2009): 238-58.

Christensen, Nancy. Figuring Style: The Legacy of Renaissance Rhetoric. Columbia, University of South Carolina Press, 2013.

van Dam, Andries. “Post-WIMP User Interfaces.” Communications of the ACM 40.2 (1997): 63-7.

Dourish, Paul. Where the Action is: The Foundations of Embodied Interaction. Cambridge: MIT Press, 2001.

Drucker, Johanna. “Humanities Approaches to Interface Theory.” Culture Machine 12 (2011).

Ekman, Ulrik. “Interaction Designs for Ubicomp Cultures.” Fibreculture 19 (2011).

—–. “Introduction.” Throughout: Art and Culture Emerging with Ubiquitous Computing. Ed. Ulrik Ekman. Cambridge: MIT Press, 2013. pp. 1-60.

Emerson, Lori. Reading Writing Interfaces. Minneapolis: University of Minnesota Press, 2014.

Grudin, Jonathan. “Introduction. A Moving Target: The Evolution of HCI.” Human-Computer Interaction Handbook. 3rd Ed. Julie A. Jacko. New York, Taylor & Francis (2012): xxvii-Ivi.

“Introduction to Trash|Track.” Mit.edu. Sensible Cities Lab, n.d. Web. 10 Aug. 2012.

Janney, Christopher. “David’s Way.” Janneysound.com. 21 Jun. 2005. Web. 16 Feb. 2012.

Johnson, Steven. Interface Culture. New York: Basic Books, 1997.

Kozel, Susan. “Sinews of Ubiquity: A Corporeal Ethics for Ubiquitous Computing.” Throughout: Art and Culture Emerging with Ubiquitous Computing. Ed. Ulrik Ekman. Cambridge: MIT Press, 2013. pp. 337-50.

Lanham, Richard. The Economics of Attention: Style and Substance in the Age of Information. Chicago: University of Chicago Press, 2006.

Latour, Bruno. “Technology is Society Made Durable.” A Sociology of Monsters? Essays on Power, Technology and Domination. Ed. John Law. New York: Routledge, 1991. pp. 103-31.

Manovich, Lev. “Inside Photoshop.” Computational Culture 1 (2011).

McCullough, Malcolm. Ambient Commons: Attention in the Age of Embodied Information. Cambridge: MIT Press, 2013.

“MIT Researchers Map the Flow of Urban Trash.” Mit.edu. Sensible Cities Lab, n.d. Web. 11 Aug. 2012.

National Highway Traffic Safety Administration. “Blueprint for Ending Distracted Driving.” Washington, DC: U.S. Department of Transportation. National Highway Traffic Safety Administration, 2012.

Quintilian. Institutio Oratoria. Retrived January 12, 2014, from Eserver.org website: http://eserver.org/rhetoric/quintilian/

Rickert, Thomas. Ambient Rhetoric. Pittsburg: University of Pittsburg Press, 2013.

Rod, Jan. “Post Human-Centered Design Approaches for Ubiquity.” Escholarship.org. Proceedings of the Digital Arts and Culture Conference. 12 Dec. 2009. Web. 18 2012.

Rogers, Yvonne. “Moving on from Weiser’s Vision of Calm Computing: Engaging Ubicomp Experiences.” Ubicomp 2006: Ubiquitous Computing: 8th International Conference : Proceedings. Eds. Dourish, Paul and Adrian Friday. Berlin: Springer, 2006. pp. 404-21.

Stiegler, Bernard. “Our Ailing Educational Institutions.” Culture Machine 5 (2003).

Tinnell, John. “Computing En Plein Air: Augmented Reality and Impressionist Aesthetics.” Convergence 20.1 (2014): 69-84.

Weiser, Mark. “The Computer for the 21st-Century.” Scientific American 265.3 (1991): 94-104.

Weiser, Mark and John Seely Brown. “The Coming Age of Calm Technology.” Beyond Calculation. Eds. Peter J. Denning and Robert M. Metcalfe. New York: Copernicus, 1997. pp. 75-85.

Notes