Shifting Aesthetics of Image-Sound Relations in the Interaction between Art, Technology, and Perception Dieter Daniels and Sandra Naumann 1. Perception and technology: the two sides of Media Aesthetics An important part of the evolution of human perception concerns the differentia- tion and (re-)synthesis of hearing and seeing over the course of natural evolution and their subsequent cultural conditioning. This aspect is represented by multi- modal integration as an element of the perceptual capacity of the individual. Several anthropological theories dating from the early twentieth century are based on the assumption that the senses had a single common precursor from which the individual sense faculties developed over the course of evolution. Also, it is allegedly possible to demonstrate that certain “primeval synesthesias” existed over the course of human development and history.1 Today, neurologists are exploring the hypothesis that during early neonatal development the sensory regions in the brain advance from synesthetic processing to neurologically diffe- rentiated, single-sense processing.2 One could argue that this development of the differentiation and (re-)synthesis of hearing and seeing is mirrored in the history of culture and technology of image and sound, which will be the main focus of this text.3 Today, the technical and cultural interlacing between visual and acoustic in- formation is so deep and diverse that it is difficult to imagine how separate the cultures and artifacts of image and sound were before the advent of audiovisual media. This separation comprised both the cultural evaluation of music and fine art as well as the physical-material rendering of visual or acoustic artifacts. Music 1 On primeval synesthesia, see Albert Wellek, 1927, “Die Farbe-Ton-Forschung und ihr erster Kongress,” Zeitschrift für Musikwissenschaft 9, pp. 576–584. 2 See Daphne Maurer, 1997, “Neonatal Synaesthesia. Implications for the processing of speech and faces,” in Simon Baron-Cohen and John E. Harrison (eds) Synaesthesia. Classic and Con- temporary Readings, p. 224. 3 For the relation of perception and technology see also: Daniels, Dieter and Naumann, Sandra, 2010, “Introduction” in Dieter Daniels and Sandra Naumann (eds) SEE THIS SOUND Audiovisu- ology. Compendium: An Interdisciplinary Survey of Audiovisual Culture, pp. 5–16. 217 has been considered to be an intellectual art related to mathematics since antiquity. Until the Middle Ages, however, painting and sculpture were understood primar- ily as crafts. If at all, one could only occasionally establish a relationship be- tween the fleeting sounds of music and lasting works of art that one would today refer to as “performative.” It was not until the nineteenth century that media technology enabled the time stream of sounds to become storable and that images started to move, so that today we perceive their synthesis almost as a matter of course. Thus from the outset, the question arises in all media based forms of art as to the relation between image and sound, namely in terms of both technology as well as aesthetics. However, the interfacing of image and sound made possible by media technology not only corresponds with a logic of machines but with the fundamental need for synesthesia embedded in human culture. Its expression ranges from the torch dance accompanying the sound of drums in a prehistoric cave, organ music and the light falling through the window of a Gothic church, to the spectacular courtly celebrations of the Baroque period and visual performances in a techno club, in which ecstatic and spiritual experiences often play a role. Over the past 150 years or so, this deep-seated desire for a synthesis of image and sound has gradually become reality. Then, as now, artists and inventors, tinkerers and entertainers have worked on achieving this. In the process, aesthetic and technical innovations meet each other at the interface between image and sound, where artistic experiment, obsessive bricolage, and genuine technical inventions emerge in an alternation between enthusiasm and desperation, be- tween success and failure. Only very few of the results are finally – often very much later – injected into the mainstream of marketing by the mass media. The ambitious artistic aims as well as the immense technical problems that were wrestled with in the “heroic” period of combining image and sound have today been replaced by digital commodities. There is therefore little awareness of the long series of historic predecessors to what can be called today’s commonplace audiovisuality. They range from designs for a Gesamtkunstwerk towards the end of the nineteenth century and the abstract films of the 1920s to the video-audio synthesizers and psychedelic events of the 1960s. Tracing this development, one can examine the close relationship between the innovation of technical processes and new forms of artistic expression and content. Electronic media – initially analog in the 1960s and then increasingly digital since the 1990s – mark a decisive turning point in this development. Their emer- gence brought about a fundamentally new relation between images and sounds, both in terms of their production as well as their reception: while human percep- tion had previously been their only point of convergence, for the first time in history, sound and light were directly combined and able to be presented as an 218 analog wave or as digital bytes in the same medium, or could be interconverted or generated out of the same code. This liquefaction of audio and video meant that processes that could previously only be produced by means of laborious mechanical techniques working directly with the media carriers now occurred virtually “by themselves” as effects in real time. The term media art, also coined in the 1990s, stands for a spectrum that in- cludes all of the electronic and digital art forms in image, sound, interaction, immersion, and communication. Today, however, the associated theory of a convergence of all art forms in the digital is already in need of revision. Exclu- sively media-technical criteria are no longer sufficient for specifying an indepen- dent art form or genre that can be clearly delineated from classic genres. It is therefore perhaps time to refer back to the basic phenomena of human perception and to formulate the question concerning media aesthetics not solely from the generative, but rather from the human-receptive side. Electronic and digital me- dia nevertheless mark a radical break in the cultural and technical change of perception. This becomes evident with an eye on a development lasting about 150 years using image-sound relations as an example. This also enables the re- formulation of the question regarding media art in order to lead it out of the often self-referential immanence of being a special field and to position it as a hybrid field of culture, media technology and economy.4 2. Aesthetic of separation and re-combination: the mechanical recording of sound and moving images In the second half of the nineteenth century, basic physical and physiological research (Ernst F. F. Chladni, Thomas Young, Hermann von Helmholtz) was applied in the form of media apparatus. The “epistemic thing” built in the scientific laboratory was transformed into media-technical applications suitable for a commercial commodity, which gave rise to an audiovisual mass media culture. An important step in the coupling of images and sounds was the development of recording technology. As early as 1857, Leon Scott had constructed a device – the “phonoautograph,” – that recorded sound as a wavy line on paper but which was not yet capable of playing back. Thomas Alva Edison was the first person to succeed, in 1877, in creating a device that played back: the “phonograph,” in which the waves of a microphone membrane were cut into a cylinder coated with 4 Cf.: Daniels, Dieter, 2011, “Hybrids of Art, Science, Technology, Perception, Entertainment and Business in Sound and Vision” in Dieter Daniels and Sandra Naumann (eds) SEE THIS SOUND. Audiovisuology II – Essays. Histories, and Theories of Audiovisual Media and Art. 219 tin foil and made audible again through a funnel-like horn. Initially, each phono- graph cylinder was a unicum; the mass production of sound storage media did not begin until 1887 with the device developed by Emile Berliner: the “gramo- phone,” whose disks, which were produced using a template, were infinitely reproducible, thus marking the beginning of today’s music industry.5 That same year, Edison outlined a device that – with the use of a phonograph cylinder – could record sounds in synchronization with short sequences of moving images. It had already been possible since the 1830s to utilize the stroboscopic ef- fect to fuse static individual images into fluid movement by means of machines such as the “phenakistiscope” (1832) or the “zoetrope” (1834). These kinds of pre- cinematographic devices emerged at about the same time the first practical photo- graphic process – the daguerreotype – was publicly presented in 1839 and enthu- siastically received worldwide. Between 1870 and 1890, Eadweard Muybridge and Étienne Jules Marey succeeded in recording motion in their phase photographs by combining dozens of cameras. However, the decisive steps toward the suc- cessful recording and playback of moving images were not taken until the 1880s with the invention of roll film, the roll film camera, and celluloid, as well as the construction of early projection devices such as Charles-Émile Reynaud’s “prax- inoscope” (1877). Thomas Alva Edison’s “kinetoscope” (1890-92), which was nothing more than a “phonograph” fitted with chronophotographic images, con- firms the parallels between sound and image technology. This development was completed by the Lumière and the Skladanowsky brothers, who in 1894 arranged the first public showings of films in the “Cinématographe” in Paris and the “Bi- oscope” in Berlin, respectively. In short, since the advent of telephone, phonograph, and film at the end of the nineteenth century, and since radio, sound film, television, audiotape, and video in the twentieth century, audiovisual culture has undergone historically unparal- leled expansion and reformation. All these media have redrawn the borders of the visual and the auditory and reconfigured their relations. In the beginning, in the nineteenth century, media first separated images and sound, then in the twen- tieth century united them again. 5 In 1923, László Moholy-Nagy suggested using the phonographic method not only to reproduce existing music, but to generate synthetic disc sounds by manually or mechanically working the wax template (1923, pp. 103–105). 220 Ill 1: Thomas Alva Edison’s first sketch for the “kinetoscope” (c. 1888) “to develop an instrument, which does for the eye what the Phonograph does for the ear.” (Thomas Edison papers, Rutgers, State University of New Jersey). The parallel emergence of storage media for images and sounds is more a story of separation than of synthesis. However, the ideas for such exclusively acoustic or optical devices are closely interlinked from the very beginning. A good example 221 is the history of transmission media for images and sounds. On the one hand, Alexander Graham Bell’s invention of the telephone in 1876 supplied the direct inspiration for Edison’s “phonograph” and led, on the other hand, to plans for the electric transmission of images by means of the photoelectric sensitivity of selenium, which had been established in 1872. Thus the fundamental concepts for a television medium designed to transmit signals via wire were formulated as early as 1878; however, due to the state of technology at the time, it was not yet possible. The history of ideas for the transmission medium television and the storage medium film operates in the gap that emerged between image and sound attributable to photography, the telephone, and the phonograph: if static images and time based sounds can be stored – and if sounds can be transmitted electri- cally – why shouldn’t it be possible to transmit and store moving pictures? These kinds of conclusions by analogy between acoustic and optical media have cha- racterized the development of radio and television as well as the sound film and the audio-video synthesizer, which will be enlarged on later. The parallel histo- ries of the individual audiovisual media have therefore wrongly reduced image and sound to separate strands; instead, they can only be regarded in terms of their complex interaction, which already contains the potential for their later synthe- sis. 3. Aesthetic of analogy: concepts for linking visual and auditory arts While sound and image still remained separate in terms of their media technology, an increasing theoretical and aesthetic interlinking of music and visual arts takes place in the course of the nineteenth century. Richard Wagner, for example, incorporated the demands made by Romantic authors such as E.T.A. Hoffmann and Friedrich Schlegel for the synthesis of the arts when, in 1849, he conceived his Gesamtkunstwerk in his essay “The Art-Work of the Future”: “The great United Art-work, which must gather up each branch of art to use it as a mean, and in some sense to undo it for the common aim of all, for the unconditioned, absolute portrayal of perfected human nature – this great United Art-work he cannot picture as depending on the arbitrary purpose of some human unit, but can only conceive it as the instinctive and associate product of the Manhood of the Future” (Wagner 1895, p. 88).6 Beginning in 1904, Alexander N. Scriabin created a utopian work that tied in with these ideas: Mysterium, a “polyphonical- 6 Online at http://users.belgacom.net/wagnerlibrary/prose/wagartfut.htm [accessed April 14, 2008]. 222 ly linked texture” consisting of words, sounds, colors, smells, movements, and the senses of taste and touch (Kienscherf 1996, p. 141). Interaction between painting and music, however, flourished in particular around the turn of the twentieth century. Thus the treatment and organization of musical material was exemplary for the painterly pursuit of abstraction, harmo- nious composition, and the depiction of the lapse of time. This becomes apparent in titles of paintings that were inspired by music, such as Wassily Kandinsky’s Improvisations (after 1908), František Kupka’s Nocturne (1910/11), or Paul Klee’s Fugue in Red (1912). Conversely, musicians oriented themselves toward visual manifestations with their concept of the Audition colorée. In his symphon- ic poem Prometheus: The Poem of Fire (1908-10), Scriabin integrated a “light voice” (Luce), which was meant to flood the auditorium in different colors in exact timing with the alternating tones. Josef Matthias Hauer not only made a significant contribution to the liberation of music from tonality and the estab- lishment of twelve-tone music, he was also intensely concerned with the rela- tionship between colors and tones. Based on Goethe’s theory of colors, he corre- lated a light-color wheel with an “acoustic color wheel” and assigned color val- ues to intervals and tonal values. The light organs that were becoming more and more popular with the availa- bility of electric light after the second half of the nineteenth century were also based on the drawing of these kinds of analogies. As with Alexander Wallace Ri- mington’s “color organ” (1893), most of the instruments played the colors by means of piano-like keys. In the early twentieth century these concrete, often arbitrary attributions were finally overcome in favor of a replication of musical structures. Among the first proponents of the open light compositions were Mary Hallock- Greenwalt who, between 1911 and 1931, constructed different versions of her “Sa- rabet,” and Thomas Wilfred, who introduced the “Clavilux” in 1922, calling the new art composed of light, form, color, and motion “Lumia.” In Germany, similar approaches to fusing means of musical and painterly expression were being pur- sued: Ludwig Hirschfeld-Mack and his fellow students at the Bauhaus Kurt Schwerdtfeger and Joseph Hartwig produced “reflecting light compositions” (1922), and Alexander László developed a “Sonchromatoscope” (1925). Parallel to the technical-industrial development of media in the nineteenth century outlined above, devices for producing audiovisual effects from aesthetic- artistic motifs were also being built. Some of the inventors of these hybrids be- tween a work of art and a media device saw them going into mass production. However, all of them remained bound to the person who created them and disap- peared along with their inventors from the world of art and technology, so that today there are only few surviving functioning examples of such devices. This 223 reflects the importance of standardization and compatibility for the proliferation and conservation of audiovisual media, which can be exemplified by 35mm film, the most long-lived global media format.7 4. Aesthetic of synchronization: the coupling of time based images and sound on film As early as the first two decades of the twentieth century, artists who wanted to expand their previous means of expression by creating “visual music” turned toward film, an art form that was still struggling for acknowledgement. The trained painter and musician Walter Ruttmann described his vision of “painting in time” transposed in filmic terms as follows: “An art for the eye that distin- guishes itself from painting in that it is time based (like music) and the artistic emphasis does not (as in an image) consist of the reduction of a (real or formal) process to a single moment, but precisely of the temporal development of formal aspects. Because this art develops temporally, one of its most important elements is the temporal rhythm of visual events. It will therefore produce an entirely new type of artist, up until now only latently existent, positioned somewhere halfway between painting and music.”8 With this in mind, in addition to Walter Ruttmann, in the 1920s and 1930s, Viking Eggeling, Hans Richter, and Oskar Fischinger produced a series of ab- stract films in which they attempted to transfer rules of musical composition to film. The oldest surviving abstract film, Walter Ruttmann’s Lichtspiel Opus I (Light-Play Opus I, 1921) was accompanied by a string quartet composition written specifically for the film by Max Butting. In addition, in order to be able to produce the fluid forms he envisioned for his “painting in time,” Ruttmann constructed a special trick table, with the aid of which he manually spliced to- gether the individual frames over the course of months to produce the film. 7 The development of the first film devices was also motivated by artistic as well as commercial interests. However, in contrast to color-organ and sound-light devices, its history is “aufgeho- ben” (“canceled out” and “preserved”) – in the ambiguous, Hegelian sense – in the medium of film. 8 Walter Ruttmann, untitled, undated [presumably ca. 1919/20], from the Walter Ruttmann estate, cited in Hein and Herzogenrath 1977, p. 64 [under the title “Malerei mit Zeit”]. 224 Ill 2: Invitation from the Ruttmann Film GmbH to the premiere of Lichtspiel Opus I, which took place in 1921 in Frankfurt am Main. The film was subsequently shown, among other venues, at the legendary Der absolute Film event in 1925 in Berlin, which Kurt Weill also attended. © Eva Riehl. Thus for many artists, film technology meant that they did not have to construct their own audiovisual devices; however, it was modified to meet their needs in that there continued to be a close relationship between technological and artistic development. One of the greatest problems was producing the synchronicity of sounds and images, which is why, very early on, the pioneers of film worked on systems to mechanically couple projectors with phonograph cylinders or gramo- phones. Oskar Messter even developed a special sheet-music reel with which the film score could be conducted to accurately coincide with the images being pro- jected, as well as a process for coordinating the piano rolls of player pianos with the films. However, putting silent films to sound was not restricted to back- ground music, but attained a certain proximity to the radio play by means of narrators, people producing sound effects, cinema organs, or recorded sounds. This art form, which emerged in the 1920s for radio, in turn took on numerous filmic techniques such as cross fading, brightening, fading in and out, dimming, 225 or the sound editing of detail and general long shots. Kurt Weill’s theory of a non-narrative, acoustic and abstract “absolute radio art,” which he formulated in 1925 with direct reference to the “absolute film” for the purpose of “thinking out for once the all-too frequently used and abused comparison of film with radio,” exemplifies this interaction between the aesthetics of the silent film and “blind” radio (Weill 1990, p. 192). Walter Ruttmann made the radiopiece Weekend five years later, the first example of this new form of art which Weill imagined. It was shot on Tri-Ergon sound film for the Deutscher Rundfunk and referred to by Ruttmann himself as a “film without images,” using the wonderfully paradoxical and so appropriate term “photographic radio art” (Goergen 1994, p. 25). With the Tri-Ergon optical sound process, which was introduced to the public for the first time in 1922, sound was recorded onto the edge of the strip of film as an optical track and retransformed by means of a photocell. “It is said that an overall me- tamorphosis requires an eleven-fold transformation,” wrote Siegfried Kracauer. “Today, the esoteric of technology already surpasses the Eleusinian mysteries.”9 Because sounds and images could now for the first time be stored on one and the same carrier and hence exactly synchronized, the combination of the two phenomena, which had previously consisted primarily in structural analogies or formal similarities, achieved a completely new quality. As a supplement to the realistic film image, optical sound was now used for the, as it were, naturalistic playback of language, noises, and music, although early sound-film theories advocated the contrapuntal use on an acoustic level. In their “Statement on the Sound-Film,” Sergei Eisenstein, Vsevolod Pudovkin, and Grigori Alexandrov write: “THE FIRST EXPERIMENTAL WORK WITH SOUND MUST BE DI- RECTED ALONG THE LINE OF ITS DISTINCT NON-SYNCHRONIZATION WITH THE VISUAL IMAGES. And only such an attack will give the necessary palpability which will later lead to the creation of an ORCHESTRAL COUN- TERPOINT of visual and aural images.”10 Many of the projects to this effect failed, however, so that the theoretical demands made by Soviet directors could only be experimented with in a few films, such as Dziga Vertov’s Enthusiasm: Symphony of the Donbass (1930). In general, sound-film practice tends more toward so-called Mickey Mousing, in which visual events and movements are translated 1:1 on the sound level. Optical sound made more than just the synchronization of sounds and images possible. Around 1930, Rudolf Pfenninger and Oskar Fischinger studied the graphic formations of the audio track, and started applying them to the film by hand. Their optically generated synthetic sound is regarded as the precursor of 9 Kracauer in 1928 in a review of the first sound films (1992, p. 299). 10 Eisenstein, Pudovkin, and Alexandrov, 1949, pp. 257-259, here p. 258 (emphasis in the original). 226 synthetic electronic sound in the same way the hand-made abstract film is viewed as the forerunner of the mechanically generated computer film (Cf. Wei- bel 1987, p. 103). 5. Aesthetic of transformation: analog electronic modulation of sound and video As early as the 1910s, the “optophone” and the theories on “optophonetics” an- ticipated the electrical generation and combination of sounds and images. Origi- nally conceived as an aid to the blind, which was intended to make optical signals audible by implementing the photosensibility of selenium, the “optophone” be- came the source of more far-reaching visions. Walter Brinkmann, for example, propagated the pure technical transformation of light into sound as a well- founded possibility for the creation of a consistent relation between both pheno- mena: “Practical possibilities of the positive solution of the problem [of color- sound research] would, for example, be given if we succeeded in removing light and sound from their media – ether and air – or furthermore, in identifying elec- tric waves as media for both of them together” (Brinkmann cited in Moholy- Nagy 1927, p. 20ff). Raoul Hausmann, who himself designed an “optophone,” not only assigned the device the potential of “representing the equivalent of any optical appearance in sound,” but of expanding and altering human sense percep- tion (Hausmann (1982 [1922]), p. 51).11 After the 1930s, however, scientists and artists such as Leon Theremin and Mary Ellen Bute began experimenting with the transformation of acoustic sig- nals into optical signals with the aid of an oscilloscope. Along with Hy Hirsh and Norman McLaren, Bute was also one of the first to employ this method of gene- rating images, during which electronic waves are represented graphically, in film during the 1950s. Nam June Paik was one of the artists to continue working with the oscilloscope over the course of the following decade. He was simultaneously interested in television technology and conducted, for example, distortions of the TV image where sound input was transformed in movement and color of elec- tronic patterns on the screen (Participation TV 1963). It was not possible to implement Brinkmann’s ideas until the 1960s – with the emergence of electronic image media and image processing devices. These comprised a fundamental paradigm change with respect to the relations between images and sound as it was now possible not only to alternately transform analog image signals into sound signals and vice versa, but to produce one and the same 11 Cf. Borck, Cornelius in: Dieter Daniels, 2008, Artists as Inventors. 227 electromagnetic wave as a sound as well as an image. Peter Weibel therefore refers to these new kinds of relations as “isomorphism” (Cf. Bódy and Weibel 1987, p. 102). During these years, an international scene of artists-engineers created an “Ei- genwelt der Apparate-Welt” (a self-contained world of the world of devices) that far outreached any of the industrially produced technology at the time (cf. Dunn 1992). Aesthetically motivated inventions again emerged as hybrids between works of art and media devices, which – like the color organs of the nineteenth and the sound-light devices of the early twentieth century – are in jeopardy of disappearing if they are no longer attended to by their creators. Steina and Woody Vasulka’s work is exemplary of the artistic examination of video/audio technology. They had become interested in it at the end of the 1960s, as it was a new medium in which “time/energy [acted] as an organizing principle of sound and image,” and in which sounds and images could be generated, inter- converted, and interact only through voltage and frequency (Vasulka and Vasul- ka, cited in Bódy and Weibel 1987, p. 102). Violin Power (1970-78) shows Stei- na Vasulka playing the violin, whose sounds at the same time distort the image. In Soundsize (1974), the size of the image as well as the frequency of the sounds is controlled by the voltage of a sound synthesizer; and in Noisefields (1974), the energy content of the image modulates the sound. Starting in about the mid- 1970s, the couple also worked with a Rutt/Etra Scan Processor, one of the nu- merous devices for processing video images that had been developed beginning in the late 1960s and which made it possible to manipulate images by means of, among other things, mixing, color transformation, or keying. During this period, the first video synthesizers, modeled on the audio synthe- sizer, were developed, enabling the production of visual material without a camera. Stephen Beck’s Direct Video Synthesizer (1970) produced images by defining their basic parameters – form, contour, color, texture, and movement. Because video synthesizers were capable of producing images in real time, Stephen Beck not only used his device for videotape compositions but for live performances, such as Illuminated Music (1972-73). Due to their ability to generate images and sounds autonomously, synthesizers are regarded as the successors of the analog computer, whose applications were considerably advanced, especially by artists such as John Whitney. Beginning in the late 1930s, the trained musician and his brother James, who had studied painting, produced abstract films that visually transposed the permutative principles of twelve-tone music. They used this me- thod of serial composition in their first sound films, the Five Abstract Film Exer- cises (1943/44). Furthermore, in order to achieve the equal status of sound and image, they used a device constructed by John that produced synthetic optical sound according to a process similar to the one used to produce animated im- 228 ages. From 1960 onward, John Whitney concerned himself with motion graph- ics, and in 1960 produced his first computer film, Catalog, which shows a range of analog effects, using an analog computer he had designed himself. He re- ceived the first artist-in-residence grant awarded in 1966 by IBM, within the framework of which he was able to investigate the aesthetic possibilities of com- puter graphics on new, digital, high-performance computers at about the same time as Stan Vanderbeek and Lilian Schwartz in collaboration with Ken Knowl- ton did at the Bell Laboratories. Still interested in exploring the relationships between musical and visual composition, it was during this same period that Whitney developed his ideas on “digital harmony,” which he published in 1980 in a book of the same name and expanded on in films in the 1980s. He composed works such as Spirals (1988) or the Moondrum series (1989-95) with the use of a special composition program that allowed him to create a “musical design in- tertwined with color design tone-for-tone, played against action-for-action.”12 John Whitney’s development is not only exemplary for the metamorphosis of the relations between sound and image in the twentieth century, in which he was significantly involved in terms of both technology and aesthetics, but also for a new kind of composition in which ideas are simultaneously formulated musically as well as visually and which is occurring again today in the context of digital real time live visuals. 6. Aesthetic of convergence: digital merging of audio and video in real time In the 1990s, electronic and digital media became a part of everyday culture for the reception and production of audiovisual “content.” This did not, however, mark the end of the interaction between artistic and technological development that has been described here; rather, it obtained a previously unimagined dynam- ism and complexity. A striking example is VJing, which emerged in the mid- 1990s in the club context, which not only “abused” commercial technology, but initiated the modification and creation of hardware and software. In analogy to mixing records by DJs, in the early days of VJing, VJs initially assembled image sequences live out of found footage and material they had filmed or animated themselves, with the aid of analog electronic devices such as video mixers and videotapes to create a new, continuous flow of images in cor- respondence to the sound of the DJ. As digitalization progressed, the analog 12 John Whitney, The John Whitney Biography Page, Available online at http://www.siggraph.org/ artdesign/profile/whitney/digiharmon.html [accessed July 25, 2007]. 229 electronic video processing, synthesizing, and editing devices were transferred to software applications. Thus VJ software integrated former analog techniques, such as scratching, multilayering, keying, color correction, or changing speed and the direction of playback. Two of the pioneers in this field were the British artists Coldcut and Hexstatic who, in the mid-1990s, developed what was to be among the first VJ software, VJamm, for live performance of their audiovisual works. Hence when VJs mix and assemble the material, which stems from a wide variety of different sources, its editing and transmutation play a pivotal role, a method that is still being successfully practiced by, among others, Addic- tive TV and TV Sheriff. Besides the application of similar or identical processes to sound and image, which was to some extent already possible with analog electronic media, genera- tive software and the real time transformation of image/sound data count as some of the genuinely new possibilities in the digital age. Because all information is based in code, it can be represented in any form, any process can be initiated or controlled, and the principles it is based on can be arbitrarily varied. Golan Levin speaks of software as an “inexhaustible, infinitely variable, time-based, audiovi- sual ›substance‹.”13 As a matter of fact, numerous artists not only fall back on existing applica- tions, they also program software according to their own ideas and use these not only to produce clips and effects, but to create self-contained visual worlds. In a series of performances, Carsten Nicolai, aka Alva Noton, for example, used software written by Karl Kliem that analyzed the sound signal of his Minimalist sound and translated it into equally reduced abstract image elements in real time, so that these became a “live” graphic representation of the music. Semiconductor, for instance, go a step further than the purely automated transformation of sounds into images. Inspired by the challenge to improvise the entire act of visualization in real time, Ruth Jarman and Joseph Gerhardt devel- oped their own software, Sonic Inc., with which they produced forms, environ- ments, and textures in their live performances, orienting themselves toward digi- tal strategies of representation. Hence the actual artistic input in the creation of an audiovisual product is in the algorithms according to which it is generated and in the interactions with the software that are carried out during the performance. 13 Levin, Golan (2000) “Painterly Interfaces for Audiovisual Performance”. M.S. Thesis. Abstract available at: http://www.flong.com/texts/publications/thesis/. 230 Ill 3: Live performance by Semiconductor using Sonic Inc. at the Mutek Festival 2007 in Montreal. © Semiconductor, photo credit: Caroline Hayeur. In view of the relevance placed on live performance and real time generation, as well as the progressive shift away from the entertainment and club context to- ward an artistic environment, the term “live cinema” is increasingly being used for VJing (cf. Jaeger 2005, and Makela 2006). As far back as 2004, Jan Rohlf summed up these developments as follows: The new possibilities presented by digital technologies … are currently reactivating concepts of audiovisual music in much the same way they were formulated by the Constructivist avant-garde in the first third of the last century. As universal production and performance in- struments, high-performance laptops together with the corresponding software are opening up new possibilities for the real time processing of sounds and images. Thus within the con- text of electronic music, the acoustic and visual live performance becomes the center of in- terest. The coupling of visual and acoustic events as the expression of a direct physical equi- valence relationship as well as a performer’s interaction with generative software applica- tions, in which development principles that have been implemented in the code automatical- ly generate sounds and/or images, is replacing the collaging and manipulation of existing sequences as techniques of DJing and VJing. Accordingly, the protagonists of this art form can no longer be distinctly classified as musicians or visual artists. As “video artists,” they embody – not without recourse to historical models – hybrid identities made up of musician, designer, performer, scientist, and programmer (Rohlf 2004, pp. 121ff). 231 This description is not only reminiscent of Ruttmann’s idea of an intermedia artist, which applies, for example, to Ryoichi Kurokawa; it also refers to tech- nological innovations that unite the production and editing of sounds and images in a single device. This is exemplified by the audio and video mixer DVJ-X1 developed by Pioneer in collaboration with Hexstatic and brought out in 2004, a device that allows the same method of real time manipulation for sounds and images, such as scratching and looping. In addition, this mixer shows how a commercial commodity can emerge from artistic experiments. Despite the unlimited transformation potential, the primacy of the auditory level can be detected in most sound-image relations. Exceptions to this are Island Playback (2005) by Katarina Matiasek and Robin Rimbaud, aka Scanner, in which they transform the coastline of an island in the Mediterranean into a sound curve and play it back, and Robotic Guitar Drone (2004) by Bull and Wounded Horse, in which a guitar is controlled by means of the MIDI messages communi- cated by digital images. In addition, attempts are being made to suspend any kind of hierarchization and to generate sounds and images from one and the same source. An example of one such approach is FarmersManual’s work Graceful Degradation (2001/02), in which real TCP and Ethernet data taken from a local computer network and the Internet are transformed into sounds and video im- ages. Information such as communication time, file volume, or source and desti- nation address are translated into parameters such as rhythm, frequency, color, form, and configuration and presented via speakers and a video projection in a two-dimensional image and sound layer. Besides these efforts to synchronize sound and the image, there are also aspi- rations to extend their relationships to include spatial, physical, and interactive aspects. The trio Sensors_Sonics_Sights (Cécile Babiole, Atau Tanaka, and Lau- rent Dailleau) and the duo 4 Hands (Jean-Marc Duchenne and Bertrand Merlier) are working on the integration of the body by creating audiovisual worlds which are triggered by gestures and movements. The Belgian group lab[au] (Manuel Abendroth, Jerome Decock, Alexandre Plennevaux, and Els Vermang) is inter- ested in architectural concepts. In collaboration with other artists, in their Liquid Space project, they have been experimenting since 2003 with the creation of spatial audiovisuals within a 360-degree multiscreen setting. The Panoscope 360°, a large semisphere that is equipped with a single-channel fisheye projec- tion and a surround-sound system and was developed by Canadian Luc Cour- chesne, affords a special spatial experience: with the aid of a joystick, one navi- gates in real time through a virtual three-dimensional world that has been created by a program, such as, e.g., Where are you? (2005). The installation Messa di Voce (2003) by Golan Levin and Zachary Lieberman also requires recipient participation: noises that are input via a microphone generate their graphic equi- 232 valents on a screen, which in turn respond to the player’s movements. This pur- suit of the creation of interactive, immersive, sensory, audiovisual spaces is without doubt one of the ultimate goals of the convergence of the arts and media. The so called “digital multimedia Gesamtkunstwerk” stands for the merging of the contradictory: the Wagnerian cult of the genius meets the do-it-yourself bri- colage of the amateur. Today, the inescapability of the combination of sounds and images in a mul- timedia lifestyle becomes even more evident in the applications of media play- ers, where “gadgets” for the visualization of music have already been integrated into the standard software. There is no longer a sound not accompanied by an image – if there is no video signal, as a kind of stopgap, visuals are generated automatically. The pitch, duration, timbre, volume, and frequency spectrum are analyzed and translated into a visual representation that is varied through random parameters. The basis for this development was the invention of the mp3 data format in the mid-1990s and the simultaneous emergence of audio players such as Winamp, Audion, and Soundjam. Whereas the first visualization software, Cthugha, which was designed by Kevin “Zaph” Burfitt in 1994, is still referred to as an “oscilloscope on Acid,” other early programs featured an aesthetic proximity to demos. Some of these plug-ins have a greater similarity with live visuals. Lennart Denninger describes his plug-in BeatHarness, which can be used both for generating visuals on the desktop of your PC as well as for live performances, as a “free automated VJ.” The ability of integrating video sequences and live images are further parallels to what used to be VJing. Software development kits (SDKs) enable users to modify or create new plug-ins, making them not only consumers, but producers or so called “prosumers” as well. Smartphone technology opens up completely new perspectives regarding this blurring of the lines between consumption and production, or rather between performer and audience in the context of live performances. One example is the iPhone app “Synk” (2010), conceptualized by Richie Hawtin aka Plastikman, which allows users to log in to the artist’s Wi-Fi network and to influence the organization of certain samples, to observe the real time programming of selected elements live on the mobile phone screen or to follow the show from the stage perspective. This creates not only interaction between artist and listener/viewer but also an “augmented experience” in which immediate and mediated percep- tion overlap. Parallel to this use of advanced technology, recent years have seen a renais- sance of analog technology, such as film projectors, video devices, tube television sets or overhead projectors. In the context of performances, these tools are used in combination with digital devices – or without – to generate live sounds and images. An example for this approach is the performance work “Inside the Black 233 Box” (2010/2011) by Sati and Yroyto. For their shows they use objects made of raw materials like paper, cardboard, motors, and lights which are manipulated in real time, filmed and projected on the screen, while the sound is picked up by contact microphones and further processed with digital means. This coupling of digital and analogue tools offers the artists not only direct and immediate access to the medium as well as the unlimited processing options of the digital, but also make the processes underlying the performances transparent to the public. Summary and outlook Starting from the initially separate recording of sound and images and their later synchronization with the aid of sound-film, progressing to their direct transfor- mation by means of analog electronic media and finally their real time generation in digital code, acoustic and visual phenomena gradually converged and merged to an “audiovisual substance”. This was accompanied by a shift in the relation- ship of seeing and hearing: for most of human history their connection was made exclusively in a subjective, sensory way, and became a technical-physical link about 150 years ago. Since the production of images and sounds is based on algorithms, it can be set in a real time feedback loop with seeing and hearing. This digital “audiovisual substance” is seemingly a direct correspondence to the senses of the performer using it. It can be designed individually and in a variety of ways. Meaning that the creative process entailed in generating audiovisual artifacts shifts from a physical instrument or apparatus to the manipulation or programming of software. But this dematerialization means that it becomes in- creasingly difficult for the recipient (e.g., the audience of such a live perfor- mance) to evaluate the respective artistic contribution and to distinguish it from the mere use of pre-programmed software applications. The current joke is that the guy behind the laptop in the club might as well be reading his e-mail while the public is raving to his sound or his visuals. At the same time, the increasing affordability of hardware and software as well as the availability of source material and programs online, provide the op- portunity to experiment on one’s own, and thus encourage greater understanding of audiovisual artistic works. It is therefore becoming less and less possible to separate the production and reception of audiovisual aesthetics from one another. Also there is a trend away from the standard laptop performance setting by making the digital interfaces physical and the live performance palpable to the public. Another trend is the way back to (neo-)analog technology, which makes the production process transparent for the public and at the same time enhances the choreography of the performer. 234 To conclude, with the apparently almost unlimited potential of digital tech- nologies some of the aesthetic dreams from the nineteenth and early twentieth century seem to be coming true. Many of the ideas at the time could only be realized by means of arduous handwork or ingenious apparatus – others simply failed due to the limitations of technology. Today we have reached the “post- heroic” age of audiovisuality, so to speak. But it may have been precisely the unfulfilled visions, which for decades im- pelled artists and researchers to adapt the insufficient means to their concepts, alter the function of existing technologies and refine them, or invent new devices and methods – and this process continues to this day when audiovisual artists modify or create tools to correspond to their needs, either by making the digital physical or by going back to analog. References Daniels, Dieter and Naumann, Sandra (2010) (eds.) SEE THIS SOUND. Audiovi- suology I – Compendium: An Interdisciplinary Survey of Audiovisual Cultu- re. Cologne: Verlag der Buchhandlung Walther König. Daniels, Dieter and Naumann, Sandra (2011) (eds.) SEE THIS SOUND. Audio- visuology II – Essays: Histories and Theories of Audiovisual Media and Art. Cologne: Verlag der Buchhandlung Walther König. Further References Bódy, Veruschka, and Peter Weibel (1987) Clip, Klapp, Bum: Von der visuellen Musik zum Musikvideo. Cologne: DuMont. Borck, Cornelius (2008) “Blindness, Seeing and Envisioning Prosthesis: The Optophone between Science, Technology, and Art” in Dieter Daniels and Barbara Schmidt (eds.) Artists as Inventors, Inventors as Artists. Hatje Cantz, pp. 108-129. Duplaix, Sophie (2004) (ed.) Sons & lumières: Une histoire du son dans l'art du XXe siècle ; ouvrage publié à l'occasion de l'exposition présentée au Centre Pompidou, Galerie 1 du 22 septembre 2004 au 3 janvier 2005. Paris. Dunn, David (1992) Eigenwelt der Apparate-Welt: Pioneers of Electronic Art. Linz: The Vasulkas. Eisenstein, Sergei, Vsevolod Pudovkin, and Grigori Alexandrov (1949) “A Sta- tement on the Sound-Film” in Sergei Eisenstein, Film Form: Essays in Film Theory. Edited and translated by Jay Leyda. Orlando: Harcourt, Brace. 235 Eskandar, Xarene and Nuengsigkapian, Prisna (2006) (eds.) vE-jA: Art + Tech- nology of Live Audio-Video. San Francisco: H4. Fischinger, Oskar (1932).“Klingende Ornamente” Deutsche Allgemeine Zeitung, July 28, 1932, supplement Kraft und Stoff. English version available at: http://www.oskarfischinger.org/Sounding.htm [accessed April 10, 2008]. Goergen, Jeanpaul (1994) “Walter Ruttmanns Tonmontagen als Ars Acustica”, Massenmedien und Kommunikation, 89. Hausmann, Raoul (1982 [1922]) “Optophonetik”, Texte bis 1933, (2), Sieg Triumph Tabak Bohnen, Michael Erlhoff (ed.) Munich: Edition Text und Kri- tik. Hein, Birgit, and Wulf Herzogenrath (1997) (eds.) Film als Film, 1910 bis heute: Vom Animationsfilm der Zwanziger bis zum Filmenvironment der Siebziger Jahre. Cologne: exh. cat. Kunstverein Köln et al. Hein, Birgit and Wulf Herzogenrath (1979) (eds.) Film as Film: Formal experi- ment in film, 1910-1975. London: Arts Council of Great Britain. Jaeger, Timothy (2006) VJ: Live Cinema Unraveled; Handbook for Live Visual Performance. San Diego: University of San Diego. Kienscherf, Barbara (1996) Das Auge hört mit: Die Idee der Farblichtmusik und ihre Problematik. Frankfurt am Main: P. Lang. Klein, Adrian Bernard (1926) Colour-Music. The Art of Light. London. Kracauer, Siegfried (1992) Der verbotene Blick: Beobachtungen, Analysen, Kritiken. Leipzig: Reclam-Verlag. Makela, Mia (2006) Live Cinema: Language and Elements. Masters thesis. Hel- sinki University of Art and Design. von Maur, Karin (1985) (ed.) Vom Klang der Bilder: Die Musik in der Kunst des 20. Jahrhunderts. Munich and Berlin: Prestel. Moholy-Nagy, László (1923) “Neue Gestaltungen in der Musik: Möglichkeiten des Grammophons,” Der Sturm, 14. Moholy-Nagy, Lászlo (1967[1927]) Malerei Fotografie Film. Berlin: Kupfer- berg. Rohlf, Jan (2004) “Generieren, nicht collagieren: Ton-Bild-Kor- respondenzen im Kontext zeitgenössischer elektronischer Musik,” Cinema 49: Musik, Marburg: Schweizer Jahrbuch für Film. Wagner, Richard (1895) “The Art-Work of the Future,” in id., Richard Wagner’s Prose Works, (1), Translated by William Ashton Ellis. London. Available at http://users.belgacom.net/wagnerlibrary/prose/wagartfut.htm [accessed April 14, 2008]. Weibel, Peter (1987) “Von der visuellen Musik zum Musikvideo” in Veruschka Bódy and Peter Weibel (eds.) Clip, Klapp, Bum: Von der visuellen Musik zum Musikvideo. Cologne: DuMont. 236 Weill, Kurt (1990) “Möglichkeiten absoluter Radiokunst,” in id., Musik und Theater: Gesammelte Schriften. Berlin: Henschelverlag Kunst und Gesell- schaft. Whitney, John. The John Whitney Biography Page, Available at http://www.siggraph.org/artdesign/profile/whitney/digiharmon.html [acces- sed July 25, 2007]. Zielinski, Siegfried (1999) Audiovisions: Cinema and Television as Entr’actes in History. Amsterdam: Amsterdam University Press. 237