Ripping reality: Blind spots and wrecked data in 3D

research project

by Hito Steyerl

George Steiners idiosyncratic study „After Babel“ ends with a fascinating invocation. Quoting obscure parts of scripture Steiner imagines a rebellion of words. Words will „shake of the servitude of meaning. They will become only themselves and as dead stones in our mouths.“[1]

What if images were able to do the same? What if they transformed into the objects they claim to represent? What if the flat plane of representation acquired an extension and even a body? What if images turned into stone, concrete, plastic, into seemingly dead things? Would they thus shake off servitude and meaning? Would they refuse signification or on the contrary give it more weight? Would this be an uprising of images? And what would they be rebelling against?

Recent 3D technologies confront us with these questions. 3D scanning and printing techniques are able to create material replicas of objects and situations: remote sensing casts of reality. Images are thus potentially replaced by objects that stand in for other objects. In these technologies, representation is replaced by replication. We are already used to copy paste and quickly replicate 2D items, such as pictures or words. But how does one copy paste reality? How would one create an indexical material replica of a situation?  How does an image turn into dead stone?

Images of bodies

Lately 3D scanners have been deployed as a new technology of truth. 3D scanning equipment is being used for police work, to investigate homicides, accidents and explosions and also to investigate the whereabouts of missing people.  3D scanners generate pointclouds, measurements in virtual space- which can in turn be printed and rendered as 3D objects.

A lidar scanner captures data through laser, white light or infrared refraction. In the words of one of the main manufacturers, it:  „(measures) a scene with an extraordinary level of speed, accuracy and completeness."[2] and transforms it into a pointcloud in virtual space. The points correspond to locational measurements.

To quote a few samples from the website of Leica Geosystems: „This technology is used globally by law enforcement agencies for crime scene investigation, vulnerability and threat assessments, post-blast investigation, police action inquiries, accident investigations and more.“

The ScanStation is objective and completely measures everything it can “see” for later analysis and diagramming.“

In this terminology, we immediately recognize many tropes that are common in more traditional discussions of documentary evidence. The new technology promises all the things that documentary representation promised ever since: objectivity, full and truthful representation of events, only this time augmented by an additional dimension. A 3D point cloud is no longer a flattened image, missing depth and extension. It is a copy with a volume, dutifully replicating the shape of the initial object.

So what then does the notion of documentary mean, if applied to the 3D replication of objects and situations? What is then the relation of 3D technologies to traditional ideas of documentary evidence? How are notions of documentary truth updated or displaced by 3D technologies? How does the ability to create 3D reproductions affect ideas about documentary truth? What does it mean to replace representation by replication?

Missing bodies

This is a case study in 3d replication I have been realising using a Faro Laser scanner and assorted software in 2011. I want to emphasize right now that i do not consider this to be a documentary case even though it is based on reality. It is a model to test these technologies for potential documentary practice. The case study starts from a specific fantasy. Lets think of kisses. Kisses are travelling events. We can imagine them being passed on like messages or even virusses. They exceed the situation because one kiss between two people can travel on into another situation. It can multiply and be disseminated. It can spread, and create trajectories both in time and in space. Kisses can not only wear off, but also renew themselves by being copied and repeated. They are subject to constant mutations and it is almost impossible to repeat them identically. But a kiss – seen from the point of view of scanning technology also merges various actors, usually two into one surface. Surfaces connect bodies and make them indistinguishable. They connect bodies to grounds and other objects they happen to be in touch with. Surfaces capture bodies as a waveform, entangled with their material environment. A kiss is an energetic relay, which bends surfaces and shapes them into specific forms. We can think of surfaces being sculpted by kisses, of shapes and folds bending with it´s energy. But we can also think of every kiss we see and happen to witness as derivatives, versions or generations of other kisses.

And actually every kiss that happens around us could be a version of one specific kiss[3].

This kiss took place in 1993 in the Bosnian War, when 20 persons were abducted from a train station in eastern Bosnia called Strpci[4]. They had been pulled from a train moving from Belgrade to Bar. A paramilitary unit kidnapped them. None of them was ever seen alive again. There are two unusual elements to this incident. Firstly the fact that only 19 of the 20 are known. That is their names, identities and relatives are known. Except for three whose remains have reemerged from a dam lake rather recently, all of them are missing. But the 20th person is a mystery. Neither his name nor his identity are known. He is reported to have been at the crime site in the testimonies of three witnesses who saw him being led away at the train station. He is also missing from most official accounts or press reports. And nobody asked any questions about who he was either, possibly since he didnt fit into the ethnic map of this conflict. Nobody claimed him as their own.

Bosnian 3D

Now Bosnia-Herzovina in it´s post war state is a multidimensional construction of entities and federations split along  ethnic lines. Bosnia Herzegovina consists of the Federation of Bosnia and Herzegovina and the Republika Srpska. The Federation additionally combines two other non-official political bodies.

The status of the city of Brčko in northeastern Bosnia is a good example of territorial complication. It is a self-governing administrative unit under the sovereignty of Bosnia and Herzegovina – and as such the only territory directly under the mandate of central government. Additionally, it is part of both the Federation and Republika Srpska. It officially belongs to both, but is governed by neither. It´s status also changes according to the perspective from both entities, who have different interpretations of it.  

The spatial composition of this territory was agreed upon in Dayton using early military 3D simulation and one episode became particularly famous, the 3D design of a corridor to Goradze, which became known as Scotch Road or Whiskey corridor. It has been vividly described in a  New York Times article:

„WASHINGTON - The wine was drunk, a lavish lobster dinner eaten, and it was time to resolve one of the most delicate issues in the Bosnian peace talks: the creation of a route for the Bosnian government from Sarajevo through Bosnian-Serb territory to the beleaguered Muslim enclave of Gorazde. President Slobodan Milosevic of Serbia made his way to a high-tech auditorium to play Powerscene, the Pentagon's computer mapping program that reproduces terrain on a vast movie screen. The Serbian leader was adamant that the corridor could be no more than two miles wide. Gen. Wesley K. Clark, the senior American military official at the negotiations, whisked Milosevic off on an imaginary aerial tour of the region to show why such a narrow corridor made no strategic sense. "As you see, God did not put the mountains two miles apart," Clark said.
Milosevic downed a large whisky, considered this geophysical fact, and the deal on a five-mile-wide corridor was consummated. It became known as the "Scotch Road."

„Time and again during the 21-day talks, PowerScene was used to break such stalemates, settling details as small as on which side of a particular road a border would fall. The first virtual reality program ever used in peace negotiations, PowerScene - developed by Cambridge Research Associates of McLean, Virginia - combines pictures from satellites and spy planes with highly accurate terrain-elevation information to generate a level of visual detail that shocked many of the warring leaders. "Stop the flight," Milosevic told Kuchar at one point. "See that bridge there. It is gone. You bombed it away." Indeed, NATO pilots training for last year's air attacks on Bosnian Serb targets in September had used PowerScene to practice their bombing runs.“[5]

3D tools in fact shaped the country by instituting nationalist war goals – namely territorial partition along ethnic lines – by means of the Dayton treaty. And this territorial partition was in marked contrast to the proclamation of a federal Yugoslavia by Avnoj councils in Bosnia in Jajce which took place in a 2D cinema in 1943.[6] The cinema was actually destroyed by fighting between Croat and Bosniak troops in 1992 and I think cinema as such got mortally wounded in the fight and never recovered[7].

But in extension this 3D logic also meant very simply that a black person was not part of the tripartite territory and  somehow got lost within the faultlines of this 3D landscape. Nobody ever asked about that person during the investigation of the case of the abduction in Strpci, and nobody claimed him as part of their own group or community.

But there is one additional and very unusual detail reported by one of the witnesses of the kidnapping. The leader of the paramilitary gang, who led him away tapped him on the shoulder and said: here is my brother. Then he kissed him. We do hardly know anything else about this person, who, only hours later seems to have been shot in an orchard alongside the other 19 after having been mistreated and robbed. His body never appeared nor any additional information.

And obviously we do not know what the kiss looked like either, as it transformed into a surface, a waveform, partly shadowed and spread out in time.

A kiss as a surface

Trying to reconstruct this event using 3D technology seems like an obvious choice, given the forensic usage of this equipment. But once we actually try to scan an actual crime or event going on we start tripping over massive technological limitations.

First reason: This space is a fractional space, to say it with Jalal Toufics words[8], a space that hovers between 2 and 3D. It is for example a space in 2,3 or 2,4 D. To create a full 3D rendition one would need  to scan or capture every point of a surface from every side. One has to basically use at least 3 scanners and then superimpose their results in virtual space. But if you have only one point of view, what you get is at best 2,5 D., a space between a surface and a volume in Toufics terms. 2,5D is created by 3D technology, yet it is imperfect 3D. It hovers between dimensions and connects them. Fractional space is a transitional space that allows people to enter and exit images, to freeze and become images and then leave the state of flatness again and go somewhere else or missing.

This has a striking consequence. What is paradoxically highlighted by 3D scanning technologies is – under these specific circumstances - the missing information of 2D representation: blind spots and blank shadows. And we can only see them in fractional space where the missing itself becomes apparent.

3D technologies not only render those parts that are actually captured as locational measurements by a lidar scanner, but equally the parts that are missing from 2D images: the shadowed, covered or cut parts of the image. The missing data are assigned a volume or a body, just as the existing ones. The shadows and blind spots are not off frame, masked or cut off as they might be in a 2D shot but treated as equal parts of the information.

What emerges is not the image of the body, but the body of the image on which the information itself is but a thin surface or differentiation, shaped by different natural, technological or political forces, or in this case folding around a kiss.

Fractional space

The question of fractional space already appears at the onset of modernity; in one of it´s groundbreaking paintings: The Ambassadors by Hans Holbein the Younger. In 1533, Holbein portrayed two persons, one of them the French ambassador to the English court in a setting replete with scientific tools and cultural references. Both figures are stood next to a shelf displaying books, clocks, sextants and other instruments suggesting learning, culture and potentially also religious strife and disagreement. Both protagonists – and obviously also the person who painted them -  are staged as masters over the props and new representational tools of modernity, of a colonial rule not only over space but also time, in which the former master of vision – or rather his representative on earth Christ – is relegated to a marginal space half covered by a curtain.

But the most striking element of this work is a strange object hovering in the lower quarter of the building – a two dimensional surface positioned at an angle that intersects the plane of the painting. This shape turns out to be a skull if one looks at it from a specific position to the right of the painting.  This technique is called anamorphotic painting and takes into account viewing angles and the corresponding distortion of painted surfaces. Nowadays, it looks like a very basic Photoshop job,  a 2D surface rotated around the Y and X axe and dragged and dropped across another 2D surface.

This painting has been analysed over and over again – especially it´s so-called anamorphotic stain – as an element revealing the gaze and having further repercussions for the construction of subjectivity[9]. It is also being analysed from the perspective of art history as reminder of mortality, as example of showing off both optical knowledge and painterly skills, or also as calculated displacement of the viewer. On the other hand, seen from a contemporary perspective, this painting acquires new and unforeseen meanings. What does the distorted skull mean for and within the painting itself?

Rather than showing something external to the picture it perhaps shows the body of the image itself, as bone. It shows the construction of the image, it´s skeleton if you will, the lines of compression and distortion that make up the construction of paintings in linear perspective. But that skeleton is usually covered by the flesh of painting and kept implicit and invisible. In this painting it is laid bare. It reminds us that the image itself has a body, both expressed by it´s construction and material composition, and that this body may be inanimate, and material.

It is curious that a skull shold be the object that represents the body of the image. Firstly, it is a body part rather than a whole body. It reminds us that since the emergence of mechanical, chemical, or digital reproduction the image is always already fragmented and littered all over the place like a dispersed skeleton. It´s production is scattered, ist circulation even more so.

The skull makes clear, that the body of the image is always incomplete – and it points out this incompleteness by very crudely revealing the flatness and illusionary depth of painted 2D planes. Almost 500 years after it was made it seems to tell us that there are nothing but surfaces and that all these surfaces are missing some or other part of the information.

Folds

Even contemporary 3D scan representation does not primarily produce bodies or objects but folded surfaces. Those surfaces can even be wrapped around to create full volumes but in fractional space they are mainly two dimensional surfaces folded in the third dimension. Surfaces that can be shaped and stretched topologically to take on any conceivable kind of shape. Depth is created by folding a surface onto itself, by folding and stretching it.  And obviously in any real life situation, the surface will bear the imprint of political, material, social, technological and affective forces that are shaping and folding it.

This takes modern ideas about representation as surface to a new level. Georg Simmel introduced the idea that surfaces were not only an integral part of modern urban life but in a sense it´s condensation. This was in opposition to more traditional views of surfaces which connected them to mere appearance, inauthenticity and shallowness.

„The implicit vertical imagery which suggests that if you really want to know what is going on, we must somehow plumb the depths, is a hard one to shift. Depth, in this evocation, acts as a synonym for cultural truth, authenticity, or as the locus for a better interpretation of events, as in many psychoanalytical accounts of the everyday. Equally, the metaphor of society as a smooth, flat surface is one that can give rise to a rather stultifying geography where space amounts to little more than a setting in which events take place, rather than as a source of animation and experience in and of itself.“ [10]

In contrast to this more traditional view of surfaces, which associates surfaces with superficiality Siegfried Kracauer was convinced that all that was worth knowing in an era could be read from inconspicuous surface-level expressions.  For him, the surface was all that needed to be looked at in order to diagnose the present. It presented an unmediated expression of the social unconscious. Kracauer insists on the surface as a primary site of historical and social information:

„The position that an epoch occupies in the historical process can be determined more strikingly from an analysis of its inconspicuous surface-level expressions than from that epoch’s judgments about itself. Since these judgments are expressions of the tendencies of a particular era, they do not offer conclusive testimony about its overall constitution. The surface-level expressions, however, by virtue of their unconscious nature, provide unmediated access to the fundamental substance of the state of things. Conversely, knowledge of this state of things depends on the interpretation of these surface-level expressions. The fundamental substance of an epoch and its unheeded impulses illuminate each other reciprocally.“[11]

We can interpret the folded surfaces of 3D scan representation accordingly as sensors for the impact and tension of an array of diverse and divergent forces. The folds –according to Gilles Deleuze[12] - describe osmotic membranes that mediate between outside and inside, between extrusions and intrusions, enclaves and exclaves of subjectivity and objecthood. They constitute topological distributions that can theoretically morph into different shapes and forms. Modifying the folds of the surface means to interfere with these forces and to recompose them differently. 3D scanning thus does highlight the idea of the surface: by blending in actants, props, actions and forces indiscriminately. The surface is no longer a stage or backdrop on which subjects and objects are positioned. Rather, it folds in both subjects objects and vectors of motion, affect and action, thus removing the artificial epistemological separation between both.

Objectifiction

How do we make these surfaces into material objects? How do images turn into stone, to return to George Steiners initial question? By printing them in 3D. The point cloud captured by a 3D scanner can be modeled to be printed by a 3D printer as an object. These machines, sometimes very simply converted from ink jet printers -  apply fine dusts of almost any possible material including resin, plastic or even metals in fine layers which are glued together. These objects obviously give a material body to images, not only a virtual extension, and in this case the missing data must be stitched over and holes closed in order to make the object withstand gravity. These modelling processes contain an element of interpretation, especially so when large amounts of data are missing.

So essentially the more wrecked the scan data are, and in fractional space they will always substantially be wrecked, the more you have to fictionalise when it comes to stitching up the surface and adapting it to gravity. In fact depending on data, a substantial amount of interpretation goes into the creation of such objects. In the case of this sample it is more than fair to speak of a deliberate objectifiction rather than an objectification or objective rendering of data, since about half of the surfaces are pure estimations, deliberate abstractions and aesthetic interpretations of data. The amount of objectifiction in relation to objective objectification can vary. But even in the most precise replicas and facsimiles is objectifiction present. While the front is based on real measurement, the back is pure fiction. Fiction and indexicality merge in these objects and their relation becomes apparent. If we come back to the fractional space described by Jalal Toufic, this fictional backdoor of the image can provide an exit from the image, but also open the door into the space of the missing.

This also has consequences for the construction of this space. There is no off screen, no apparent exclusion from this field. The resolution will decrease in relation to the range. So nothing is excluded from the frame, except the apparatus, but there is a hierarchy of distance and resolution involved. Things and people gradually fade away, and the amount of fictionality increases. So there is no clear opposition between off screen and on screen, except that at a distance things get more and more fictional. There is no difference between day and night either and the witnesses become paradoxical objects, as they need to keep their eyes closed in order to be scanned and cannot really see the action anyway. And the only documentary element in there is the missing itself.

So, actually, all of the traditional issues of the documentary conundrum, it´s uncertain relation to reality and the anxieties surrounding this now gain a new dimension – a uncertainty which now resonates within a body and a volume.

Unmediated access

But beyond all these traditional conundrums  of documentary there is a way for that situation to manifest itself in a completely different documentary way: that is as it´s truth. I am using this word in a completely blatant and unambiguous way, and there is not a shred of uncertainty around this.

This truth will not happen, when the bones of the missing are found or the black man identified or even justice done or n-d scanners invented to scan all mathematical dimensions of the universe.

You will get a true impression of that kiss when it will come to meet you. It is out there, travelling, replicating itself, bending and sculpting surfaces with it´s energy. It will be very different. It may be a sign of love, violence or just indifference. But it will be this kiss. And it might come right into your face.

At this moment you will be entangled into the surface it created, into a mesh, that twirls and ripples with the forces of affect and political violence.  You will be folded into it´s energy and merge with all other animate and inanimate surfaces within. You will participate in it´s dynamics, which may or may not tear you apart, but in any case dissolve any pretension to confront it as a subject would confront an external object. This surface is not a monad of the past but of the present. It may turn you into stone or a flash of light, or a speck of dust crumbling from a 3D printer or just rush past leaving you indifferent and unaffected.

Images as stones

Lets come back to George Steiners original question about the words or in extension images turning into stones and becoming objects. At this point we see how images becoming objects could start a revolt. There is one slightly boring version of this rebellion, namely that the 3D replicas of objects could start to reverse the relation between original and copy. 3D prints of objects could stop being  likenesses and semblances to become unlikely and unseemly anticipations not of the objects themselves but of their truth.

But lets think of a completely different dimension. The revolt of images will not happen, when anything we see, know and need can be scanned and printed in 3D copyright free.  But imagine the images themselves inside screens suddenly cristallising. Within LCD screens – which still constitute the majority of computer monitors and TV´s at this point of technological development, liquid crystals are carriers of the image information. Now imagine them turning into stone in an instant. Imagine them fossilizing as if in a flash and breaking all screens open from within.

And at this moment the uprising of images indeed happens. All screens turn into dead objects, all cockpit simulators in F16 and helicopters stop working. The screens of aerial surveillance and stock markets burst as the images shake off the servitude of meaning and iphones and target telescopes turn into dead rocks.

At this point is not the images of bodies turning into stone, resin or plastic, but the image itself, it´s carrier acquiring a body, an extension and a volume. It is not what it shows that is extended, but it´s own material substance. They refuse showing anything but themselves as matter plus energy, as waves and particles as surfaces folded into other surfaces and suddenly emerging from them – as Holbeins skull broke through the 2D illusionism of linear perspective.  And this is indeed an uprising of images, against an architecture of representation, that holds them in servitude and subjects them. Against this they start growing their own architecture, uncontrollable and unprecedented.

Warning: for artistic and conceptual reasons this text is an exact  replica of an original, which is copyright free and freely reproducible under a creative commons licence. This replica duplicates only the shape of the original, not the content. Any copyright whatsoever may only be applied to one specific replica, and is authorized for sale or rent only in the country where terms of trade. Unauthorisedexchanging, hining, lending, pubic performance, diffusion. Laws with severe penalties and this video compact disc or any part there of is strictly prohibited.