Marking Up Iconography: Scholarly Editions Beyond Text
Paper presented at the TEI Conference and Members' Meeting 2015 in Lyon.
— image sources are identified and listed at the end of the post —
 Imagine you’re in a museum. (I’m sure most of us have been so that shouldn’t be too much of a stretch.) Anyway, you’re in a museum and you see this image hanging on the wall. You step closer and stare at it and notice things. Maybe you see a man, turning away, and a barely clothed woman, lying on a bed, clinging to him. Either way, you identify and interpret something whether you want to or not. And then you turn to the side and search for a label, some sort of information panel next to the painting that will tell you all you need to know about what it is that you’re seeing. (I hope I’m not the only one who does this almost automatically.) Well, imagine that there is no label. Imagine you’re in a museum that has no labels at all.
It has only this – and this – and this –
And you might begin to draw connections between this – and this – and this –
Imagine that all you know is what you see and what your mind supplies of what you have seen and read and learned ever since you opened your eyes to this world.
 Erwin Panofsky, one of the great art historians of the 20th century, stated – and I’m roughly translating – that a strictly formal observation and with it description of an image is a physical impossibility.(1) Every description of content is an interpretation of content because the description already reframes the formal factors of presentation into symbols of something that is presented and thus it extends from a formal sphere into a sphere of meaning.(2)
 Suffice to say, he was the one who came up with the iconographic method, building on some preliminary work by Aby Warburg.
Three Strata of Meaning 1. primary: pre-iconographical description 2. secondary: iconographical analysis 3. tertiary: iconological interpretation
 The method employs three steps that reach from the narrowest understanding to the broadest understanding although it should be noted that they are permeable and that’s what makes it a little complicated:(3)
 The primary layer of meaning attempts a description of the artwork stripped down to what anyone might recognize, regardless of the observer’s education, only based on their immediate experience of existence. So: man – woman – etc.
 The actual iconographic identification, however, requires knowledge of literary sources and acquaintance with certain themes and ideas. This step enables us to identify the man and woman in Fig. 1 as Joseph and Potiphar’s wife from the Bible, in case anyone was wondering.
 And then there’s the third layer of meaning, the iconology. It’s the most divisive because it’s the most abstract and encompasses the whole Weltanschauung. In the example cited above, an iconological question could be why this particular theme was so popular during the Renaissance and the Baroque.
 Right, why am I telling you this in such great detail? Well, there’s no comparable systematic approach to describing the content of images and if any art historian is here and wants to protest, then I’m more than happy to be proven wrong.
 Now when it comes to marking up – and we all know this – it’s about making something that was implicit explicit(4) and this has to be done in a formal way of expression.(5) Transcribing a manuscript has its own set of issues but – and maybe this is this arguable – you are interpreting symbols into text that were some form of text to begin with whereas with images we have to try to interpret symbols into text that are not text, or, let’s say, not expressed with a set of letters, and we have to try and represent meaning in a language that is far more removed from the language it was originally conveyed in – the Bildsprache or visual language – than it is when it comes to text and literature. I was reminded of this during the opening keynote(6) when we heard about literate coding, literate programming, programs as a kind of literature: that suggests a proximity between code and text that doesn’t exist when it comes to images.
 Because of this, with images, instead of transcribing them and then annotating those transcriptions with a semantic layer, we jump right into the semantic layer, possibly only pointing to coordinates of a zone on a surface of a digital facsimile and ascribing that zone content.
 Which brings us to Iconclass. Iconclass is the most widely used classification system/controlled vocabulary for the notation of the content of images. It was developed by Henri van de Waal, starting in the 1950s.(7) He was a professor of art history in Leiden and a friend of Erwin Panofsky (he actually wrote an obituary for him).(8) Iconclass was based on Panofsky’s schema but Henri van de Waal merged the first two layers and left out the last, meaning that he combined the pre-iconographic and the iconographic layer and left out the iconology. Nevertheless, it’s a very elaborate system; the most elaborate when it comes to this.
 And that’s probably the reason why it’s used in almost every digital edition or similar project that deals with iconography in some way. There are other vocabularies such as the Getty Art & Architecture Thesaurus but I didn’t really come across it in the projects that I looked at and anyway, I’m not sure that it matters in the context of this presentation which vocabulary is referenced.
 Let’s divide this into a few sections because if we’re just talking about digital projects with some sort of text-image relation then there’s such an abundance that it would be impossible to survey. What interests me primarily are projects that are decidedly scholarly editions or in that periphery and when looking for projects that mark-up iconography, one has to also consider digital image archives and tools used in scholarly projects for text-image-linking although it must be said that they are usually used for linking a transcription to the corresponding part of the digital facsimile, so the focus is again on text.
 Here I’ve listed a selection of projects that fall into these categories. Obviously it’s only a very, very small selection. As far as the insights are concerned that this might offer, it becomes apparent even by looking at this small selection that the standard practice is to tag images with content keywords that are usually part of a hierarchy of components; either by using the Iconclass system or by developing a system that’s specific to a project as it happened with the William Blake Archive where they considered Iconclass but ultimately decided to design their own controlled vocabulary of characteristics because of the idiosyncracy of Blake’s illustrations.(9)
 I hope I don’t appear dismissive of these efforts by not going into more depth here but I would like to leave it at that for the moment and pose questions that I found neither answered nor even discussed.
 Because what about iconographic variance similar to the textual variance that we record in the critical apparatus? Isn’t that one of the key aspects of a critical edition? To reflect the living and transforming nature of “a” work, to reconstruct its appearance to the reader, or rather observer, and the meaning connected to that, to trace its usage and the stems of transmission. It doesn’t necessarily matter if you’re a disciple of Lachmann or have a New Philology approach because in both cases it’s crucial to recognize the connections, correspondences and deviations across a corpus, that is to say, beyond one witness, at least if you have more than one.
 Let me illustrate what I mean. I present to you the Ascende Calve pope prophecies. It’s a series of prophecies about, well, popes, consisting of 15 short prophetic texts in combination with a symbolic depiction of a pope and a motto each. (What we see here is the first prophecy about pope Nicholaus III. He was from the family of the Orsini and well known for his nepotism and that’s one often-cited explanation for the bears at his side, representing his nephews that he made cardinals.(10) Just to give you an idea about the symbolic nature of the iconographic elements.) The prophecies originated in the middle of the 14th century in the milieu of the Spiritual Franciscans and imitated the schema of an earlier series of pope prophecies, the Genus nequam series (also with 15 prophecies, illustrations and mottos). Starting with the Great Western Schism they gained popularity – probably because they were used as political propaganda – and were widely circulated together as the Vaticinia de summis pontificibus. Over 100 manuscripts seem to have survived.(11)
 Martha H. Fleming edited the Genus nequam series in print form in 1999. I’ll just read out a few quotes from her introduction to the edition.
“Certainly the total effect of picture plus text (and motto) is greater than that of either component alone. Additionally, each provides a means of understanding, even decoding, the other.”
– Martha H. Fleming(12)
“The images cannot be placed in a clutch at the end of the edition; they must be part of the edition. Unfortunately it is not possible to construct an adequate apparatus for an ‘edition’ of the images similar to that for the text, and certainly it is not possible to construct a composite image or an emended image.”
– Martha H. Fleming(13)
 Now, I remembered these prophecies from a history seminar that I took some years ago and it struck me how something like this that could never be properly edited in print appears almost predestined for a digital edition. Also, when someone says that something’s impossible it’s like an invitation to prove otherwise.
 TEI stands for Text Encoding Initiative and it is the de facto standard for digital editions and that makes a lot of sense, simply because of the primacy of text-based hermeneutics, not just in today’s Geisteswissenschaften (I’m sorry, I just prefer that to the term humanities) but also going back to ancient times. If the edition of these prophecies were an actual, real project instead of a mental exercise of mine, people would probably want to use the TEI to encode the text which leaves us with what for the images? A standard that has yet to be invented? Tagging with content keywords? Or would they simply chose not to edit this if it can’t be done satisfactorily and certainly there’s a lot of other material out there in need of editing.
 I think if we truly want to not just replicate the traditional methodology of editing but to harness the potential of the digital, we have to take a step back or step outside the box. If you were at the talk of Elena Pierazzo on Thursday,(14) I’m interested in the conceptual model and not the logical model and not the implementation that can be expressed in a lot of ways but might ultimately be lacking in analytical power and thus scholarship if the focus lies solely in the technical solution and presentation and not in rendering the information that we communicate to the computer meaningful.
 That was my attempt at philosophizing, now back to the situation at hand although I won’t get too deep into any nitty-gritty encoding today.
 But, let’s think out loud and throw some stuff at the wall. We would probably use the <surface> and <zone> elements to define the areas of the digital facsimile that we want to describe. We could put our own <taxonomy> in the TEI header so that it’s project-specific but with references to something like Iconclass wherever they apply. And then in the <zone> elements we could use the @ana attribute to refer to categories in our taxonomy as an indication of the content of the <zone>. There are probably better solutions but the principle is clear. It’s possible to use the <app> element in a <zone> and use <lem> for the lemma and <rdg> for the readings to construct a critical apparatus but this feels like a problematic misappropriation to me and not just because it has such a strictly textual tradition. I think it would be sensible to establish the correspondences and variations without choosing a Leithandschrift or “main” ideal-typical text/image to which everything else is set in a relation of variation. With 100 manuscripts that would even be hardly doable for text so usually you would make a selection of manuscripts, a handful or less, but with the iconographic program the interesting part would be to look at the bigger picture – when and where do certain elements appear or disappear. Also, we cannot transcribe a lemma and readings so we’d have the problem of how to point there. Certainly, it would be nice to be able to view a single iconographic element in a comparative way, for example to see the variations as a sort of tooltip when you’re hovering over such an element. (If we do allow us to think about the presentation. That’s a screenshot of a mock-up I did with the Openseadragon viewer.)
 To link corresponding elements we could maybe use <linkGrp> and within that <link> but to be honest it strikes me as a little arbitrary and a kind of meager representation of knowledge. Although every member of this audience has more TEI expertise than me, I concede that there probably are ways of doing this, of working around these issues but then I get the feeling of putting the cart before the horse and, like, if it doesn’t fit, use a bigger hammer. Obviously, if you’re doing a project, you have to have a result so you have to get creative and hammer away but I want to approach this from the other, the conceptual side without the need for immediate results.
 So let’s take that step back and throw some other stuff at the wall:
 If we take another look at the first prophecy, we notice that in some manuscripts the pope isn’t accompanied by two bears but by a bear and a dragon (as you can see on the right-hand side). Why do we conceive of the dragon as a variation of the bear? It’s rather obvious but it becomes even more obvious if we highlight some iconographic elements –
and superimpose these layers of markings over each other (as I did here with six manuscripts) –
 We draw a connection between dragon and bear because they fill the same position in a spatial schema, a topographical superstructure. The superstructure is then a frame of reference. It exists in different shapes and forms because we could also argue that the layout of the prophecies has a superstructure or that we have, indeed, semantic superstructures – they don’t describe the ideal type but the maximal type. The occurrence of a work in a witness has a unique structure that feeds on the superstructure as well as feeds back into it. Thus, the superstructure encompasses all possible occurrences. It is ideational, global and abstract. If the structure describes a manifest expression of a work, the superstructure describes the work as a container of all instances of this work.
 This rough concept that I plan on developing into a proper model as my PhD already shows some indications how we might be able to capture the iconographic program with its interrelations.
 Instead of constructing a critical apparatus – a practice firmly rooted in print methodology – we’d represent the superstructure and each structure. So, in TEI terms, for the superstructure we’d have an imaginary surface and zones to indicate the existence of components and maybe also their approximate positions and every possible semantic occupancy, making the superstructure equivocal. This could grow as we go along. In parallel, we’d model each structure, that is to say, the surface and zones of each witness from which we’d refer to corresponding elements in the superstructure on the one hand and our taxonomy or Iconclass directly for the specific semantic occurrence on the other hand. The variance of iconographic elements would then be dynamically determined, depending on the point of view, via the cross-references to the superstructure. The absence of elements would be marked by the absence of a reference.
 Representing such an ideational superstructure or superstructures – depending on the granularity – would also have the benefit of maybe allowing us to map iconographic elements from different manuscripts together to create that elusive composite image that Martha H. Fleming was dreaming of.
 My presentation had the subtitle “scholarly editions beyond text” but for my PhD project I’m going to take a theoretical approach to units of meaning that will hopefully be able to address different kinds of transmission variance, maybe even textual although there are a lot of problems connected to that as I’m sure you can imagine.
 Looking at the time, I’ll skip the summary and get straight to my closing remark:
 My general appeal would be to regard intellectual creations – works – as something more than separate texts or illustrations. We should, in my opinion, strive to meaningfully decode and encode as much as we can of that which we perceive and connect in our mind, unknown by anyone else but especially so a machine.
— all following links were accessed on 5 November 2015 —
(1) Cf. Erwin Panofsky, “Zum Problem der Beschreibung und Inhaltsdeutung von Werken der bildenden Kunst,” (1932/1964) in Ikonographie und Ikonologie: Theorien, Entwicklung, Probleme (Bildende Kunst als Zeichensystem; vol. 1), ed. by Ekkehard Kaemmerling, Köln 1979, p. 185-206, here p. 187. Panofsky calls it “ein Ding der Unmöglichkeit”. (↑)
(2) Ibidem. Die Inhaltsbeschreibung müsse “die rein formalen Darstellungsfaktoren bereits zu Symbolen von etwas Dargestelltem umgedeutet haben”, wodurch sie “aus einer rein formalen Sphäre schon in eine Sinnregion hinauf[wachse]”. (↑)
(3) However, most criticisms of Panofsky’s method aim – as far as I can tell – at the non-universal applicability when it comes to certain epochs and art movements. For example, one of the leading critics of the iconographic method, Svetlana Alpers, notes in regard to Dutch painting that iconographers are interested in ascribing something meaning where there is no meaning inherent, emphasizing instead the meaninglessness of the paintings (“völlige Bedeutungslosigkeit der Gemälde”), cf. Sabine Poeschel, “Einleitung: Zur Aktualität der Ikonographie,” in Ikonographie: Neue Wege der Forschung, ed. by Sabine Poeschel, Darmstadt 2010, p. 7-12, here p. 8-9. This, at the very least, doesn’t ring true for the illustrations of the Ascende calve prophecies (see paragraph 16f.) that first of all were very likely a complex way of encoding and transporting meaning in a political context and second of all were numerously interpreted to carry meaning in the following centuries with differing attempts at deciphering them long before the iconographic method came to be. (↑)
(4) Not least of all because computation itself is a “process of making explicit, information that was implicit” (David Kirsh, “When is Information Explicitly Represented?” The Vancouver Studies in Cognitive Science (1990), p. 340-365, here p. 340). (↑)
(5) Of course, John Unsworth already discussed the issues humanists might have with that some time ago, namely: “What is important about the requirement of formal expression is that it puts humanities computing, or rather the computing humanist, in the position of having to do two things that mostly, in the humanities, we don’t do: provide unambiguous expressions of ideas, and provide them according to stated rules.” (John Unsworth, “What is Humanities Computing and What is Not?” Jahrbuch für Computerphilologie 4 (2002), p. 71-84, http://www.computerphilologie.uni-muenchen.de/jg02/unsworth.html). (↑)
(6) Given by Milad Doueihi on Wednesday, 28 October 2015, 4 pm, during the opening session of the TEI Conference, http://tei2015.huma-num.fr/en/keynote-speakers/#doueihi. (↑)
(7) For a short history of Iconclass, see http://www.iconclass.nl/about-iconclass/history-of-iconclass. (↑)
(8) Henri van de Waal, “In Memoriam Erwin Panofsky, March 30 1892 – March 14 1968”, spoken at a gathering on 14 April 1968, printed after van de Waal’s death in Mededelingen der Koninklijke Nederlandse Akademie van Wetenschappen, Afd. Letterkunde 35/6 (1972), p. 227-237, http://www.dwc.knaw.nl/DL/publications/PU00009846.pdf. (↑)
(9) Cf. Morris Eaves, “Picture Problems: X-Editing Images 1992-2010,” Digital Humanities Quarterly 3/3 (2009), paragraph 26, http://www.digitalhumanities.org/dhq/vol/3/3/000052/000052.html#p26. (↑)
(10) Bears translate to orsi in Italian or ursi in Latin. This explanation can be found in a few manuscript catalogues that in turn base their interpretations on “Erläuterungsschriften” from the 16th and 17th century, e.g. in the catalogue H. J. Hermann, Die italienischen Handschriften des Dugento und Trecento: Teil 2 – Oberitalienische Handschriften der zweiten Hälfte des 14. Jahrhunderts (Beschreibendes Verzeichnis der illuminierten Handschriften in Österreich. V; vol.: Die illuminierten Handschriften und Inkunabeln der Nationalbibliothek in Wien), Leipzig 1929, p. 200-205, http://www.manuscripta-mediaevalia.de/hs/katalogseiten/HSK0775_b200_jpg.htm. On p. 200-201 Hermann interprets the bears as the nephews Bertoldo and Latino Malabranca. As a source he mostly cites from an “Erläuterungsschrift” by a Paulus Princeps de la Scala (Cologne, 1570) that he calls Scaliger. He also mentions the “Erläuterungsschrift” by a Gregorius de Laude (Naples, 1660), cf. p. 200 and 205. (↑)
(11) For a list of the manuscripts, see Hélène Millet, Les successeurs du pape aux ours: Histoire d’un livre prophétique médiéval illustré (Vaticinia de summis pontificibus), Turnhout 2004, p. 213-216. (↑)
(12) Martha H. Fleming, The late medieval Pope prophecies: The Genus nequam group (Medieval and Renaissance Texts and Studies; vol. 204), Arizona 1999, https://archive.org/stream/latemedievalpope00flemuoft, p. 10. (↑)
Fig. 1: Rembrandt Harmensz. van Rijn, Jozef en de vrouw van Potifar (1634), https://www.rijksmuseum.nl/en/collection/RP-P-1961-999
Fig. 2: Eugène Delacroix, The Abduction of Rebecca (1846), http://www.metmuseum.org/collection/the-collection-online/search/438814
Fig. 3: Artemisia Gentileschi, Esther before Ahasuerus (probably between 1630 and 1652/53), http://www.metmuseum.org/collection/the-collection-online/search/436453
Fig. 4: Lucas Cranach the Elder, The Judgement of Paris (ca. 1528), http://www.metmuseum.org/collection/the-collection-online/search/436037
Fig. 5: Georg Pencz, from the The Story of Joseph series (1546), http://www.metmuseum.org/collection/the-collection-online/search/399033
Fig. 6: Guido Reni, Joseph and Potiphar’s Wife (ca. 1630), http://www.getty.edu/art/collection/objects/867/guido-reni-joseph-and-potiphar’s-wife-italian-about-1630/
Fig. 7: Erwin Panofsky (1892 – 1968), photo taken from http://www.gerda-henkel-stiftung.de/?page_id=76600
Fig. 8: Henri van de Waal (1910 – 1972), photo taken from https://www.flickr.com/photos/dictionaryofarthistorians/5126139314/
Fig. 9: Small selection of projects, screenshot of my own Powerpoint presentation
(for a project list with links, see ↓below)
Fig. 10: First Ascende calve prophecy in manuscript S
(for a list of the manuscripts, see ↓below)
Fig. 11: First Ascende calve prophecy in manuscript K
Fig. 12: First Ascende calve prophecy in manuscript B
Fig. 13: First Ascende calve prophecy in manuscript S with highlighted components: prophetic text (green), motto (blue), illustration (red), name of pope (yellow)
Fig. 14: First Ascende calve prophecy in manuscript K with highlighted components: prophetic text (green), motto (blue), illustration (red), name of pope (yellow)
Fig. 15: First Ascende calve prophecy in manuscript B with highlighted components: prophetic text (green), motto (blue), illustration (red), name of pope (yellow)
Fig. 16: Screenshot of Martha H. Fleming (Ed.), The late medieval Pope prophecies: The Genus nequam group (Medieval and Renaissance Texts and Studies; vol. 204), Arizona 1999, p. 148-149, https://archive.org/stream/latemedievalpope00flemuoft
Fig. 18: Illustration of the first Ascende calve prophecy in manuscript S
Fig. 19: Illustration of the first Ascende calve prophecy in manuscript Ch
Fig. 20: Illustration of the first Ascende calve prophecy with highlighted iconographic elements in manuscript S
Fig. 21: Illustration of the first Ascende calve prophecy with highlighted iconographic elements in manuscript Ch
Fig. 22: Topographical superstructure of the first Ascende calve prophecy; I created this by layering the element layers of the following manuscripts: Ch, S, K, L, N, W
Digitized Ascende calve Manuscripts:
- S = St. Gallen, Vadianische Sammlung, Ms. 342
- W = Wien, Österreichische Nationalbibliothek, Cod. 13648
- L = London, British Library, Arundel 117
- K = Kremsmünster, Stiftsbibliothek, CC Cim. 6
- N = New York, Pierpont Morgan Library, Ms. 402
- B = Bologna, Biblioteca dell’Archiginnasio, Ms. A.2848
- Ch = Châlons-en-Champagne, Bibliothèque municipal, MS. 68
- Cambridge, Parker Library, Ms. 404, fol. 41r (without illustrations, one of the earliest textual witnesses)
- Vatican, Vatican Apostolic Library, Ross. 374
- Paris, Bibliothèque Nationale, MS lat. 10834
- London, British Library, Harley 1340
- Digital Scholarly Editions
- William Blake Archive
- Library of Congress / Univ. of North Carolina at Chapel Hill
- Emblematica Online
- HAB Wolfenbüttel / Univ. of Illinois
- Emblem Database
- BSB Munich
- French Emblems at Glasgow
- Univ. of Glasgow
- Dutch Love Emblems from the 17th Century
- Univ. of Utrecht
- Text-Image-Edition of the >Welsche Gast< by Thomasin von Zerklaere
- Univ. of Heidelberg
- William Blake Archive
- Digital Image Archives
- Text-Image-Linking Tools
- Text-Image Linking Environment (TILE)
- Maryland Institute for Technology in the Humanities / Indiana Univ.
- TextGrid Text-Image-Link-Editor
- UVic Image Markup Tool
- Univ. of Victoria
- Text-Image Linking Environment (TILE)
Eaves, Morris. “Picture Problems: X-Editing Images 1992-2010,” Digital Humanities Quarterly 3/3 (2009), http://www.digitalhumanities.org/dhq/vol/3/3/000052/000052.html.
Fleming, Martha H. The late medieval Pope prophecies: The Genus nequam group (Medieval and Renaissance Texts and Studies; vol. 204), Arizona 1999, https://archive.org/stream/latemedievalpope00flemuoft.
Hermann, H. J. Die italienischen Handschriften des Dugento und Trecento: Teil 2 – Oberitalienische Handschriften der zweiten Hälfte des 14. Jahrhunderts (Beschreibendes Verzeichnis der illuminierten Handschriften in Österreich. V; vol.: Die illuminierten Handschriften und Inkunabeln der Nationalbibliothek in Wien), Leipzig 1929, p. 200-205, http://www.manuscripta-mediaevalia.de/hs/katalogseiten/HSK0775_b200_jpg.htm.
Kirsh, David. “When is Information Explicitly Represented?” The Vancouver Studies in Cognitive Science (1990), p. 340-365.
Millet, Hélène. Les successeurs du pape aux ours: Histoire d’un livre prophétique médiéval illustré (Vaticinia de summis pontificibus), Turnhout 2004.
Panofsky, Erwin. “Zum Problem der Beschreibung und Inhaltsdeutung von Werken der bildenden Kunst,” (1932/1964) in Ikonographie und Ikonologie: Theorien, Entwicklung, Probleme (Bildende Kunst als Zeichensystem; vol. 1), ed. by Ekkehard Kaemmerling, Köln 1979, p. 185-206.
Poeschel, Sabine. “Einleitung: Zur Aktualität der Ikonographie,” in Ikonographie: Neue Wege der Forschung, ed. by Sabine Poeschel, Darmstadt 2010, p. 7-12.
Unsworth, John. “What is Humanities Computing and What is Not?” Jahrbuch für Computerphilologie 4 (2002), p. 71-84, http://www.computerphilologie.uni-muenchen.de/jg02/unsworth.html.
Waal, Henri van de. “In Memoriam Erwin Panofsky, March 30 1892 – March 14 1968,” Mededelingen der Koninklijke Nederlandse Akademie van Wetenschappen, Afd. Letterkunde 35/6 (1972), p. 227-237, http://www.dwc.knaw.nl/DL/publications/PU00009846.pdf.
See Figures. Featured image corresponds to fig. 7.