Lauren Klein‘s paper looked at two 19th century pioneers of data visualization to see what we could learn from them. She asked,

What is the story we tell about the origins of modern data visualization?

What alternative histories emerge? What new forms might we imagine, and what new arguments might we make, if we told that story differently?

Lauren looked at Elizabeth Peabody for an alternative history who is often overlooked because her visualizations are seen as opaque. She compared it to Playfair who is generally considered the first in the canonical history of visualization. Lauren asked why visualizations need to be clear? Why not imagine visualizations that are opaque and learn from them? Her project is a digital recreation project of Peabody’s thinking.

Elizabeth Palmer Peabody (1804-1984) ran a bookstore out of Boston that acted as a salon for the transcendentalists. In 1856 she published a Chronological History of the United States for schools. She traveled around to promote her textbook with a roll of mural charts like domestic rugs (see above). Her charts were based on a Polish process that generated overviews of history.

For modern mavens of visualization like Edward Tufte these charts would not be clear and therefore not effective. By contrast Lauren sees the visualizations of Peabody not as clarifying but as a tool of process or knowedge production. You make knowledge rather than consume it when you make a chart. The clarity to those who didn’t make it is besides the point.

Peabody also sold workbooks for students of school that used the textbook so that they could follow the lessons and rules to generate patterns. Hers is is an argument for making and this making has a historical context. Peabody rejected a single interpretation of history and imagined a visualization system that encourages different interpretations.

This led to one of the points of the talk and that was that the very idea of visualization is itself historically situated and should be examined. And this led to looking again at the canonical works of William Playfair.

She then showed us some of Playfair’s visualizations (from The Commercial and Political Atlas) that are much more readable and for that reason he is often seen as a pioneer in data visualization. Playfair is widely considered one of the first to abstract phenomena to data for visualization. Lauren pointed out how Playfair was not sure how his visualizations would be interpreted, but he did want them to make an impression that was “simple and complete.” He was good at this.

She then showed Lyra: A Visualization Design Environment, an open source alternative to Tableau. There are a lot of Playfair emulators who use things from Lyra to everyday tools like Excel to recreate Playfair’s charts. There are plenty of tools now out there with which one can create visualizations including try to emulate Playfair.

What is interesting is that the designers of the professional tools made decisions about what visualizations should or could do. Thus we see a lot of line and bar charts and little resembling Peabody’s. The widely held belief is that visualization should condense and clarify.

Recreating Peabody

Lauren then shifted to describing an ongoing project to recreate some of Playfair and Peabody’s charts with different tools. They found the existing tools, like D3, hard to use. The tools all assume you start with data. This made her think of the status of data and its relationship to visualization.

She pointed out that when you use a tool for visualization you don’t worry about the shape of the curve, you let the tool do that. Playfair did, however worry about it. He had to engrave the curves by hand and he played with the lines trying to make them attractive to the eye.

Watt, for whom Playfair worked, suggested to him that he put the tables next to the charts. He did this in the first two editions of his book (and then removed the tables for the third.) Even with those charts some of the graphs are hard to recreate. To make one of Playfair charts they had to use data from two different places in Playfair. Again, almost all tools, like D3, now depend on data. The dependence on data is structurally woven in, unlike more artistic tools like Illustrator.

She then showed an engraving error detail and discussed how it could have come about due to Playfair being tired when making the copper plate. In the digital artefact we don’t see such errors – we only see the finished project. The digital masks the labour. Only in Github are changes/labour saved and viewable.

Then she showed the prototypes her team has made including a “build” mode where you can construct a Peabody chart. They are now planning a large scale project using LEDs on fabric to create a physical prototype as that would be closer to the fabric charts Peabody made.

This returned her to labour, especially the labour of women. Peabody made copies of the charts for classes that adopted her book. Alas, none of these survived, but we do have evidence of the drudgery in her letters.

To Lauren the Peabody charts remind her of quilts and she showed examples of quilts from Louisiana that were a form of community knowledge constructing genealogies. Such quilts have only recently been recognized as knowledge comparable to the logocentric knowledge we normally respect.

Lauren closed with a speculative experiment. How would we think differently if Peabody’s charts had been adopted as the standard to be emulated rather than the line charts of Playfair? How might we know differently?

Her team’s recreations of both the Playfair and Peabody charts are just such a sort of speculation – understanding though making.

You can watch the video with slides here.

Stan Ruecker gave the 3DH talk on the 23rd of June with the enigmatic title The Digital Is Gravy. He explained the title in reference to gravy being the what gives flavour to the steak. In his case, he wanted to show us how physical prototyping can give substance (steak) to the digital.

Stan started with an example of a physical prototype that materializes bubblelines that was developed by Milena Radzikowska who showed it at Congress 2016 in Calgary. (See Materializing the Visual.) He suggested that materialization of a visualization slows down analysis and leads to other lines of thought.

At the IIT Institute for Design Stan is weaving physical prototyping into digital design projects. His main research goal is to find ways to encourage people to have multiple opinions. He want to build information systems that encourage the discovery of different perspectives and the presentation of multiple opinions on a phenomenon. The idea is to encourage reflective interpretation rather than dogmatism.

How prototypes build understanding

He listed some ways that prototyping can build understanding:

• Build something to collect information
• The prototype is itself a kind of evidence
• Learning through making. You don’t even need to finish a prototype. “Fail early and fail often.”
• Prototype is also a representation of the topic area

Why physicality is important

After returning to the materialized bubblelines he talked

• Materialized prototypes take time differently which can lead
• It can produce results that can be used for comparison (with other results)
• It can engage physical intelligence – embodied experience can leverage different ways of knowing
• It involves collaboration (over time)  that involves community knowing
• It encourages multiple perspectives from different people and different points of view

My experience with the pleasures of physical prototyping in a group reinforces the way the making of the

Timelines

He then talked about a project around timelines that has built on work Johanna Drucker did. He had gone through multiple prototypes from digital to physical as he tried to find ways to represent different types of time. He tried creating a 3D model in Unity but that didn’t really work for them. He now has a number of student designers who are physically modelling what the timeline could be like if you manipulated it physically and then that was uploaded to the digital representation (the gravy.)

Physical Qualitative Analysis

He then talked about how a multinational team is designing physical analytical tools. The idea is that people can analyze a text and model an understanding of it in a physical 3D space. It is grounded theory – you build up an emergent understanding. They tried creating a floating model like a Calder sculpture. They tried modelling technical support conversations. They used a wired up coat rack – hacking what they had at hand.

My first reaction is that doing this physically would be so slow. But that is the point. Slow down and think by building. They tried a digital table and that was no fun so they started making all sorts of physical

I’m guessing it would be interesting to look at Ann Blair’s Too Much To Know where she talks about the history of note taking and physical ways of organizing information like excerpt cabinets.

Stan then talked about a successful line of prototypes that had transparent panels that could be organized, joined, and on which ideas could be put with post-it notes. Doing this in a team encourages users to different views on a subject as the panels have two sides and can be jointed to have even more.

Finally, they are now trying to bring these back to the digital so that once you have an arrangement of panels with notes you can digitize it and bring it into the computer. This also suggests the possibility of automatically generating the model on the computer from the text.

He commented on how he has no industry industry interested in the analysis of conversations.

And that was the end.

Even when you deal with relative frequency you can get what look like wild variations. These often are not indicative of something in the time, but indicate a small sample size. The diachronic datasets often have far fewer books per year in the early centuries than later so the results of searches can vary. One book with the search pattern can appear like a dramatic bump in early years.

There are also problems with claims made about data. There is a “real world” from which we then capture (capta) information. That information is not given but captured. It is then manipulated to produce more and more surrogates. The surrogates are then used to produce visualizations where you pick what you want users to see and how. All of these are acts of interpretation.

What we have are problems with tools and problems of data. We can see this in how women are represented datamining, which is what this talk is about. She organized her talk around the steps that get us from the world to a visualization. Her central example was Matt Jocker’s work in Macroanalysis on gender that seemed to suggest we can use text mining to differentiate between women and men writing.

World 2 Capta

She started with the problem of what data we have of women’s writing. The data is not given by the “real” world. It is gathered and people gathering often have biased accounting systems. Decisions made about what is literature or what is high literature affect the mining downstream.

We need to be able to ask “How is data structured and does it have problems?”

Women are absent in the archive – they are getting erased. Laura thinks these erasures sustain the illusion.

Capta 2 Data or Data Munging

She then talked about the munging of data – how it is cleaned up and enriched. She talked about how Matt Jockers has presented differences in data munging.

The Algorithms

Then she talked about the algorithms, many of which have problems. Moritz Hardt arranged a conference on How Big Data is Unfair. Hardt showed how the algorithms can be biased.

Sara Hajian is another person who has talked about algorithm unfairness. She has shown how it shows prestigious job ads to men. Preferential culture is unfair. Why Big Data Needs Thick Data is a paper that argues that we need both.

Laura insisted that the solution is not to give up on big data, but that we need to keep working on big data to make it fair and not give it up.

Data Manipulation to Visualization

Laura then shifted to problems with how data is manipulated and visualized to make arguments. She mentioned Jan Rybicki’s article Vive la différence that shows how ideas about writing like a man and like a woman don’t work. Even Matt Jockers concludes that gender doesn’t explain much. Coherence, author, genre, decade do a much better job. That said, Matt concluded that gender was a strong signal.

Visualizations then pick up on simplifications.

Lucy Suchman looks at systems thinking. Systems are a problem, but they are important as networks of relations. The articulation of relations in a system is perfomative, not a given. Gender characteristics can be exaggerated – that can be the production of gender. There are various reasons why people choose to perform gender and their sex may not matter.

There is also an act of gender in analyzing the data. “What I do is tame ambiguity.”

Calculative exactitude is not the same as precision. Computers don’t make binary oppositions; people do. (See Ted Underwood, The Real Problem with Distant Reading.) Machine learning algorithms are good at teasing out loose family resemblances, not clear cut differences and one of the problems with gender is that it isn’t binary. Feminists distinguished between sex vs. gender. We now have transgender, cisgender … and exaggerated gender.

Now that we look for writing scales we can look for a lot more than a binary.

Is complexity just one more politically correct thing we want to do? Mandell is working with Piper to see if they can use the texts themselves to generate genders.

It is also true that sometimes we don’t want complexity. Sometimes we want simple forceful graphics.

Special Problems posed by Visualizing Literary Objects

Laura’s last move was to  then looked at gender in literary texts and discuss the problem of mining gender in literary texts with characters. To that end she invoked Blakey Vermeule, Why Do We Care About Literary Characters? about Miss Bates and marriage in Austen’s Emma.

Authors make things stand out in various ways using repetition which may through off bag-of-words algorithms. Novels try to portray the stereotypical and then violate it – “The economy of character.

Novels are performing both bias and the analysis of bias – they can create and unmask biases. How is text mining going to track that.

In A Matter of Scale, Jockers talks about checking confirmation bias to which Flanders replies about how we all operate with community consensus.

The lone objective researcher is an old model – how can we analyze in a community that develops consensus using text mining? To do this Laura Mandell believes we need capta open to examination, dissensus driving change, open examination of the algorithms and then how visualizations represent the capta.

Can we augment current visualizations to better serve humanists and, at the same time, make humanistic methods into systematic visualizations that are useful across disciplines outside the humanities?

She commented that there is no lack of visualizations, but most of them have their origins in the sciences. Further, evidence and argument get collapsed in visualization, something we want to tease apart. In doing this, can we create a set of visualization conventions that make humanities methods useful to other disciplines? Some of the things important to the humanities that we want to make evidence include: partial evidence, situated knowledge, and complex and non-singular interpretations.

Project development is part of what we have been focusing on. We have had to ask ourselves “what is the problem?” We had to break the problem down, agree on practices, frame the project, and sketch ideas.

Johanna talked about how we ran a charette on what was outside the frame. She showed some of the designs. Now we have a bunch of design challenges for inside the frame. One principle we are working with is that a visualization can’t be only data driven. There has to be a dialogue between the graphical display and the data. Thus we can have visualization driven data and vice versa.

We broke the tasks down to:

• Survey visualization types
• Study pictorial conventions
• Create graphical activators
• Propose some epistemological / hermeneutical dimensions
• Use three dimensionality
• Apply to cases
• Consider generalizability

Visualization Types

Johanna then went through showed the typology we are working with:

• Facsimiles are visual
• XML markup also has visual features, as do word processing views
• Charts, Graphs, Maps, Timelines
• 3D renderings, Augmented realities, Simulations
• Imaging techniques out of material sciences

Graphical Activators

She talked about graphical primitives and how we need to be systematic about the graphical and interactive features we can play with. What can we do with different primitives? What would blurring mean? What happens when we add animation/movement, interactivity, sound?

With all these graphical features, then the question is how can we combine the activators with interpretative principles.

Using the 3rd Dimension as Interpretation

She then talked about how we can use additional dimensions to add interpretation. She showed some rich examples of how a chart could be sliced and projected. We can distort to produce perspectives. The graphical manipulation lets us engage with the data visually. You can do anamorphic mapping that lets us see the data differently.

She then talked about perspectivization – when you add a perspective to the points. You dimensionalize the data. You add people to the points. Can we use iconography?

She showed ideas for different problems like the hairball problem. She showed ideas for how visualizations that are linked can affect each other. She showed ideas for the too much Twitter problem.

She talked about the problem of how to connect different ideological taxonomies for time like biblical and scientific time without collapsing them? How can we show the points of contact without reducing one to the other?

She then talked about the issue of generalizability. Can we generalize the ideas she has been working with? How can we humanize the presentation of data? Can we deconstruct visualizations?

Some of the questions and discussion after her talk touched on:

• To what extent are visualizations culturally specific?
• Does adding more graphical features not just add more of the same? Does it really challenge the visualization or does it add humanistic authority?
• How is adding more dimensions a critique of display rather than just more display?
• We talked about the time of making the visualization and the time of the unfolding of the visualization.
• We talked about how time can represent something or model something.
• Can we imagine games of making visualizations? How does the making of the visualization constitute a visualization? Can a way of making visualizations be more useful?
• How can any visualization have the APIs to be connected to physical controls and physical materializations?

Grimshaw has been interested in game sound for some time and how sound helps create an immersive experience. He is also interested in how games sonify others in a multi-player game (how you hear others). He is also interested in virtual reality and how sound can be used to give verisimilitude.

Why rethink sound? He started by discussing problems with definitions of sound and trying to redefine sound to understand sonic virtuality. The standard definition is that sound is a sound wave. The problem is that there are really two definitions:

• sound is an oscillation of pressure or sound wave, or
• sound is an auditory sensation produced by such waves (both from the ANSI documentation)

He mentioned another definition that I rather liked, that sound is “a mechanical disturbance in the medium.” This is from an acoustics textbook: Howard, D. M., & Angus, J. (1996). Acoustics and psychoacoustics. Oxford: Focal Press.

Not all sounds produce an auditory sensation (like ultrasound) and not all sensations are created by sound waves (eg. tinnitus). For that matter, sound also gets defined as that which happens in the brain. The paradox is:

• Not all sounds evoke a sound, and
• Not all sounds are evoked by sound.

He then talked about the McGurk effect when what we see overrides what we hear. Mouth movements cause us to hear differently so as to maintain a coherent version of the world. Perhaps sound waves are not all there is to sound. See https://www.youtube.com/watch?v=G-lN8vWm3m0

He provided some interesting examples of sounds that we interpreted differently.

What is interesting is that we defined sound as the sound source, as in that sound is a bird. This shows how our everyday sense of sound has nothing to do with waves.

Then there is the question of “where is the sound?” Does the sound come from the ventriloquist or from their dummy. The effect is known as synchresis (see here). We have the ability to keep more than one mapping system of sound sources. We locate sound very well, but in some cases, like cinema, we locate it in the world of the film. We can separate the location of the heard sound from its actual location.

Some other definitions include:

• Democritus said sound is a stream of particles emitted by a thing (a phonon)
• Sound is an event (Aristotle)
• Sound is the property of an object
• Sounds are secondary objects and pure events
• Sound is cochlear (involving sound waves) and non-cochlear sound (synaesthesisa)

Needless to say, the language of science around sound is very different from our everyday language.

His definition is for “sonic virtuality”:

Sound is an emergent perception arising primarily in the auditory cortex and that is formed through spatio-temporal processes in an embodied system.

A sonic aggregate is all the things that go into forming the perception of sound (like what you see.) Some is exosonus (what is outside) and some is endosonus (non sensuous components).

He talked about how for some animals there might be a sense of “smound” which is some combination of sound and smell.

The emergence of sound can determine epistemic perspective, though in some cases the perspective forms the sound. Imagined sound is just as much sound as exosonus sound.

In sonic virtuality, sound localization is a cognitive offloading of the location of sound onto the world. We tend to put sound where it makes sense for it to be.

Cognitively what seems to happen is that we form hypotheses about sound aggregate and eventually select emergent version of sound. This is embodied cognition which is time pressured – ie. pressured to decide quickly. We don’t know for sure.

Immersion and Presence

He then shifted to talking about games and the difference between immersion and presence. Immersion is supposedly objective – how close to reality is the simulation of sensory stimuli. Presence seems more subjective.

The way we locate sound out in the world is what leads to differentiation of self and not-self and that leads to sense of presence. Sound tells us about space.

If we want a better sense of presence in virtual reality – is increasing the simulation the way to go? VR systems try to deliver discrete sensory stimuli of greater and greater verisimilitude.

RV or real virtuality suggests a different approach – that of an appropriate level of stimulation that lets the brain make sense of the virtual space. You want the brain to actively engage.

If we model sound as perception then can we extract it? Can we extract sound? This is an area called neural decoding. (Nishimoto, S., Vu, A. T., Naselaris, T., Benjamini, Y., Yu, B., & Gallant, J. L. (2011). “Reconstructing visual experiences from brain activity evoked by natural movies.” Current Biology, 21, 1641–1646.) It seems they can now reconstruct what someone saw from the brain imaging.

Sonification

At the end he talked about sonification which connects to the 3DH project. Sonification is the audio equivalent to visualization. What is the value of representing data with sound? He gave some examples of sonification:

• A geiger counter is a sonification of radiation
• In radio astronomy sonification is used to help finding interesting or anomalous moments in radio waves. We can’t stop listening the way we can look away.
• PEEP (PDF) is a tool that sonifies network activity.

If we can transform large amounts of data into sound, what would we do? Each sensory modality has some things it is good at and some it is not so good at.

• Sound is good at time.
• Ambiguity is hard to visualize and often left off. Sound might be a way to keep ambiguity.
• Sounds can have meaning that could be used (but sound waves do not.)

Are there some sound primitives? Yes! there are some sound primitives that seem to be evolutionarily encoded in us like the sound of something rapidly approaching. Our brains seem to be attuned to certain sound wave attacks. What are the sound primitives that we can manipulate?

• Attack
• Loudness
• Tone(s)
• Texture (timbre)

Discussion

Some of the points that came up during discussion include:

Are there ways that sound can contradict vision as in a Jacques Tati movie like Playtime? It turns out that in most situations vision dominates hearing, but in others hearing can override vision. It seems that hearing is very sensitive to temporal changes, as in changes in rythym.

Are there ways of understanding the cultural and social in interpreting of sound?

Note

This was updated with corrections from Grimshaw.

• Geoffrey Rockwell: Reading Olympia. Visual Programming for Surveillance

  https://lecture2go.uni-hamburg.de/veranstaltungen/-/v/19384

• Martin Warnke: Visualizing a Thousand Years on Jewish Cemetaries and the dH Situation

  https://lecture2go.uni-hamburg.de/veranstaltungen/-/v/19384

• Johanna Drucker: Graphesis: Visual Forms of Knowledge Production

  https://lecture2go.uni-hamburg.de/veranstaltungen/-/v/19272

• Geoffrey Rockwell: The Strange Attraction of the Graph

  https://lecture2go.uni-hamburg.de/veranstaltungen/-/v/19218

I (Geoffrey Rockwell) gave the May 12th lecture on the subject of visual programming languages (VPL). I started by providing a surveillance context for understanding why VPLs are developed to provide a way into programming. The context was the CSEC slide deck leaked by Snowden that shows the Olympia Network Knowledge Engine which allows analysts to access other tools from the 5-Eyes services. Olympia includes a VPL for creating “chains” that automate surveillance processes (see the slide above in which the VPL is introduced.) I argued that in many ways we in the humanities also do surveillance (of cultural history) and we pay attention to tools like Olympia developed to help analysts automate interpretative tasks. I also argued that we need to study these types of slide decks as examples of how big data analysis is conceived. These are the types of tools being developed to spy on us and manage us. They are used by governments and corporations. We need to learn to read the software and documentation of algorithmic management.

The heart of the talk was a survey of VPLs. I argued that we have had specialized formal visual languages for some time for describing wiring diagrams or signalling plans for train stations. These languages allow someone to formally represent a process or design. I then gave a brief history of visual programming and then turned to VPLs in the digital humanities. This connected to a survey of some types of VPLs as I wanted to go beyond the pipe-and-flow types of VPL. I then summarized some of the opportunities and challenges for VPLs in the digital humanities and returned to Olympia. VPLs only work when there is a community that develops and understands the semantics of their visual language. Wiring diagrams work because people understand what a line connecting two icons means and what the icons mean in the context of electronics. For visualization in general and VPLs in particular to work in the humanities we need to develop both a visual literacy and a discussion around the meaning of visual semantics. One way to do that is to learn to read VPLs like Olympia. Again, the humanities need to take seriously these new types of documents as important and worth studying – both PowerPoint decks (that are handed around as a form of communication) and software like VPLs.

Visual Programming in the Digital Humanities

One of the first projects in the digital humanities to prototype a VPL for text analysis was the EyeConTact project by Geoffrey Rockwell and John Bradley. See also a paper Seeing the Text Through the Trees: Visualization and Interactivity in Textual Applications from LLC in 1999. This was inspired by scientific visualization tools like Explorer. Before that there were many projects that shared flowcharts of their programs. For example we have flowcharts of both how computers fit in scholarly concording and how the concording tools worked for PRORA. One can see how generations of programmers raised on flowcharting their programs would desire a flowcharting tool that actually was the programming.

The SEASR project developed a much more sophisticated VPL called Meandre. See Ian Milligan’s discussion of using Meandre. Meandre was designed to allow humanists a way of using all the power of SEASR. Alas, it doesn’t seem to be still maintained.

The best system currently available is built on an open VPL called Orange. Aris Xanthos has developed text analysis modules for Orange called Textable. Xanthos has a paper on TEXTABLE: programmation visuelle pour l’analyse de données textuelles (French). Orange is a well supported VPL that can be extended.

Opportunities and Challenges

Some of the points about the suitability of VPLs to the digital humanities that I made at the end include:

• VPLs are intuitively attractive
• Visual vocabulary is not always clear. What does a pipe mean? What direction do pipes go? Left to right? What flows through a pipe?
• Domain specific applications work best.
• We need to develop a community of use
• VPLs are good at the visualization of process (and data and results in that context). They show rather than hide the processes in a way that can be explored and fiddled with. They are good for showing the chain of filters and transformations that data goes through.
• VPLs are slower than traditional coding for proficient programmers.
• They can be fiddly.
• It is hard to handle a big codebase with a VPL as you end up hiding chains.
• The are good a showing chains of processes, but not at showing highly interactive systems.

Conclusions

Above all, as mentioned above, we need to learn to read visualizations (including VPLs) in the humanities. These are forms of communication that are increasingly important in the algorithmic state. They are used widely in business and government. They are essential to understanding how big data is consumed and used. I propose that developing a discourse and hermeneutics of visualization is fundamental to developing better visualization tools. The two go hand in hand.

Warnke leads the Institute for Culture and Aesthetics of Digital Media and a research group on Media Cultures of Computer Simulation (German), both at the Leuphana University Lüneburg. He studies knowledge orders of the digital and simulation. His talk had three parts.

1. First he looked at a dH project that is about visualization of a Jewish cemetery,
2. Then he discussed the general situation of such projects,
3. And he concluded by talking about dH in general.

The project he started with is called Relations in Space – Visualization of topological micro structures (German) It is a project working on the visualization of cemeteries including the Jewish cemetery in Hamburg-Altona. The project was funded from 2014 – 2016. It is now winding down.

Jewish tombs are not removed over time the way Christian ones are. They are left for eternity. As you walk the cemetery you go through time. The tombs tell us about names, what they did, good times, bad times, and the languages of the dead.

The challenge with the Altona cemetery was that there were two different sources of data to be merged. It is the map that does the job of synthesis. It does the magic of gathering information. You can make inferences that you couldn’t from just the data. He showed a simple map of the tombs and then showed variants. He talked about some inferences that can be from the map like why there are a line of womens’ graves when people are supposed to buried in order of death. It could have been an epidemic in the birthing house that killed a number of women at the same time.

There were two different databases that came from different groups and were not easily merged. Historians of architecture had gathered information about shapes of tombs and the epigraphists gathered information about inscriptions. For art historians cemeteries are miniature towns that can tell us about sculptural fashions.

The art historians had learned XML and oXygen and XSLT and were able to develop technical independence. The text on Jewish tombs is rich and poetic. They have acrostics. Warnke showed the XML for both the sculptural database and the inscriptions.

He then talked about data gathering they did on the field to build an accurate map. They used IncScape for vector graphs which works with XML. All the XML was brought in and then merged by a software company. They output merged XML that could be read into HyperImage a tools they have been developing for annotating images and then outputting interactive explorations.  HyperImage can be queried. The project is at http://www.uni-lueneburg.de/hyperimage/HI_Altona/ . Here you can see the interface.

One of the challenges for the programmers is that the historians constantly change their data. The programmers want stable data. They had to develop a versioning system.

Warnke showed the system and talked about the change in perspective from “distant reading” to “close reading” and back. Manovich doesn’t look any more at the single image – he looks only at large numbers. The HyperImage system lets you go in and out.

He commented on how when you do real projects you eventually hit the limits of your tools. The HyperImage editing tool ran into trouble with the Altona cemetery. He showed a different cemetery and how you can search the database and map the results. Thus the visualization becomes not just a way to explore the database, but also a way of viewing results.

Digital Humanities Projects

Warnke then talked about the situation of such projects. There are three miseries:

• Misery of the business model – One needs to spend time on the business model for projects rather than the project and the research. And, there are no good business models.
• Misery of infrastructures – Everything has to be redone every 5 years. University computing centres don’t want to maintain individual projects. Big infrastructure projects like Dariah can’t afford to either. Existing projects need to be updated as you are trying to do new ones. He talked about solutions like Prometheus which makes a number of databases available.
• Misery of funding – At the end all the projects are poorly funded. We are tempted to use commercial systems like Instagram to save our data where commerce maintains things, but that has problems too.

He also mentioned the joy of doing all this in an still open field – there is a joy in doing new things.

dH in General

Warnke concluded with some comments on dH in general. DH has a name now, which is a sign of its maturity. It is a scene. But it is also very text centric – what about images, sound, film, and so on.

One reason for the field being text centric is a problem with some multimedia like film where one can’t quote a clip the way you can quote a text passage. It is hard to openly address and reference materials other than text and this is the foundation of humanities work.

The relationship with CS also varies. In some cases CS is treated as an ancillary science. At the same time CS folk don’t think they can learn anything from the humanities. They don’t think they have any history. They think they can learn what we do quickly. Perhaps they are right – that we wallow in trivial subtleties.

He closed on idea of humanists becoming self-enabling. Kittler famously argued one should learn to read code as part of literacy.

To develop a conceptual blueprint for next generation digital humanities visualizations.

What would that mean? How can we do it? To do this we need to understand where we are and where we have to go and her talk did that by touching on:

1. How visualizations have an imprinted form of argument that comes from their origins.
2. Understand ideas about languages of form – ideas about how one can systematize the visual.
3. Look at how contemporary DH people use visualizations and what work do they want them to do.
4. Understand conventions of pictorial imagery and how most visualizations are pictorially impoverished.
5. Identify the epistemological challenges ahead.

She noted that 3DH is focusing on the visualizations of humanities documents and humanistic inquiry. Humanists are engaged in the production, interpretation, and preservation of human record. We need to think about problems of our practices like interpretation.

Types of Visualization in the Humanities

What kinds of visualizations do we use? Johanna Drucker gave a concise overview of the major types of visualizations used in the humanities. Each of these have different visual traditions and relationships to data.

1. Digitizations/remediations of original – These are visualizations that represent an original like a facsimile or a electronic text. They are digital surrogates.
2. Data-driven displays – These don’t represent an original, but represent some abstraction or analysis. Some types might include charts, graphs, maps, and timelines.
3. Visual renderings – These are complex 3D constructions and fantasies that use codes of pictorial representation with little data. They are extrapolations of the data. They augment the data. Some types include 3D renderings, augmented reality and virtual reality. They are often based on minimal data giving the illusion of repleteness.
4. Computationally processed visualizations – These are the special forms of imaging applied to artefacts like manuscripts. They adapt imaging techniques from the material sciences like MRI or x-ray scans.

All of these types of visualizations carry epistemological baggage, often from the sciences, but also from gaming (in the case of renderings.)

Examples

She the showed example images and talked about their limits. We can remediate the already remediated.

• Ben Fry, On The Origin of the Species: The Preservation of Favoured Traces. This visualization gives a nice overview of the editing process, but it can be difficult to explore deeper.
• Mapping Gothic France. This site provides a number of different visualizations to help understand gothic churches in France. She showed a visualization from the Comparisons section where you can stack outlines of naves and so on.
• InscriptiFact. A database of images where they have applied imaging techniques to bring out information in inscriptions.

Historical Origins: Imprints of Disciplines

Drucker gave a quick tour through some of the types of visualizations and how they are imprinted with their origins. They carry the baggage of their history of use. We need to understand these histories in order to understand how they will be interpreted or overintepreted.

• The table is one of the earliest and main forms of visualizing data. It is a powerful interpretative tool and we forget how it uses visual arrangement. It is invisible as a visualization.
• The tree (as in tree of life or family tree) has spiritual origins. It bears notions of continuity or, in the case of the tree of life it bears notions of hierarchy. Trees carry structure in subtle ways. Think of the family tree of consanguinity (who can inherit) – showing a mythic notion of inheritance.
• Charts have their origin in political arithmetic. They are a way of showing abstract data from human situations so that people can be managed.

Graphical “language of form”

Drucker then turned to the idea of a “language of form”. The languages of architecture (think Palladio) are a predecessor to the more recent idea of a language of visual form. These languages of form are often used in discussions of information visualization, but they have a history. This idea comes from the aspirations of the visual arts to be as authoritative as the sciences. One of the early attempts to develop such a language is in Humbert de Superville‘s Essai sur les signes inconditionnels dans l’art. He developed a language from which more complex works can be drawn. Kandinsky’s Point and Line to Plane 1926 was another attempt that breaks with with 19th century realism, developing a stable graphic language which becomes a foundation of graphic design languages. It is an attempt at an abstract set of signs. She talked about how we can mine the inventory of modern art for ideas. She showed the lino cuts of 1961 of Anton Stankowski whose Functional Graphics look extraordinarily like templates for the visualizations we use today. He imagined ways to make invisible processes visible. She then mentioned how perceptual psychology also developed a language of form trying to find a graphic vocabulary.

Important to data visualization is Jacques Bertin and his Semiology of Graphics. In this he distills 7 graphic variables with which show information: size, tone, texture, color, orientation, shape, and position. Drucker added that in dynamic situation we need to add: motion, rate of motion, direction of motion, and the sound of motion. Graphical systems make use of these variables. They also carry semantic value. As a principle, we should use things for what they are good at showing.

Drucker then showed some types of visualizations that haven’t been used like architectural plans. We don’t use perspective – we obliterate dimensions.
When we leave out perspective we leave the perspective of the speaker out. This creates the illusion that it is as if the visualizations speak for themselves. We also lose the ability to use distortion or translation of perspective.

Another type that we haven’t used is the cabinet of curiosities like Wormius’ one. She talked about the complexity of the image and how much data it carries using perspective, tonal value.

She compared a Moretti graph of Hamlet to a Daniel Maclise painting of the play within the play. She showed a Charneaux lingerie image that shows how lingerie adds structure to the body. She showed a cartoon showing a step by step process. All these to show how impoverished visualizations were.

What is the work of visualizations and what do we want to do

Visualization can be a type of fiction that obscures a lot in order to show an overview or gestalt. Some of the things we want to do include:

• Add dimensions and perspective back – flat screens are lacking
• Translate images through rendering – can we use the visual for what it is good at?
• We want to be shown degrees of certainty.
• Map views can make it look as if the same space is the same – we want to show distortions and how maps are of their time. We want to avoid historical anachronism and use data to build a map rather than structure the data with a map.
• We want to use renderings to hold evidence not to obscure provide illusion of it.

What is the work ahead

Drucker closed by talking about the epistemological issues and graphic challenges ahead.

• Partial knowledge: how do we show what we don’t know – figure without ground
• How can we show evidence and see what shape it takes rather than imposing shape
• How can we situate knowledge – provide a point of view
• How can we be clear about the historical specificity and diverse ontologies
• How can we show process – visual and non-visual
• How can we provide for annotation – commentary and non-visual
• How can we visualize the methodological. How can we show contradiction, incompleteness, doubt, uncertainty, and parallax.
• How can we show non-standard/variable metrics – affective metrics, diverse scales
• How can we make a semantically legible system

I (Geoffrey Rockwell) gave the second lecture on Thursday the 14th with the title The Strange Attraction of the Graph (video). I started with the image above which is of a PowerPoint slide from one of the decks shared by Edward Snowden. This is the Summary of the CSEC Slides (see my blog entry on these slides) where CSEC showed what their Olympia system could do. The Summary slide shows the results of big data operations in Olympia starting with a target (phone number) and getting a summary of their telecommunications contacts. The image was not in the slides shared by either of the media companies (Fantastico or Globe and Mail) that reported on this as it has too much information. Instead hackers reconstructed it from video that showed it in the background. That gives it the particular redacted and cut-up quality.

I showed this slide as an example of a visualization we want to interpret. My talk addressed the question of how we can interpret visualizations like this, namely graphs in the computing sense of sets of linked points. I didn’t develop a general hermeneutics of visualization, or talk that much about this CSEC slide, but stayed focused on one type of visualization, the graph with nodes (vertices) and edges on a plane.

Here are some of the approaches I took.

What is a graph?

It is important to draw on the traditions of computing to understand visualizations, especially if we want to understand how they are rendered by computers, how they are conceived by programmers, and how the tools work. For this reason we want to understand the basics of graphs. Graphs are representations of sets of points and their links. The simplest heuristic to interpreting a graph is to ask what the points represent and what the lines connecting points represent. That’s all there is to interpret in a simple graph which shows a set of nodes or vertices with edges between some of them. There is no information in the distance between nodes or their position on the screen; all that is generated by the tool based on rules about how to make graphs pretty and easy to read. (See the poetics below.)

I should add that a key heuristic to interpreting any visualization is ask what is metrical (based on a measurement in the data) and what is not. For example, in word clouds, the location of the word and its colour is often random while the size of the word and its centrality is based on the frequency. Many of the graphic features can have nothing to do with the data being represented which means it has nothing to do with the phenomenon the data comes from.

When rendering graphs a computer needs ways to represents the points, ways to arrange the objects, ways to represent the edges, and ways of decorating the background plane. The rules or algorithms used to determine how the points, lines and plane are drawn often have to do more with aesthetic choices than with some feature of the thing being represented. This can mislead us to overinterpret a graph.

What are some of the types of graphs?

The simple of idea of a network graph has been used in many familiar contexts to represent very different types of things from sentences to networks. We have traditions of using and interpreting them that are important to understand as interpretative expectations can bleed from one to another. Some of the types of familiar graphs that I showed include:

• Family trees where there are conventions of layering the nodes in generations. These use the layered location to arrange the generations of children nodes so you can tell where the third generation is quickly. We could say that these use a structured surface for the plane where most do not.
• More generally tree graphs usually start from one node and there is interpretative expectation that things evolve out from there. Examples might be trees of knowledge or trees of life.
• One particular type of tree diagram that is used in data representation are dendograms which show the clustering of things at different levels.
• In linguistics we see parse trees that show the parts of a sentence.
• State Diagrams are less familiar, but they and flow charts can show processes. John B. Smith used state diagrams for rhetorical moves in “Computer Crticism” (1978).
• Flow Charts can also help with making decisions (decision trees). These can be considered as part of a larger class of diagrams for reasoning with. Such diagrams don’t represent something that already exists, but they let you generate knowledge.
• Visual Programming environments are an extension of knowledge generating diagrams that allow one to “pipe and flow” data through simple processes.
• Radial charts are a form of arrangement that puts the points around a circle and then connect them with curves. TextArc (not really a graph, but worth mentioning) and Saklovskie’s NewRadial are examples.
• Arc Diagrams are another form of arrangement that puts the points on a line and then creates arcs between them. This can be used in timelines.
• Sociograms or social network graphs are in many ways the most important tradition and they go back to work by Jacob L. Moreno in the 1930s. They show the relationships between groups of people and are used in ethnography and sociology. They are important because they show us, people, in networks of relationships. They have come to form our imagination of how relationships can be shown. I compared this way of representing a relationship to that in a painting like David’s Death of Socrates. I showed a sociogram tool called RezoViz that we developed as part of Voyant. Here it is showing people and other entities in Humanist.
• Citation Networks show a particular type of relationship between academics – that of citation like this one of Comparative Literature 2004-14. Who cites who. What people or articles are central.

The Poetics, Ontology and Epistemology of Graphs

Poetics: Expanding on the heuristics for interpreting a graph mentioned above, it is useful to ask how the graphs we see are made (poetics.) What sort of data are network graphs typically used for? What sorts of algorithms are there for generating (rendering) the graphs. A good place to look for the algorithms and the aesthetics they encode is the computing literature and code libraries out there that people use. Di Battista et. al. in Algorithms for Drawing Graphs: an Annotated Bibliography (PDF) (1994) provides a great overview. He lists the aesthetics for attractive simple graphs as:

• display symmetry;
• avoid edge crossings;
• avoid bends in edges;
• keep edge lengths uniform;
• distribute vertices uniformly. (p. 7)

One of the ways these aesthetic guidelines are achieved with real code is to create a physics models of the data as a collection of rings connected by springs and then calculate and show the rings (nodes) in tension with each other. This is why network visualization tools often have sliders to control tension or how much nodes repel each other.

Epistemology: a large topic that I didn’t have time to go into, is how we make and then read meaning from visualizations in general and graphs in particular. This is obviously connected to the issue of the interpretation. I mentioned Ben Schneiderman’s Visual Information Seeking Mantra from “They Eyes Have It” (1996).

Visual Information Seeking Mantra:

• Overview first,
• Zoom and filter,
• Details on demand

This mantra is both normative and descriptive in that it is meant to capture how people read visualizations and therefore how you should design them. The idea is that users begin by getting an overview, then they zoom in and explore parts/relationships, and lastly they check details that interest them. We need to learn a lot more about how exploration works cognitively as compared to reading.

Another approach to the generation of meaning is to look at the graphical features (points, lines, plane) and how they are used for what. The assumption is that there is some sort of analogy between the linking of points and the real-world relationships being graphed. The graph creates a visual model of the abstract network of relationships. One can see how the imagination begins to dominate our conception. Relationships are not lines between solitary points, but that is how we have come to “see” them thanks to sociograms and other network graphs.

A direction I would like to take further would be to draw on how artists think of abstract art. I particular Kandinsky’s Point and Line to Plane strikes me as useful artist’s view on the use of these graphic primitives. For example, he makes a point about External and Internal Experience. One can experience something from outside it (as if through a window above it) or one can experience it inside the phenomenon.

Ontology: connected to the epistemology of the graph is the ontology. What are graphs and what do they represent. I argued that they are models of models. They are representations of data sets, not of the thing studied. The datasets in turn are measurements or observations of the thing studied. Sometimes there are intermediate models giving us layers on layers. The point is that they show interpretations not the thing. This is a point Moretti makes and Smith actually theorized in 1978 in his structuralist “Computer Criticism.”  One could say that network graphs pretend to show the underlying structure of the phenomenon and that is part of their attraction.

Interestingly graphs are not necessarily quantitative. One can represent with a graph a table of friend’s names and whether or not they are friends. All the cells are names and interpretations of friendship entered by hand. There is no quantitative, only categorical data.

Why the attraction?

This brought me back to the title and the attraction of the graph. This part was more speculative. Some reasons why network graphs in general and the CSEC one in particular are attractive include:

• Simple graphs are simple – just points and lines. This leave lots of room for the imagination to see things into them and to explore them into significance.
• Visual exploration of a rich graph feels much more open and free than following a sequence of statements as one might find in a text, syllogism, or code which is read in a linear way. They feel like they can be consulted rather than having to be read end to end.
• They show time in the space of the plane. Everything that might be complex and hidden in time is arranged out on the planar screen to be seen in one glance.
• Another way to put this is that they provide both overviews and then details. There is something attractive to the way one can explore a rich graph.
• Or we could say that time (and distance) are collapsed into and abstraction for the screen.
• This puts everything into an arrangement for a gestalt view where you have the impression you can see the whole. You can grok it! which establishes yet another connection, this time between you and the visualization.
• One of the things compressed is the process of generating observations and then the visualization. The messiness is hidden in the white box – the black box you don’t even know is there.
• Finally, the network graph has become the visualization of the networked age. I showed early and later graphs of the internet, graphs of communication and how computers work, graphs of network cables, ontological graphs and graphs of surveillance software architecture. The graph is the form with which we imagine virtual culture. It is a form without distance (time) or position (space). It is the simplest form of abstraction.

Of course, it also has little to do with the material form of computing or networks or software, but that is the point of the virtual. It approximates the way we imagine pure abstraction.

As for the communications graph I started with, it had further attractions as it had traces of its generation (time) and redaction. It has a context which can be unpacked from this visualization. In this way the graph is far richer than most graphs that hide all the messy.

What makes an attraction strange? I didn’t have time to explain my title, but the idea of an attractor in systems is that it is something towards which they evolve. I was using the term metaphorically for the way we can feel there is an interpretation that acts as an attractor pulling us towards the emergence. The attractor can be felt as a hidden structure that attracts the arrangement of the visualization. This sense that there is an attractor operating on us in visualizations is what is strange. For that matter the attractors are strange. They don’t exist in the sense of some underlying truth which we get closer to in certain visualizations, but we are driven to keep on trying as if attracted to their flame.

What are some guidelines for design?

The problem with network graphs is that it is easy to overinterpret them. I was asked during question period if I had suggestions for how to design network graphs that would be hermeneutically more robust. Here are some of the ideas I put forward:

• Provide controls so the user can play with the visualization and see that there is no right arrangement. Controls also give some representation of the user’s perspective making the point that all visualizations are from a perspective.
• By extension one can animate the graph as many based on physics models are. The nodes can move around on their own as the springs bounce them around. This makes it clear there is no objective arrangement. The user can also drag stuff and watch what rubber bands pull what else. This manipulation is not only satisfying, but it helps the user understand what is based on the observations and what is a rendering decision.
• Present the graph with other linked visualizations that let the user see different and conflicting views onto their data. For humanists it is also a good idea to give them access to the “original” text or metadata so they can use other reading methods to see if there are reasons for the patterns being seen. This is an example where giving details allows intuitions to be checked in ways that humanists are comfortable with.
• I closed by showing a Mathematica notebook where I could show the code with the visualization. The notebook is a form of meta interface which visually arranges text, code and results in a way that is supposed to recall the scientist’s notebook. This means that any visualizations (even interactive ones) are woven together with their code. This way you can see the logic of the rendering.
• Reflecting later I wondered if Kandinsky’s point about internal experience couldn’t be revised into a guideline – namely that visualizations should look as if they are viewed from inside the experience. This would mean that it is clear that not everything can be seen (lines might always lead off the screen).