Content

Drawing Interactions

2018

Techniques for the transcription, analysis and presentation of social interaction, in a week long hack session, with funding.

That was the pitch Saul Albert made to me. Tasked with coming up with prototypes, I happily accepted. After all, these themes and the practical work around it became a large part of my PhD. I’d forked one app and wrote another from scratch, and done so while around the others involved – Pat Healey, Claude Heath and Sophie Skatch.

A day exploring, a day presenting, and for me the time inbetween deep in code with an exacting challenge: it was fun, rewarding work, and everything we did has gone down well. I’m certainly happy with the app I made. Watch the demo above, and read more in the diary posts below.

Diary entries

the hack starts

the week hack session starts. claude and sophie talk through how their art practice has informed their research on social interaction, and we all discuss how that could inform graphical tools and techniques for the transcription, analysis and presentation of social interaction… a fun day with old friends doing good work.

having thought i might be making all sorts of acetate-and-pens-and-displays hacks, it becomes pretty clear that a tablet+pen app that could support their kind of approach is achievable, and would be a great platform to then experiment from.

hands-on with time

first challenge for the drawing interactions prototype app is to get ‘hands-on with time’. what does that mean? well, clearly controlling playback is key when analysing video data. but that also frames the problem in an unhelpful way, where playback is what’s desired. rather the analyst’s task is really to see actions-through-time.

pragmatically, when i read or hear discussions of video data analysis, working frame-by-frame comes up time and time again, along with repeatedly studying the same tiny snippets. but if i think of the ‘gold standard’ of controlling video playback – the jog-shuttle controllers of older analogue equipment, or the ‘J-K-L’ of modern editing software – they don’t really address those needs.

so what might? i’ve been thinking about the combination of scrolling filmstrips and touch interfaces for years, promising direct manipulation of video. also, in that documentary the filmstrips are not just user-interface elements for the composer, but displays for the audience. such an interface might get an analyst ‘hands on with time’, and might better serve a watching audience. this is no small point, as the analysis is often done in groups, during ‘data sessions’. others would be able to tap the timeline for control – rather than one person owning the mouse – and all present would have an easy understanding of the flow of time as the app is used.

of course, maybe this is all post-hoc rationalisation. i’ve been wanting to code this kind of filmstrip control up for years, and now i have.

a little postscript: that panasonic jog-shuttle controller was amazing. the physical control had all the right haptics, but there was also something about the physical constraints of the tape playback system. you never lost track of where you were, as the tape came to a stop, and starting speeding back. time had momentum. so should this.

ready for the unveil

after an intense week, the drawing interactions app is ready to be unveiled. the iPad Pro and Apple Pencil turn out to be amazing hardware, and i’ve really enjoyed the deep dive into the kind of precise, expressive, powerful code that’s required for this kind of iOS app. it

  • plays a video
  • lets you navigate around the video by direct manipulation of the video’s filmstrip.
  • when paused, you can draw annotations with a stylus.
  • these drawings also become ‘snap’ points on the timeline
  • these drawings are also drawn into the timeline, partly to identify the snap points, and partly with the hope they can become compound illustrations in their own right
  • when not paused, you can highlight movements by following them with the stylus

i got there. huzzah! that last feature, drawing-through-time, i’m particularly pleased with. of course, there are bugs and plenty of things it doesn’t do. but it’s demoable, and that’s what we need for tomorrow’s workshop.

workshop time

the drawing interactions project is not just the app. and even if it were, what is an app without users. so: a workshop, at ‘new directions in ethnomethodology’.

sketching posture

the project takes a fresh approach to (e.g. conversation analytic) transcription, based on long-standing artistic and drafting skills. so, here we are in the workshop, all learning life drawing. while tracing people’s outlines from photo and video source material can get you a long way (something i quickly learnt coming into product design with an engineering background), it can also constrain what can be communicated – or even seen.

cue sophie, whose quick and economical illustrations convey qualities like posture and weight.

a room full of ethnographers drawing you

if you ever wondered what it might look like to have a room full of ethnographers learning life drawing with you as the model… well, it’s like this. an unexpected turn of events, to say the least.

i blame sophie, to the left in the photo =]

designer infographics?

the app, as conceived for a prototype, is all about exploratory research. of course, ultimately the insights and the backing evidence need to be distilled for publication.

happily, sylvaine tuncer, barry brown and others have been working on a design-informed exploration of ways to standardise presentation. right up my alley… if we can continue this project, there’s so much that could be done, and i’d love to do it.

drawing interactions journal paper

Drawing as transcription: how do graphical techniques inform interaction analysis?

Drawing as a form of analytical inscription can provide researchers with highly flexible methods for exploring embodied interaction. Graphical techniques can combine spatial layouts, trajectories of action and anatomical detail, as well as rich descriptions of movement and temporal effects. This paper introduces some of the possibilities and challenges of adapting graphical techniques from life drawing and still life for interaction research. We demonstrate how many of these techniques are used in interaction research by illustrating the postural configurations and movements of participants in a ballet class. We then discuss a prototype software tool that is being developed to support interaction analysis specifically in the context of a collaborative data analysis session.

Albert, S., Heath, C., Skach, S., Harris, M., Miller, M., & Healey, P. (2019). Drawing as transcription: how do graphical techniques inform interaction analysis? Social Interaction. Video-Based Studies of Human Sociality, 2(1). https://doi.org/10.7146/si.v2i1.113145

Open Access / Creative Commons BY-NC-ND

drawing, multimodality and interaction analytics

based on the drawing interactions work, we were asked to run a day-long workshop for the national center for research methods. happily, “drawing, multimodality and interaction analytics” sold out early, and went well. i even got some sketches out of it, courtesy of sophie.

happy punters aside, this spurred a day-or-so sprint of app development. i wanted to show some live demos using the sources and drawing techniques that would be presented during the sessions. and that needed a document based app. and that needed a way to persist the timestamped drawings, and so on. which it now all does. and with that, the app is no longer just a prototype. it has practical value to others.