Content

tagged: phd

Liveness: an interactional account

Coming off stage, I’ve often wondered to myself whether the audience would have had a better show if I’d pressed play on a particularly good studio take, and bobbed my head to my email inbox instead. My thing is a kind of improvised film, but anybody who has used a laptop on stage must have felt something similar at some point. My instinct is to make that laptop a better tool, or turn the interface into something legible for the audience. As a designer I’d need a way of reasoning about the situation… and not being able to reason about live events was the thread that kept on pulling.

That thread led to a place on the Media and Arts Technology programme at Queen Mary University of London, and some years later, a PhD dissertation. The examiners judged it “a very good piece of work: thorough, clear, based on strong and wide-ranging literature review, and reporting several rounds of well-conceived and well-conducted research”. toby*spark? dr*spark!

You can get a quick hit here, where my work got into national newspapers. The most appropriate introduction is probably this conference talk, where I get to play a little with everybody being there live, and sell my epistemic position: audiences and interaction, empirically.

But the real thing, well it goes something like this…

Dissertation Outline

Music in concert, sport in a stadium, comedy in a club, drama in a theatre. These are live events. And there is something to being there live. Something that transcends the performance genre. Something that is about being there at an event, in the moment: caught in the din of the crowd, or suddenly aware you could hear a pin drop. This something is the topic of this dissertation.

The research works to transform the vernacular live into operational processes and phenomena. The goal is to produce an account of ‘being there live’ by describing how features common to live events shape the experiences had in them. This will necessarily pare away much of the richness of these genres of performance, as it will examples of individual events. What is sought is a foundational understanding, upon which such richness can be better considered.

The thesis is that an interactional analysis provides the most simple, clearly expressed and easily understood account of the liveness of live events. This is drawn from empirical investigation of the factors that contribute to the sense of being there live. In particular, it questions whether there are general patterns of interaction that can be used to generalise across live events.

1. On the liveness of live events

First, the notion of the liveness of live events is set out. Chapter one uses existing literature to focus the research on human interaction. It does this by developing the examples of music in concert, comedy in a club and drama in a theatre and relating these to theoretical accounts of performance and liveness. Chapter one concludes with the need to establish a more perspicuous account of liveness than found in the literature.

2. Examining an event

The accounts of live events switch from written sources to direct observation in chapter two: a stand-up comedy event is described and analysed. The idea of performance as an interactional achievement is supported, most simply by the performer stating the act is going to be a dialogue with the audience, but more pervasively in the elaboration of some of the work identified in chapter one to this new data. The study of mass-interaction is limited however, and this chapter ultimately shows the need for a better understanding of what to look for in human interaction, and a different approach to data collection.

3. Audiences and interaction

The observational study of stand-up comedy demonstrated a gap in theory, method and instrumentation for the study of mass-interaction in live events. Chapter three addresses this gap by reviewing literature on human interaction. Applying the concerns of this literature in the context of live events leads to consideration of mass spectatorship. Is there a distinction between people who are merely massed together and audiences? This chapter argues that there is and that it consists in the specific kinds of social organisation involved.

4. Experimenting with performance

The literature reviewed in chapter three also motivates an experimental programme. Chapter four presents the first, establishing Comedy Lab. A live performance experiment is staged that tests audience responses to a robot performer’s gaze and gesture. This chapter provides the first direct evidence of individual performer–audience dynamics within an audience, and establishes the viability of live performance experiments.

(See also: tobyz.net project page/)

5. Experimenting with audiences, part one

The two main Comedy Lab experiments are presented in chapter five and six. Having successfully gained evidence of a social effect of co-presence in the first experiment, these two test the social effects of co-presence to the fullest extent practicable. This requires an expansion of the instrumentation, which opens chapter five. The basic premise of the experiment that follows is to have the performer as either an interacting party or not, and see what performer–audience and audience–audience dynamics are identifiable. The experiment contrasts live and recorded performance, directly addressing a topic that animates so much of the debate identified in chapter one. The data provide good evidence for social dynamics within the audience, but little evidence for performer–audience interaction. This emphasises that both conditions are live events, as even though the recorded condition is ostensibly not live, a live audience is present regardless and it is this that matters. Overall, the results affirm that events are socially structured situations with heterogeneous audiences.

6. Experimenting with audiences, part two

The second main Comedy Lab experiment is presented in chapter six. The manipulation is now of the audience. The basic premise is to vary the exposure of individuals within the audience. The experiment contrasts being lit and being in the dark, when all around are lit or not. The data provide strong evidence for social dynamics within the audience, and limited evidence for performer–audience dynamics. Spotlighting individuals reduces their responses, while everyone being lit increases their responses: it is the effect of being picked out not being lit \emph{per se} that matters. The results affirm that live events are social-spatial environments with heterogeneous audiences.

7. Visualising performer–audience dynamics

In pursuing Comedy Lab, challenges of capturing the behaviour of performers and audiences were repeatedly addressed. Beyond the issues of instrumentation already discussed, the data sources were diverse, and their combination and interpretation required original work throughout. Building on this work, a further contribution of method is made in chapter seven. A method to facilitate inductive analyses of performer–audience dynamics is presented, along with the actual dataset visualiser tool developed. In the same way that video serves the study of face-to-face dialogue, augmented video and interactive visualisation can serve the study of live audiences.

(See also: tobyz.net project page/)

8. Liveness: an interactional account

The opening chapter set out the thesis that an interactional analysis should provide the simplest, most perspicuous account of the liveness of live events. In the chapters leading to the final chapter, eight, an empirical understanding of the interactional dynamics of particular live events has been put forward. This is now synthesised into an interactional account of liveness.

First, the Comedy Lab results are discussed as a response to the apparent paradox set up earlier in the dissertation. The programmatic hypothesis is that across live events, generalised patterns of mass interaction should be identifiable. However the interactional mechanisms that are well understood are dyadic and are found in everyday contexts. At first sight, live events – massed! an escape from the everyday! – would seem to be neither.

Following this, the interactional account of liveness is described. The concept of social topography is introduced and the nature of experience considered. It is argued that the experiments provide evidence that the kinds of experience-shaping conversations had after an event – “did you enjoy it?” – are happening, pervasively, during the event. With different interactional resources, they cannot be the complex verbal constructions of dialogue outright, but nonetheless they are there: moments of interaction that can change the whole trajectory of an experience. The interactional understanding of liveness put forward is then used to variously underpin, and undermine, some ideas of liveness encountered in the literature.

The exposition is completed with a consideration of how this account can provide a systematic basis for design. It argues that people have been long been alive to the issue of liveness and that technological interventions in particular can be powerful ways of reconfiguring experiences unique to live events. Further, as the dynamics of the interactions amongst audience members have been shown key to the experience of a live event, if practitioners attend to this directly new opportunities for intervention will open up.

Finally, the investigation of unfocussed interactions is discussed as future work, with specific challenges and risks informed by the Comedy Lab analysis. And it is noted that in measuring what is going between audience members, in making sense of those measures, in doing this with a much finer grain than anyone else has considered, and relating all this to experience… that this shows the need for a different orientation from performance studies, cognitive psychology, or even audience studies.

Document and dataset

project | 2017

Visualising Performer–Audience Dynamics

Live performances involve complex interactions between a large number of co-present people. Performance has been defined in terms of these performer–audience dynamics (Fischer-Lichte 2014), but little is known about how they manifest. One reason for this is the empirical challenge of capturing the behaviour of performers and massed audiences. Video-based approaches typical of human interaction research elsewhere do not scale, and interest in audience response has led to diverse techniques of instrumentation being explored (eg. physiological in Silva et al. 2013, continuous report in Stevens et al. 2014). Another reason is the difficulty of interpreting the resulting data. Again, inductive discovery of phenomena as successfully practised with video data (eg. Bavelas 2016) becomes problematic when starting with numerical data sets – you cannot watch a spreadsheet, after all…

A spoken paper presented at the International Symposium on Performance Science, Reykjavík 2017. The talk is a good way to see what I got up to during my PhD… and hey, there’s no stats and lots of pretty pictures.

project | 2017 | downloads: ComedyLabDatasetViewer-HitTest.png · ComedyLabDatasetViewer-Promo-5up.png · ComedyLabDatasetViewer-Promo-Aud11.png · ComedyLabDatasetViewer-Promo-BlurZoom.png · ComedyLabDatasetViewer-Promo-HideTop.png · ComedyLabDatasetViewer-Promo-Perf.png · ISPS2017-tobyspark-abstracts2.jpg · ISPS2017-tobyspark-abstracts3.jpg

Comedy Lab

Three live performance experiments researching performer-audience-audience interaction. They are the empirical contribution of my PhD on ‘liveness’, and required the visualising performer–audience dynamics work.

Comedy Lab: Human vs. Robot

An experiment that tests audience responses to a robot performer’s gaze and gesture. In collaboration with Kleomenis Katevas and part of Hack the Barbican. For my PhD, it provides the first direct evidence of individual performer–audience dynamics within an audience, and establishes the viability of live performance experiments.

Comedy Lab: Live vs. recorded

The experiment contrasts live and recorded performance – directly addressing a topic that animates so much of the debate around ‘liveness’. The data provides good evidence for social dynamics within the audience, but little evidence for performer–audience interaction. While these audiences were indifferent to live vs. mediated performance, the results affirm that events are social-spatial environments with heterogeneous audiences. The results emphasise that both conditions are live events, as even though the recorded condition is ostensibly not live, a live audience is present regardless and it is this that matters.

Comedy Lab: Lit vs. all lit

The experiment contrasts being lit and being in the dark, when all around are lit or not. The data provides strong evidence for social dynamics within the audience, and limited evidence for performer–audience dynamics. Spotlighting individuals reduces their responses, while everyone being lit increases their responses: it is the effect of being picked out not being lit \emph{per se} that matters. The results affirm that live events are social-spatial environments with heterogeneous audiences.

project | 2013 | downloads: 2013-05-31_13-25-20_tbz-minime-011.jpg · 2013-06-01_22-50-59_tbz-minime-026.jpg · 2013-06-03_23-11-09_tbz-minime-020.jpg · 2013-06-04_17-19-54_tbz-minime-062.jpg · 2013-08-05_17-25-59_tbz-minime-018.jpg · comedylab-8aug2013-tobycam-15-dsc_8847.jpg · comedylab-8aug2013-tobycam-17-dsc_8898.jpg · comedylab-8aug2013-tobycam-19-dsc_8956.jpg · comedylab-8aug2013-tobycam-22-dsc_9002.jpg · comedylab-8aug2013-tobycam-4-dsc_8750.jpg · comedylab-8aug2013-tobycam-6-dsc_9011.jpg

comedy lab » on tour, unannounced

an email comes in from a performance studies phd candidate asking if they could watch the whole robot routine from comedy lab: human vs. robot. damn right. i’d love to see someone write about that performance as a performance.

but, better than that staging and its weird audiences (given the advertised title, robo-fetishists and journalists?) there is comedy lab #4: on tour, unannounced. the premise: robot stand-up, to unsuspecting audiences, at established comedy nights. that came a year later with the opportunity to use another robothespian (thanks oxford brookes!). it addressed the ecological validity issues, and should simply be more fun to watch.

for on tour, unannounced we kept the performance the same – or rather, each performance used the same audience responsive system to tailor the delivery in realtime. there’s a surprising paucity in the literature about how audiences respond differently to the same production; the idea was this should be interesting data. so i’ve taken the opportunity to extract from the data set the camera footage of the stage from each night of the tour. and now that is public, at the links below.

the alternative comedy memorial society

gits and shiggles

angel comedy

the robot comedy lab experiments form chapter 4 of my phd thesis ‘liveness: an interactional account’

Four: Experimenting with performance

The literature reviewed in chapter three also motivates an experimental programme. Chapter four presents the first, establishing Comedy Lab. A live performance experiment is staged that tests audience responses to a robot performer’s gaze and gesture. This chapter provides the first direct evidence of individual performer–audience dynamics within an audience, and establishes the viability of live performance experiments.

http://tobyz.net/project/phd

there are currently two published papers –

and finally, on ‘there is a surprising paucity…’, i’d recommend starting with gardair’s mention of mervant-roux.

diary | 03 may 2019 | tagged: comedy lab · phd · qmat · research

thesis published

it’s a funny thing, handing in a thesis, submitting corrections and so on, but not being able to link anyone to the work. finally, so long after may, but at least not so long after the viva, here it is. all 169 pages of it.

https://qmro.qmul.ac.uk/xmlui/handle/123456789/30624

diary | 20 dec 2017 | tagged: phd · research · qmat

renaissance garb means dr*spark

dressed up as a renaissance italian, doffed my hat, and got handed a certificate… that was a placeholder, saying the real one will be in the post. truly a doctor, but yet still one little thing!

best of all, is that first-born is no longer my totem of not having got this done; the bigger and better she got, the more egregious the not-being-a-doctor was.

diary | 18 dec 2017 | tagged: phd · research · qmat

viva

the dissertation had done the talking, and the viva was good conversation about it wrapped up with a “dr. harris” handshake. phew, and woah. having been in a death-grip with the dissertation draft for so long, nothing in the whole experience could touch the wholesomeness of simply hearing “i read it all, and it’s good”.

supervisor –

Dear All,

I’m delighted to report that Toby Harris successfully defended his thesis "Liveness: An Interactional Account” this morning.
The external said: “that was a sheer pleasure”. (very) minor corrections.

Pat.


Pat Healey,
Professor of Human Interaction,
Head of the Cognitive Science Research Group,
Queen Mary University of London

external examiner –

This is a richly intriguing study of the processes of interaction between performers, audiences and environments in stand-up comedy – a nice topic to choose since it is one where, even more than in straight theatrical contexts, ‘liveness’ is intuitively felt to be crucial. But as Matthew Harris says, what constitutes ‘liveness’ and how precisely it operates and matters, remains elusive – if pugnaciously clung to!

The conclusions reached and offered – which more than anything insist on the value and necessity of seeing all audience contexts as socially structured situations – both rings right, and seems to be based well in the details of the data presented. And the cautions at the end, about the risks with moving to higher levels of abstraction (wherein ‘the audience’ might become increasingly massified, rather than understood processually) looks good and valuable.

The specific claims made – that the ‘liveness’ of the performer matters little (e.g. by replacing him/her with a robot, or with a recording) – will nicely infuriate those with an over-investment in the concept, and will need careful presentation when this work is published. The subsequent experiment on the role of spotlighting or darkness on the kinds and levels of interaction audiences have with each other, and with the performer are also nicely counter-intuitive.

internal examiner –

I greatly enjoyed reading this thesis. It strikes a good balance between theory and experiment and makes several well-defined contributions. The literature reviews show a keen insight and a good synthesis of ideas, and the motivation for each of the experiments is made clear. The writing is polished and engaging, and the order of ideas in each chapter is easy to follow.

diary | 06 oct 2017 | tagged: phd · research · qmat

isps » performer-audience dynamics talk

had a lot of fun with my talk ‘visualising performer–audience dynamics’ at ISPS 2017. with a title like that, some play with the ‘spoken paper’ format had to be had.

pleasingly, people were coming up to me to say how much they enjoyed it for the rest of the conference. huzzah!

i recorded it, and have stitched it together with the slides. the video is here, on the project page.

diary | 01 sep 2017 | tagged: comedy lab · performer–audience dynamics · phd · qmat · research · iceland · talk

hic et nunc

at some point in my PhD, i found out the phrase i started out with – “the here and now of us together” – had some cultural richness i can’t deny some pleasure over. in there is the imperative motto for the satisfaction of desire. “I need it, Here and Now”

so here is a little indulgence for my talk at ISPS2017 in the making. while it amuses me, i have to face up to the fact that wearing a slogan t-shirt in latin is clearly a dick move.

diary | 28 aug 2017 | tagged: phd · performer–audience dynamics

submission

…finally.

diary | 09 may 2017 | tagged: phd · research · qmat

accepted for ISPS2017

‘visualising performer–audience dynamics’ spoken paper accepted at ISPS 2017, the international symposium on performance science. this is doubly good, as i’ve long been keen to visit reykjavík and explore iceland.

diary | 13 apr 2017 | tagged: comedy lab · performer–audience dynamics · phd · qmat · research

robot comedy lab: journal paper

the robot stand-up work got a proper write-up. well, part of it got a proper write-up, but so it goes.

This paper demonstrates how humanoid robots can be used to probe the complex social signals that contribute to the experience of live performance. Using qualitative, ethnographic work as a starting point we can generate specific hypotheses about the use of social signals in performance and use a robot to operationalize and test them. […] Moreover, this paper provides insight into the nature of live performance. We showed that audiences have to be treated as heterogeneous, with individual responses differentiated in part by the interaction they are having with the performer. Equally, performances should be further understood in terms of these interactions. Successful performance manages the dynamics of these interactions to the performer’s- and audiences’-benefit.

pdf download

diary | 25 aug 2015 | tagged: comedy lab · phd · qmat · research

oriented-to test

need a hit-test for people orienting to others. akin to gaze, but the interest here is what it looks like you’re attending to. but what should that hit-test be? visualisation and parameter tweaking to the rescue…

diary | 03 feb 2015 | tagged: comedy lab · performer–audience dynamics · phd · qmat · research

through the eyes

with the visualiser established, it was trivial to attach the free view camera to the head pose node and boom!: first-person perspective. to be able to see through the eyes of anyone present is such a big thing.

diary | 13 jan 2015 | tagged: comedy lab · performer–audience dynamics · phd · qmat · research

robot comedy lab: workshop paper

minos gave a seminar on his engineering efforts for robot stand-up, we back-and-forthed on the wider framing of the work, and a bit of that is published here. his write-up.

workshop paper presented at humanoid robots and creativity, a workshop at humanoids 2014.

pdf download

diary | 18 nov 2014 | tagged: comedy lab · phd · qmat · research

rotation matrix ambiguities

the head pose arrows look like they’re pointing in the right direction… right? well, of course, it’s not that simple.

the dataset processing script vicon exporter applies an offset to the raw angle-axis fixture pose, to account for the hat not being straight. the quick and dirty way to get these offsets is to say at a certain time everybody is looking directly forward. that might have been ok if i’d thought to make it part of the experiment procedure, but i didn’t, and even if i had i’ve got my doubts. but we have a visualiser! it is interactive! it can be hacked to nudge things around!

except that the visualiser just points an arrow at a gaze vector, and that’s doesn’t give you a definitive orientation to nudge around. this opens up a can of worms where everything that could have thwarted it working, did.

“The interpretation of a rotation matrix can be subject to many ambiguities.”
http://en.wikipedia.org/wiki/Rotation_matrix#Ambiguities

hard-won code –

DATASET VISUALISER

// Now write MATLAB code to console which will generate correct offsets from this viewer's modelling with SceneKit
for (NSUInteger i = 0; i < [self.subjectNodes count]; ++i)
{
	// Vicon Exporter calculates gaze vector as
	// gaze = [1 0 0] * rm * subjectOffsets{subjectIndex};
	// rm = Rotation matrix from World to Mocap = Rwm
	// subjectOffsets = rotation matrix from Mocap to Offset (ie Gaze) = Rmo

	// In this viewer, we model a hierarchy of
	// Origin Node -> Audience Node -> Mocap Node -> Offset Node, rendered as axes.
	// The Mocap node is rotated with Rmw (ie. rm') to comply with reality.
	// Aha. This is because in this viewer we are rotating the coordinate space not a point as per exporter

	// By manually rotating the offset node so it's axes register with the head pose in video, we should be able to export a rotation matrix
	// We need to get Rmo as rotation of point
	// Rmo as rotation of point = Rom as rotation of coordinate space

	// In this viewer, we have
	// Note i. these are rotations of coordinate space
	// Note ii. we're doing this by taking 3x3 rotation matrix out of 4x4 translation matrix
	// [mocapNode worldTransform] = Rwm
	// [offsetNode transform] = Rmo
	// [offsetNode worldTransform] = Rwo

	// We want Rom as rotation of coordinate space
	// Therefore Offset = Rom = Rmo' = [offsetNode transform]'

	// CATransform3D is however transposed from rotation matrix in MATLAB.
	// Therefore Offset = [offsetNode transform]

	SCNNode* node = self.subjectNodes[i][@"node"];
	SCNNode* mocapNode = [node childNodeWithName:@"mocap" recursively:YES];
	SCNNode* offsetNode = [mocapNode childNodeWithName:@"axes" recursively:YES];

	// mocapNode has rotation animation applied to it. Use presentation node to get rendered position.
	mocapNode = [mocapNode presentationNode];

	CATransform3D Rom = [offsetNode transform];

	printf("offsets{%lu} = [%f, %f, %f; %f, %f, %f; %f, %f, %f];\n",
		   (unsigned long)i+1,
		   Rom.m11, Rom.m12, Rom.m13,
		   Rom.m21, Rom.m22, Rom.m23,
		   Rom.m31, Rom.m32, Rom.m33
		   );

	// BUT! For this to actually work, this requires Vicon Exporter to be
	// [1 0 0] * subjectOffsets{subjectIndex} * rm;
	// note matrix multiplication order

	// Isn't 3D maths fun.
	// "The interpretation of a rotation matrix can be subject to many ambiguities."
	// http://en.wikipedia.org/wiki/Rotation_matrix#Ambiguities
}

VICON EXPORTER

poseData = [];
for frame=1:stopAt
	poseline = [frameToTime(frame, dataStartTime, dataSampleRate)];
	frameData = reshape(data(frame,:), entriesPerSubject, []);
	for subjectIndex = 1:subjectCount

		%% POSITION
		position = frameData(4:6,subjectIndex)';

		%% ORIENTATION
		% Vicon V-File uses axis-angle represented in three datum, the axis is the xyz vector and the angle is the magnitude of the vector
		% [x y z, |xyz| ]
		ax = frameData(1:3,:);
		ax = [ax; sqrt(sum(ax'.^2,2))'];
		rotation = ax(:,subjectIndex)';

		%% ORIENTATION CORRECTED FOR OFF-AXIS ORIENTATION OF MARKER STRUCTURE
		rm = vrrotvec2mat(rotation);

		%% if generating offsets via calcOffset then use this
		% rotation = vrrotmat2vec(rm * offsets{subjectIndex});
		% gazeDirection = subjectForwards{subjectIndex} * rm * offsets{subjectIndex};

		%% if generating offsets via Comedy Lab Dataset Viewer then use this
		% rotation = vrrotmat2vec(offsets{subjectIndex} * rm); %actually, don't do this as it creates some axis-angle with imaginary components.
		gazeDirection = [1 0 0] * offsets{subjectIndex} * rm;

		poseline = [poseline position rotation gazeDirection];
	end
	poseData = [poseData; poseline];
end

diary | 19 aug 2014 | tagged: comedy lab · performer–audience dynamics · phd · qmat · research

writing up

if only writing up the phd was always like this. beautiful room, good friends, excellent facilitation by thinkingwriting.

diary | 12 aug 2014 | tagged: phd · qmat · research

visualising everything

visualising head pose, light state, laugh state, computer vision happiness, breathing belt. and, teh pretty. huzzah.

diary | 21 jun 2014 | tagged: comedy lab · performer–audience dynamics · phd · qmat · research

comedy lab » angel comedy

third gig: angel comedy. again, an established comedy club and again a different proposition. a nightly, free venue, known to be packed. wednesdays was newcomers night which, again, was somewhat appropriate.

what i remember most vividly has not to do with our role in it, but was rather the compère warming up the crowd after the interval. it was a masterclass in rallying a crowd into an audience (probably particularly warranted given the recruitment message of ‘free’ combined with inexperienced acts). i rue to this day not recording it.

diary | 04 jun 2014 | tagged: comedy lab · phd · qmat · research

comedy lab » gits and shiggles

the second gig of our tour investigating robo-standup in front of ‘real’ audiences: gits and shiggles at the half moon, putney. a regular night there, we were booked amongst established comedians for their third birthday special. was very happy to see the headline act was katherine ryan, whose attitude gets me every time.

shown previously was artie on-stage being introduced. he (it, really) has to be on stage throughout, so we needed to cover him up for a surprise reveal. aside from the many serious set-up issues, i’m pleased i managed to fashion the ‘?’ in a spare moment. to my eye, makes the difference.

artie has to be on stage throughout as we need to position him precisely in advance. that, and he can’t walk. the precise positioning is because we need to be able to point and gesture at audience members: short of having a full kinematic model of artie and the three dimensional position of each audience member identified, we manually set the articulations required to point and look at every audience seat within view, while noting where each audience seat appears in the computer vision’s view. the view is actually a superpower we grant to artie, the ability to have see from way above his head, and do that in the dark. we position a small near-infrared gig-e vision camera in the venue’s rigging along with a pair of discreet infra-red floodlights. this view is shown above, a frame grabbed during setup that has hung around since.

diary | 03 jun 2014 | tagged: comedy lab · phd · qmat · research

1

2

3