I think that is you done, so congrats!
and with that, it’s done. an academic book chapter submitted, tying together my art practice and the research it prompted. i’m proud of it, the writing has been a personal ordeal but the result is a defining statement on my position. i’m also proud of the form – it’s simultaneously a robust academic contribution grounded in empirical research and a highly personal account. quite radical, that.
diary | 17 nov 2025 | tagged: research · phd · live in live cinema · engaged audiences · writing
i’m a co-author of Audience Interaction: Approaches to researching the social dynamics of live audiences, a chapter in the Routledge Companion to Audiences and the Performing Arts. at some point, it got a top to bottom re-write from pat, my phd supervisor. the topic being pretty much that of my phd, witnessing this re-write was quite something. for all those hours of supervision discussing, probing, honing… this is the distilled version of what was in his head; squint, and it’s as if he got to write my thesis instead. i’m proud of what i wrote and i think it holds up, so this was wild, in a good way. like watching a grand master, able to appreciate every detail.
Live events are social encounters: people in a live audience do not just react to a work, they react to the people around them. Other people’s laughter, applause, coughing, fidgeting, in-breaths and silences all contribute to the experience of live events. Importantly, these behaviours are not just clues to people’s inner responses – they are public social signals that are actively interpreted by others. As a result, there are a number of connections between the organisation of large-scale audience interactions and small-scale social interactions like conversation. These connections provide a useful way of thinking about the dynamics of audience responses. It has implications for what responses we focus on, how we measure them and how we model them. It helps to explain how responses develop and propagate through an audience. It also changes our understanding of what influences people’s moment-by-moment experience of live events; performers and audience members alike.
diary | 06 oct 2022 | tagged: research · phd · engaged audiences · writing
Daniela Massiceti, Lida Theodorou, Luisa Zintgraf, Matthew Tobias Harris, Simone Stumpf, Cecily Morrison, Edward Cutrell, Katja Hoffmann
Object recognition has made great advances in the last decade, but predominately still relies on many high-quality training examples per object category. In contrast, learning new objects from only a few examples could enable many impactful applications from robotics to user personalization. Most few-shot learning research, however, has been driven by benchmark datasets that lack the high variation that these applications will face when deployed in the real-world. To close this gap, we present the ORBIT dataset, grounded in a real-world application of teachable object recognizers for people who are blind/low vision. The full dataset contains 4,733 videos of 588 objects recorded by 97 people who are blind/low-vision on their mobile phones, and we mark a subset of 3,822 videos of 486 objects collected by 77 collectors as the benchmark dataset. We propose a user-centric evaluation protocol to evaluate machine learning models for a teachable object recognition task on this benchmark dataset. The code for loading the dataset, computing all benchmark metrics, and running the baseline models is available at https://github.com/microsoft/ORBIT-Dataset
Finally, the published dataset: ORBIT Dataset
diary | 31 mar 2021 | tagged: orbit · research
To complement the release of the dataset, here’s the infrastructure used to create it – my code open sourced for future dataset collection projects to build on our work. And as our work shows, machine learning needs more inclusive datasets.
https://github.com/orbit-a11y/ORBIT-Camera
https://github.com/orbit-a11y/orbit_data
Data captured, collated, archived, and now sent to the vault. My role, done.
diary | 20 feb 2021 | tagged: orbit · research
Title:
The liveness of live events, and how we might design for thatAbstract:
This talk will draw a line through Toby’s practice and research to argue for an interactional account of liveness, and where that might lead as we figure out new relationships between the digital and face-to-face; it’ll have to be a remote talk, after all. Topics will include why I really should have just read my email on stage, instrumenting auditoriums, teaching a humanoid robot stagecraft, visualising performer–audience dynamics, and why the HCI and privacy considerations of data-backed immersive theatre might be the crucible for our near future every day.I’m interested in the MRL’s take on any or all of these topics, and am open to collaboration. More on my website at http://tobyz.net
Biography:
Toby’s practice spans art, design and engineering, “fascinated by the liveness of live events, and how we can design for that” (http://tobyz.net). He performs worldwide as part of the renown audio-visual collective D-Fuse (http://dfuse.com). He develops and sells hardware, software and design services for events (http://sparklive.net). His practice led to a PhD on liveness and performer–audience–audience interaction. He’s currently focussed on live data.
…and happily, the talk was super-well received. quite the relief, as it’s a storied lab (e.g. long-time blast theory collaborators) with an embarassment of ‘best of CHI’ papers.
Data collection phase one comes to an end: test and training imagery for 545 things, in the form of 4568 videos. Having built the system with barely a page of test data, it never gets old seeing the paginator having to truncate itself.
Innovations in technology are changing every part of the performance landscape […] We are interested in the concept of liveness and togetherness in the context of these changes in technology.
— https://bristolbathcreative.org/pathfinders/expanded-performance
crikey! i have something to say on the matter. no need to rehash that here, though. instead, see this artefact from the workshop. happily, some participants picked one of my contributions from the 100 questions session, and spent some time on it: “how do we move tech away from spectacle to… liveness?”
it took the calm of a day or two later to really realise what i was looking at. that was the question that, more or less, animated the start of my PhD. i spent years chasing that thread, the tech falling away to explore the underlying phenomena through study of human interaction, audiences and an appropriation of experimental psyschology; i have a fine-grained idea of where that question leads. and yet, here is a completely different take, from two completely different minds.
i bring this up because i don’t know how to reconcile the potential of that workshop room with the fixed number of fellowships they are eventually going to fund (and the necessarily constrained research that will result). but that session felt like the start of something good, the above a glimpse of a better way.
based on the drawing interactions work, we were asked to run a day-long workshop for the national center for research methods. happily, “drawing, multimodality and interaction analytics” sold out early, and went well. i even got some sketches out of it, courtesy of sophie.
happy punters aside, this spurred a day-or-so sprint of app development. i wanted to show some live demos using the sources and drawing techniques that would be presented during the sessions. and that needed a document based app. and that needed a way to persist the timestamped drawings, and so on. which it now all does. and with that, the app is no longer just a prototype. it has practical value to others.
diary | 28 nov 2019 | tagged: drawing interactions · research · code
if you are blind, a modern phone can be life-changing. apps like taptapsee connect the phone’s camera to a sighted person so someone can tell you what’s in front of you. though there might be a delay matchmaking that person, and you might not feel entirely comfortable sharing your life with a stranger. could the phone recognise things by itself?
the answer is yes and no, and it’s what i’m working on at city, unversity of london for a while. yes, in that machine learning algorithms have been trained on image datasets and phones running them really can pick things out of the camera feed. the no comes in practice, as the things being picked out don’t seem to be what’s salient to the visually impaired. so, amongst other things, i’m building an iOS app so the blind can show us what’s important to them. a camera app, for the blind.
diary | 21 oct 2019 | tagged: orbit · research
an email comes in from a performance studies phd candidate asking if they could watch the whole robot routine from comedy lab: human vs. robot. damn right. i’d love to see someone write about that performance as a performance.
but, better than that staging and its weird audiences (given the advertised title, robo-fetishists and journalists?) there is comedy lab #4: on tour, unannounced. the premise: robot stand-up, to unsuspecting audiences, at established comedy nights. that came a year later with the opportunity to use another robothespian (thanks oxford brookes!). it addressed the ecological validity issues, and should simply be more fun to watch.
for on tour, unannounced we kept the performance the same – or rather, each performance used the same audience responsive system to tailor the delivery in realtime. there’s a surprising paucity in the literature about how audiences respond differently to the same production; the idea was this should be interesting data. so i’ve taken the opportunity to extract from the data set the camera footage of the stage from each night of the tour. and now that is public, at the links below.
the alternative comedy memorial society
gits and shiggles
angel comedy
the robot comedy lab experiments form chapter 4 of my phd thesis ‘liveness: an interactional account’
Four: Experimenting with performance
The literature reviewed in chapter three also motivates an experimental programme. Chapter four presents the first, establishing Comedy Lab. A live performance experiment is staged that tests audience responses to a robot performer’s gaze and gesture. This chapter provides the first direct evidence of individual performer–audience dynamics within an audience, and establishes the viability of live performance experiments.
there are currently two published papers –
robot stand-up: engineering a comic performance
http://tobyz.net/diary/2014/11/robot-standup-workshop-paper
robot comedy lab: experimenting with the dynamics of live performance
http://tobyz.net/diary/2015/08/robot-standup-journal-paper
and finally, on ‘there is a surprising paucity…’, i’d recommend starting with gardair’s mention of mervant-roux.
diary | 03 may 2019 | tagged: comedy lab · phd · qmat · research
i had a note squirrelled away that there were human-machine co-composition results still buried in the folkrnn.org write-up stats. and to me, that’s the heart of it.
with positive noises from journal paper reviewers in the air, i wrangled the time to go digging.
the following is the co-composition analysis the webapp’s stats management command now also produces.
Refining the folkrnn.org session data above, we can analyze only those tunes which are in some way a tweak of the one that came before. This iterative process of human-directed tweaks of the machine-generated tunes demonstrates co-composition using the folkrnn.org system. In numbers –
Of the 24657 tunes generated on folkrnn.org, 14088 keep the generation parameters from the previous tune while changing one or more (57%).
This happened within 4007 ‘iterative’ sequences of, on average 6 tune generations (mean: 5.9, stddev: 8.7).
The frequency of the generate parameters used now becomes:
key: 0.28, meter: 0.24, model: 0.093, seed locked: 0.34, start abc is excerpt: 0.02, start abc: 0.2, temp: 0.39
One feature now possible to expose is whether the user has identified a salient phrase in the prior tune, and has primed the generation of the new tune with this phrase. This is the strongest metric of co-composition available on folkrnn.org. This is reported above as ‘start_abc is excerpt’, tested for phrases comprising five characters or more (e.g. five notes, or fewer with phrasing), and as per other generation metrics reported here, not counting subsequent generations with that metric unchanged. This happened 283 times (2%)
Further evidence of human-machine co-composition can be seen on themachinefolksession.org, where 239 of the ‘iterative’ folkrnn.org tunes were archived. Using the tune saliency metric used by the themachinefolksession.org homepage, the most noteworthy of these tunes is ‘Green Electrodes’. This was generated in the key C Dorian (//folkrnn.org/tune/5139), and as archived (https://themachinefolksession.org/tune/294) the user has manually added a variation set in the key E Dorian. This shows a limitation of folkrnn.org, that all tunes are generated in a variant of C (a consequence of an optimisation made while training the RNN on the corpus of existing tunes), and shows that the human editing features of themachinefolksession.org have been used by users to work around such a limitation. Also, while not co-composition per-se, the act of the user naming the machine generated tune shows it has some value to them.
Direct evidence of the user’s intent can be seen in ‘Rounding Derry’ (https://themachinefolksession.org/tune/587). The user generated tune ‘FolkRNN Tune №24807’ on a fresh load of folkrnn.org, i.e. default parameters, randomised seed. The user played this tune twice, and then selected the musical phrase ‘C2EG ACEG|CGEG FDB,G,’ and set this for the start_abc generation parameter. The user generated the next iteration, played it back, and archived on themachinefolksession.org with a title of their creation. There, the user writes –
‘Generated from a pleasant 2 measure section of a random sequence, I liked this particularly because of the first 4 bars and then the jump to the 10th interval key center(?) in the second section. Also my first contribution!’
diary | 02 may 2019 | tagged: machine folk · research · code
Drawing as transcription: how do graphical techniques inform interaction analysis?
Drawing as a form of analytical inscription can provide researchers with highly flexible methods for exploring embodied interaction. Graphical techniques can combine spatial layouts, trajectories of action and anatomical detail, as well as rich descriptions of movement and temporal effects. This paper introduces some of the possibilities and challenges of adapting graphical techniques from life drawing and still life for interaction research. We demonstrate how many of these techniques are used in interaction research by illustrating the postural configurations and movements of participants in a ballet class. We then discuss a prototype software tool that is being developed to support interaction analysis specifically in the context of a collaborative data analysis session.
Albert, S., Heath, C., Skach, S., Harris, M., Miller, M., & Healey, P. (2019). Drawing as transcription: how do graphical techniques inform interaction analysis? Social Interaction. Video-Based Studies of Human Sociality, 2(1). https://doi.org/10.7146/si.v2i1.113145
Open Access / Creative Commons BY-NC-ND
folkrnn.org can now generate tunes in a swedish folk idiom. bob having moved to KTH in sweden, had got some new students to create a folkrnn model trained on a corpus of swedish folk tunes. and herein lies a tale of how things are never as simple as they seem.
the tale goes something like this: here we have a model that already works with the command-line version of folkrnn. and the webapp folkrnn.org parses models and automatically configures itself. a simple drop-in, right?
first of all, this model is sufficiently different that the defaults for the meter, key and tempo are no longer appropriate. so a per-model defaults system was needed.
then, those meter, key composition parameters are differently formatted in this corpus, which pretty much broke everything. piecemeal hacks weren’t cutting it, so a sane system was needed that standardised on one format and sanely bridged to the raw tokens of each model.
after the satisfaction of seeing it working, bob noticed that the generated tunes were of poor quality. when a user of folkrnn.org generates a tune with a previous model, setting the meter and key to the first two tokens is exactly what the model expects, and it can then fill in the rest drawing from the countless examples of that combination found in the corpus. but with this new model, or rather the corpus it was trained on, a new parameter precedes these two. so the mechanics that kicks off each tune needed to now cope with an extra, optional term.
so expose this value in the composition panel? that seems undesirable, as this parameter is effectively a technical option subsumed by the musical choice of meter. and manually choosing it doesn’t guarantee you’re choosing a combination found in the corpus, so the generated tunes are still mostly of poor quality.
at this point, one might think that exactly what RNNs do is choose appropriate values. but it’s not that simple, as the RNN encounters this preceeding value first, before the meter value set by the user. it can choose an appropriate meter from the unit-note-length, but not the other way round. so a thousand runs of the RNN and a resulting frequency table later, folkrnn.org is now wired to generate an appropriate pairing akin to the RNN running backwards. those thousand runs also showed that only a subset of the meters and keys found in the corpus are used to start the tune, so now the compose panel only shows those, which makes for a much less daunting drop-down, and fewer misses for generated tune quality.
unit-note-length is now effectively a hidden variable, which does the right thing… providing you don’t want to iteratively refine a composition, as it may vary from tune generated to tune generated. rather than exposing the parameter after all, and then have to implement pinning in as per the seed parameter’s control, a better idea was had: make the initial ABC field also handle this header part of the tune. so rather than just copying-and-pasting-in snippets of a tune, you could paste in the tune from the start, including this unit note length header. this is neat because as well as providing the advanced feature of being able to specify the unit note lenth value, it makes the UI work better for naïve users: why couldn’t you copy and paste in a whole tune before?
as per the theme there, implementing this wasn’t just a neat few lines of back-end python, as now the interface code that is loaded into the browser needs to be able to parse out and verify these header lines, and so on.
Swedish Model
https://github.com/tobyspark/folk-rnn-webapp/pull/133
Feature: composer models have defaults for meter, mode and tempo
https://github.com/tobyspark/folk-rnn-webapp/pull/134
Feature: set unit note length within prime tokens, for models with L tokens
https://github.com/tobyspark/folk-rnn-webapp/pull/136
Feature: Headers in start ABC
https://github.com/tobyspark/folk-rnn-webapp/pull/138
diary | 15 jan 2019 | tagged: machine folk · research · code
for any given tune, how much activity surrounded it?
for any given session, what happened?
what are the usage trends of folkrnn.org and of themachinefolksession.org?
to answer these kinds of questions, enter stats, a django management command for processing the use data of composer and archiver apps for insight / write-up in academic papers.
i used this to write the following, with input from bob and oded. it will surely be edited further for publication, but this is as it stands right now.
During the first 235 days of activity at folkrnn.org, 24562 tunes were generated by – our heuristics suggest – 5700 users. Activity for the first 18 weeks averages a median of 155 tunes weekly. In the subsequent 15 weeks to the time of writing, overall use increased, with a median of 665 tunes generated weekly. This period also features usage spikes. One week, correlating to an interview in Swedish media, shows 2.7x the median tunes generated. The largest, correlating to a mention in German media, shows an 18.4x increase. These results show making our tool available to users of the web has translated into actual use, and that use is increasing. Further, media attention brings increased use, and this use is similarly engaged, judged by similar patterns of downloading MIDI and archiving tunes to themachinefolksession.org.
Of the fields available for users to influence the generation process on folkrnn.org, the temperature was used more often then the others (key, meter, initial ABC, and random seed). Perhaps this is because changing temperature results in more obviously dramatic changes in the generated material. Increasing the temperature from 1 to 2 will often yield tunes that do not sound traditional at all. If changes were made to the generate parameters, the frequency of the resulting tune being played, downloaded or archived increased from 0.78 to 0.87.
Over the same period since launch, themachinefolksession.org has seen tunes 551 contributed. Of these tunes, 82 have had further iterations contributed in the form of ‘settings’; the site currently hosts 92 settings in total. 69 tunes have live recordings contributed; the site currently hosts 64 recordings in total (a single performance may encompass many tunes). These results show around 100 concrete examples of machine-human co-creation have been documented.
Of the 551 contributed tunes, 406 were generated on, and archived from, folkrnn.org. Of these entirely machine-generated tunes, 32 have had human edits contributed; themachinefolksession.org currently hosts 37 settings of folkrnn generated tunes in total. These examples in particular attributable human iteration of, or inspiration by, machine produced scores.
Further value of machine produced scores can be seen by the 30 registered users who have selected 136 tunes or settings as being noteworthy enough to add to their tunebooks. Per the algorithm used by the home page of themachinefolksession.org to surface ‘interesting’ tunes, “Why are you and your 5,599,881 parameters so hard to understand?” is the most, with 4 settings and 5 recordings.
While these results are encouraging, most content-affecting activity on themachinefolksession.org has been from the administrators; co-author Sturm accounts for 70% of such activity. To motivate the use of the websites, we are experimenting with e.g. ‘tune of the month’, see above, and have organised a composition competition.
The composition competition was open to everyone but targeted primarily at music students. Submission included both a score for a set ensemble and an accompanying text describing how the composer used a folkrnn model in the composition of the piece. The judging panel - the first author was joined by Profs. Elaine Chew and Sageev Oore - considered the musical quality of the piece as well as the creative use of the model. The winning piece Gwyl Werin by Derri Lewis was performed by the New Music Players at a concert organised in partnership with the 2018 O’Reilly AI Conference in London. Lewis said he didn’t want to be ‘too picky’ about the tunes, rather he selected a tune to work from after only a few trails. He describes using the tune as a tone row and generating both harmonic, melodic and motivic material out of it. Though the tune itself, as generated by the system, does not appear directly in the piece.
diary | 11 jan 2019 | tagged: machine folk · research · code
having been selected for the interactive machine-learning for music exhibition at the 19th international society for music information retreival conference, the time had come. nice to see a photo back from set-up, with the poster (PDF) commanding attention in the room.
diary | 23 sep 2018 | tagged: machine folk · research
the folk-rnn webapp was selected for the 19th international society for music information retreival conference, as part of their interactive machine-learning for music exhibition. so poster time! nice to not have to futz around with html+css, instead just draw directly… i’m pretty happy with how it turned out (PDF).
diary | 12 sep 2018 | tagged: machine folk · research
the community site. straight-up django (“the web framework for perfectionists with deadlines”), but there’s a lot going on.
Tweak: Search includes author names
https://github.com/tobyspark/folk-rnn-webapp/pull/109
Feature: Tune of the month
https://github.com/tobyspark/folk-rnn-webapp/pull/92
Tweaks: Tempo, Attributions, Add settings, ABC
https://github.com/tobyspark/folk-rnn-webapp/pull/90
Feature: Tunes page has order by added or popularity
https://github.com/tobyspark/folk-rnn-webapp/pull/85
Tweaks and fixes
https://github.com/tobyspark/folk-rnn-webapp/pull/73
Feature: Surface interesting tunes
https://github.com/tobyspark/folk-rnn-webapp/pull/64
Feature: Search
https://github.com/tobyspark/folk-rnn-webapp/pull/58
Feature: Tunebook
https://github.com/tobyspark/folk-rnn-webapp/pull/57
Feature: User files upload and serve configuration
https://github.com/tobyspark/folk-rnn-webapp/pull/32
Feature: Archiver mk.ii
https://github.com/tobyspark/folk-rnn-webapp/pull/25
https://github.com/tobyspark/folk-rnn-webapp/pull/37
https://github.com/tobyspark/folk-rnn-webapp/pull/50
diary | 29 jul 2018 | tagged: machine folk · research · code
We demonstrate 1) a web-based implementation of a gen- erative machine learning model trained on transcriptions of folk music from Ireland and the UK (http://folkrnn.org, live since March 2018); 2) an online repository of work created by machines (https://themachinefolksession.org/, live since June 2018). These two websites provides a way for the public to engage with some of the outcomes of our research investigating the application of machine learning to music practice, as well as the evaluation of machine learning applied in such contexts. Our machine learning model is built around a text-based vocabulary, which provides a very compact but expressive representation of melody-focused music. The specific kind of model we use consists of three hidden layers of long short-term memory (LSTM) units. We trained this model on over 23,000 transcriptions crowd-sourced from an online community devoted to these kinds of folk music. Several compositions created with our application have been performed so far, and recorded and posted online. We are also organising a composition competition using our web-based implementation, the winning piece of which will be performed at the 2018 O’Reilly AI conference in London in October.
Matthew Tobias Harris
Queen Mary University of London
London E1 4NS, UKBob L. Sturm
Royal Institute of Technology KTH
Lindstedtsvägen 24, SE-100 44 Stockholm, SwedenOded Ben-Tal
Kingston University
Kingston Hill, Kingston upon Thames, Surrey KT2 7LB, UK
Submitted to the interactive machine-learning for music (IML4M) @exhibition at the 19th international society for music information retreival conference. PDF
diary | 18 jul 2018 | tagged: machine folk · research
The conversational rollercoaster: Conversation analysis and the public science of talk
How does talk work, and can we engage the public in a dialogue about the scientific study of talk? This article presents a history, critical evaluation and empirical illustration of the public science of talk. We chart the public ethos of conversation analysis that treats talk as an inherently public phenomenon and its transcribed recordings as public data. We examine the inherent contradictions that conversation analysis is simultaneously obscure yet highly cited; it studies an object that people understand intuitively, yet routinely produces counter-intuitive findings about talk. We describe a novel methodology for engaging the public in a science exhibition event and show how our ‘conversational rollercoaster’ used live recording, transcription and public-led analysis to address the challenge of demonstrating how talk can become an informative object of scientific research. We conclude by encouraging researchers not only to engage in a public dialogue but also to find ways to actively engage people in taking a scientific approach to talk as a pervasive, structural feature of their everyday lives.
Albert, S., Albury, C., Alexander, M., Harris, M. T., Hofstetter, E., Holmes, E. J. B., & Stokoe, E. (2018). The conversational rollercoaster: Conversation analysis and the public science of talk. Discourse Studies, 20(3), 397–424. https://doi.org/10.1177/1461445618754571
PDF available from Loughborough University Institutional Repository
About
Projects
Diary
