tagged: live illustration

co-present collaboration over, it was time to address remote performance. ideas explored, we started on the basics. which proved quite resilient to ‘just working’.

the basic premise is all performers need to stream to each other, so they can collaborate. and these streams need to also be put together into some kind of performance stream. but not side-by-side as per every video-chat service, but on top of each other: so illustrators can work in the same space, adapting each other’s work, or e.g. an illustrator incorporate imagery of the music making into the work. streams as photoshop layers.

while i hold out hope that some kind of peer-to-peer streaming solution will arrive, the current state of things just wasn’t there. some kind of central server, if only to coordinate, seems necessary given finicky home networks. while i can do that, it was beyond the remit of skilling-up others. and still wouldn’t get us to something turnkey-like.

in my research, i did find a service that unlocked remote performance for us: stream voodoo. it’s a video-chat web app that does the basics like any other service,
and they’re oriented to live performance, so focus on low latency and full audio. but the killer feature is ‘magic links’, which make the individual feeds in that main video chat screen available on their own, as web pages. this means those links can be used as web browser sources in apps like OBS Studio and VDMX.

the screenshot above shows a test session proof of concept with sound artist xavier velastín, where we had

• cam, xavier, and myself able to see and hear each other in a video chat
• xavier’s camera feed and cam’s illustration feed being processed and composited together in VDMX
• xavier’s audio being sent to OBS from VDMX, via Rogue Amoeba’s Loopback
• VDMX outputting to OBS via Syphon
• OBS broadcasting to twitch

overall though, the process was quite frustrating. xavier’s laptop was having issues, and stream voodoo often failed probably due to poor free-trial UX rather than the streaming itself. note to former self: if you’re facilitating something like this for somebody else, don’t wait for them to see the value in paying, just pay up, have it work, and then talk about value.

diary | 11 oct 2021 | tagged: live illustration

i think live illustration is interesting and surprisingly underexplored, so it’s been great to see friend camille aubry’s scribing practice taking off. we spent some time a while back articulating the potential of her practice, since then she’s been accepted into the pervasive media studio, and now this: live illustration expanded, a DYCP funded research and development project. broadly in two parts: co-present? and remote?

first came a collaboration around co-present performance, with cam exploring projection mapping with alex from limbic cinema and performance with poet laurence hoo. covid unsurprisingly thwarted plans of an in-situ audience, a shame given illustrators’ ability to respond in the moment really opens up the scope for what a live performance can mean (a formative experience: the dragon that wasn’t a crocodile). nonetheless, it was wonderful to simply be in a room with people working towards a performance, and help film its realisation.

second, came a collaboration around remote performance…

diary | 30 jul 2021 | tagged: live illustration · pervasive media studio

prototype one was feature complete for our base platform to experiment with… except that it slowed to a crawl in it’s first real-world session. prototype two has the engineering work to maintain a snappy drawing experience and UI. this hasn’t made the drawing itself performant, rather it has made the engine responsive regardless of the drawing performance. it keeps the act of drawing syncronous along with the rest of UI, while pushing the updating of the whole canvas in the background.

prototype one looked something like this, the standard fill-in-the-placeholder function to draw into the view –

	override func draw(_ rect: CGRect)

the prototype two equivalent looks like this. it’s… a lot more complicated.

	// Drawing canvas for the live stroke, which changes during stroking given prediction etc.
	private var activeStrokeLayer: CALayer!
	private var drawingLayerDelegate: DrawingLayerDelegate!
	    
	// Drawing canvas for the completed strokes; async, accumulating, and other sophistry
	private var completedStrokesLayer: CALayer!
	private var layerDelegate: LayerDelegate!
	private var completedStrokesQueue: DispatchQueue!
	private var completedStrokesBlocks: [DispatchWorkItem]!
	private var completedStrokesContext: CGContext!
	
	// Draw methods to abstract what’s going on behind the scenes
	func drawCompletedStrokes(onlyAdd: [Stroke]? = nil)
	func drawActiveStroke()
	
	// When receiving a new set of strokes, if any stroke from the old set is missing from the new, we need to completely redraw. Conversely, new strokes can be drawn on top of the existing canvas.
	func diff(_ oldElements: [Stroke], _ newElements: [Stroke]) -> diffResult

oh, and the timestamps on the audio timeline are nicely formatted now.

diary | 12 aug 2019 | tagged: spark scribe · live illustration · code · pervasive media studio

friend camille aubry was having thoughts about how her scribing practice could develop. i got an email that said “digital scribe?”; as she suspected, that’s something i have thoughts about too.

live illustration is something i’ve long played with – e.g. the concept and tooling for kinetxt – and in drawing interactions i have a recent iPad code-base for drawing-and-media. i’d been scheming that something in this vein would be interesting for future engaging-audiences work.

taking camille’s question as a prompt, i started coding. a base prototype was needed. first, to verify the basics: can camille draw with an iPad and its pencil? is the 10.5” size i have too limited? can my drawing code compete with a single marker pen and paper? with that established, we’d then have a platform to iteratively develop and test ideas.

so: spark scribe, an iOS app for illustrators documenting live events.

the core idea i have for the app is to combine the act of drawing with a dictaphone. by timestamping the strokes to the audio recording, the context for what is being drawn can always be returned to. plus, taking the user interface approach of drawing interactions, fluid and artful interactions should be possible.

despite the drawing interactions codebase, this is entirely new.

• timeline implemented with UIScrollView and CATiledLayer (the bespoke approach in drawing interactions was quick to code and effective in use, but this approach gives e.g. the inertial scrolling an iOS user would expect)
• persistence using native Swift serialisation and GRDB (‘because one sometimes enjoys a sharp tool’; Core Data didn’t work out)
• drawing code based on the newer apple example Leveraging Touch Input for Drawing Apps (it’s unoptimised but a better foundation than the prior example used in drawing interactions)

prototype one was used for the pervasive media studio’s lunchtime talk. it soon became clear that the drawing code needed work as the app slowed to a crawl. this can’t have made drawing with it easy for camille, let alone it being a new tool. the drawing example code being ‘unoptimsed’ refers to a few things, but mostly that it redraws every stroke, on every change. this was fine to verify the look, and tested fine in sessions of a few minutes at my desk, where the total number of strokes stays low. but is now proven not fine for the quantities of strokes that start to build up scribing a real talk. now i knew this would have to be addressed at some point, so it’s just brought that engineering task from the near-future to now. plus, thanks to this scribing session, i have real-world data i can load-test the app with.

Features: Prototype One
- Drawing engine
	- A3 sized canvas with pan and zoom
	- Pen tool with undo/redo
- Audio engine
	- Records, plays back talk with timeline
	- On playback, scrolling timeline displays illustration as drawn at that point in time
- Persistence
	- Persists illustration + audio
	- Imports and exports
		- UTI `net.sparklive.scribe.archive`
		- extension `.sparkscribe` 
		- format is a package wrapping
			- `strokes.json`
			- `audio.m4a`

diary | 26 jul 2019 | tagged: spark scribe · live illustration · code · pervasive media studio

that was the positive, here’s the negative. well, it was an outdoor performance, and it rained. which sucks, and its a shame that because of rain cover the audience couldn’t crowd around us performing it, seeing the creative hub and being able to join in with our text clients and general heckling to do this idea or that. but that is life.

what i feel is the real shame, is that despite the promise seen in rehearsal the performance didn’t pick up on the storytelling thread, which is perhaps one of the consequences of having to start earlier. this was to allow the other (truly open-air) act the chance to perform once the rain had passed, and it meant we started in sudden chaos rather than well briefed order, and with me dealing with an unpaired wii-mote that refused to play nice.

similarly because of the earlier start, we didn’t get a slot to properly tweak the canvas to the wall either, so the text was too small for the amount of relief texture we were projecting onto, and our texts weren’t exactly aligned with the wall edge… and so on, its easy to get hyper-critical about these things when you’ve so much time invested in it. i should say that the tech backing was fantastic - shout out to fanny - with the projectors being a dream to use, so bright, crisp and colour accurate.

however, the image quality speaks for itself, so i should stop whining and wait to see it objectively through the footage shot by the mapping documentation team.

diary | 14 may 2009 | tagged: *spark · vj · kinetxt · mapping · name · novak · live cinema · live illustration

after three days of prep, the performance. to quote the internet:

“It is a beautiful project and one that brings together three local illustrators and a poet for the festival performance. Despite some last minute technical difficulties - including rain forcing the start time forward by almost two hours, the show delivers with some moments of true beauty and the six busily active artists and live soundtrack make for an absorbing spectacle.” - lucy benson

lucy wrote that for friend sean healy’s skynoise blog and publication back in oz. so not exactly impartial, but on the flip side that should be in print!

diary | 14 may 2009 | tagged: *spark · vj · kinetxt · mapping · name · novak · live cinema · live illustration

from rehearsal to the performance proper. one of the challenges for us of projecting onto a building rather than a cinematic screen is that you don’t have the clear frame for your ‘canvas’, rather you’re just lighting up bits of a facade: type in the void. this gives a problem for making an art-pad the backdrop of your canvas: you don’t want a bright white rectangle somewhere on your building, you want the lit pen-strokes floating in the wider surface. so here we are for one part of the performance proper with our illustrators experimenting with black paper and tipp-ex (correction fluid) type pens.

diary | 14 may 2009 | tagged: *spark · vj · kinetxt · mapping · live illustration

…and now having used it in anger, here we have

• Added bounds feature, to give you all the sizing information you need to block out your calligraphy renderers.
• Fixed a crashing bug triggered by sending a clear all lines signal mid-stroke
• Added an advanced example derived from KineTXT development. Use space to send chunks of calligraphy to the screen, as if you were writing on a horizontal scroll.

diary | 11 may 2009 | tagged: quartz composer · live illustration · vj · code · mac os · kinetxt · release | downloads: SPK-Calligraphy-v1.2.zip

…and here is the bugfix release.

• fixed purge last object exception
• removed unused boilerplate methods
• added zPos to animator
• ordered ports (arrange in @dynamic line)
• fixed x,y mis-patch in sample qtz file

diary | 07 may 2009 | tagged: *spark · quartz composer · live illustration · vj · code · mac os · release | downloads: SPK-Calligraphy-v1.1.zip

a little sneak peek of a quartz composer plug-in in development: spk-calligraphy, a set of patches for recording and playing back 2d strokes. the basic patch is equivalent to the kineme GL line structure patch, but draws the line as if it were a chisel nib at 45° and with a flow fade-out. the other two are what is going to enable a big part of the next kinetxt development: handwriting to go alongside the rendered text.

diary | 27 mar 2009 | tagged: *spark · quartz composer · live illustration · vj · code · mac os · kinetxt

the creative hub. props to all involved: you can see the energy.

diary | 30 oct 2008 | tagged: *spark · vj · kinetxt · imageradio · live cinema · live illustration

been dragged into facebook to get a peek at the photos, digital politics aside, it’s certainly good when you see the festival director write “this was such an amazing event!”. huzzah!

diary | 11 mar 2008 | tagged: *spark · vj · kinetxt · name · novak · live cinema · live illustration

heroic all-day performance by ilan katin, based around live drawing. magic moments with with kids in the audience shouting out their suggestions and guessing the form taking shape, ilan writing and rubbing out responses… this dragon wasn’t a crocodile, but you had to be there.

diary | 02 may 2007 | tagged: vj · live illustration · mapping