This is a Max patch / LED tape project I developed for the show Machine Learning Hamlet; or, Text Like Daggers with Meta-Phys Ed at the Brick in July 2021.
Director: Jesse Freedman
Performed by: Sophie Zmorrod Laruelle
Video Design: Lacey Erb
Costume Design: Karen Boyer
Lighting Design: Elizabeth M Stewart
Scenic Design: Kyu Shin
Sound Design: Eamon Goodman
Stage Management: Lenyn Hernandez
Prod. Management: Kelly Lamanna
Intern: Nam Pham
Concept
Machine Learning Hamlet is Hamlet as performed by a super powerful AI, who is playing Hamlet and making decisions as one would while playing a video game.
As the computer performs its version of Hamlet, there is one audience member on stage who is the test-subject-audience. The computer is observing the audience members reactions, to learn about human emotion and response to the play.
To bring this idea into the world of design, the team selected moments of the play in which the computer is learning a lot - namely, in the soliloquies and final sword fight. In these moments of learning, the computer tracks the audience members face, and the LED tape installation - a visual representation of a neural network, leading back to the screen on which we see the AI - activates. When in reality the LED tape is changing based on motion-data, the display to the audience(s) indicates that we are tracking the on-stage audience members face, and that the neural network above is learning from that facial tracking. Much like a personal computer's fan may get loud when it is working very hard, the LED tape activates and changes rapidly.
Photos of the performances are on their way, but in the meantime, you can check out the livestream recordings here:
Patch description
Max is motion tracking 2 video feeds, one of the performer and one of an onstage audience member. That motion tracking data is used to send HSL data to two swatch objects, which output R G B values as numbers from 0 - 255. This is then sent in 3 directions - to LED tape via an Enttec DMX USB Pro, to Isadora over the network, and to a compute running Ableton via the network.
This way, the video and sound designers may use the same numerical data that Max is, to create their own effects. Video is discussing exploring color effects that match what Max is doing to the LED tape.
Max Patch
This is the presenter view of the patch for easy show operation. The "cue button" pauses the motion tracking and sets the LED tape to a neutral white. Turning the motion tracking back on allows the LEDs to resume color changing. This is done via a series of gate switches.
Video In
Lacey Erb, the video designer, and I needed to find a way to both share 2 video feeds. Max is running on a Mac, and Isadora is on a PC, so Syphon or Spout wouldn't quite fit the bill. The max NDI objects turned an error - it turns out they call to a library on the NDI website that sometimes goes offline for..indeterminate amounts of time. It's a known issue mentioned on the github.
Here's a video showing what we figured out:
Essentially, the Max computer is plugged into 1 camera, which is then jit.grabbed into max as well as livestreamed out over obs.ninja, where Isadora watches and pulls that video in.
The other camera is streamed to obs.ninja via the Isadora computer, and we use Syphoner to watch the stream and pull the feed into Max.
With that working, the next step was meeting in person to test Isadora receiving the motion tracking data. It worked! I did not document this well.
LED tape in process
Because we intend to use a lot of LED tape, we need a lot of power. We are using a 32 channel decoder for RGBW tape, and two PSUs that each provide 2 channels of power to the decoder.
For testing purposes, I did not uncoil the LED tape. This is generally not recommended due to heat issues.
I set up 2 sets of LED tape and tested them, with some confusion along the way, using my Pocket Console. The confusion turned out to all be caused not by an issue with the decoder or my wiring, but because the pocket console's patch had been altered to no longer be 1:1. Whoops! Once that was corrected, it was a good way to check the LED tape set up was correct before incorporating Max.
LED tape tests with Max
These are my video tests of the motion tracking with LED tape at home, before loading in to the theatre. I wanted to ensure everything was working before bringing it into the space.
This is the patch as of July 18, 2021, before we went into tech:
Tech
Once in tech, the patch worked mostly as planned - we did experiment with other mathematical ways to play with the numbers being sent via DMX, though ultimately landed back where we had started.
There was a brief foray into a few "cued" moments, with the LED tape remaining a static, pre-selected color, though ultimately the decision was made to keep the data stream active and the color changes happening live.
One major change was the addition of a flash of white every time the LED tape was activated - a way to signal its entrance and pull the viewers immediately into the world of the soliloquy. This was also a means of cleaning up the transition - the data stream is on or off, and the LED tape does not have a controllable intensity parameter, instead dimming based on the RGBW parameters individually. Activating the patch without the flash of white meant that random colors - or no color at all - would pop on when we started tracking. By bumping the white channels to 255 and then fading them to 0 over a few seconds, we were able to ease into the color changing while still boldly announcing the presence of the tracking.
Future Uses
I would love to be able to use this patch again, for this particular show.
If that does not happen, however, there are many potential ways to use LED tape and Max. Perhaps there is a version where actual facial tracking data is used as opposed to motion data - this would require a very controlled environment for the viewer as well as a tighter focus on the actor, assuming the same camera set up as in this project.
Equipment:
Physical:
Pocket Console (not necessary, but useful for testing)
Power strip - for plugging the PSUs in, to give power feed control
Special Thanks
Ryan Holsopple for guidance & advice
Ivan Kutyrev for checking my power plan & making sure I wouldn't set anything on fire
Mountain V. on Amazon for his very detailed explanation of how he used the products we purchased, which influenced my decision-making in purchasing
Ben Moll for additional Isadora support - as the network engineer for MLH
Anya Kopischke for additional Max support - as the Max operator for MLH
