Ideation

Concept

This project involved three different types of users: the audience, the performers (people in the class), and me, the controller/composer. The performers each have their own custom-created web environment for programming and seeing the live output of a P5 sketch. Their code is sent to a central server where the outputs are combined together in interesting ways, with my input being used to refine the final product. The performance is done to music, where the server sends the amplitude/pitch as live variables to the clients to use as a basis for music visualization.

Development

Performer View / Code Editor

The first hurdle was simply creating a web environment that's able to support writing P5 code in real time with syntax highlighting and live output for both the sketch and overall performance. As such, I spent the first week of the project refining this component of the experience, and this is where I ended up:

Getting P5 code to run arbitrarily on input event was quite a hassle. I initially tried a simple eval function with an accompanying canvas injection into the output div, but I could never get it to detect the P5 library alongside the user's script. Eventually I took a cue from P5 live and used an iframe to inject the scripts into its 'srcdoc' property. Here's the code for how that was accomplished.

<pre id="codeContainer" contenteditable="true">
      <code id="codeBlock" class="language-js">
      ...
      </code>
</pre>
...
<iframe id="output" sandbox="allow-same-origin allow-scripts" volume="0"></iframe>

function renderOutput(songName, songTime, songPlaying){
    // Save codeblock element
    let codeBlock = document.getElementById("codeBlock")
    // Get code of input without HTML element wrappers
    let codeBase = codeBlock.innerHTML.replace(/(<([^>]+)>)/ig,"")
        .replace(/&amp;/g, "&").replace(/&lt;/g, "<").replace(/&gt;/g, ">");
    // Apply to output iframe
    document.getElementById("output").srcdoc = replacementHTML
    // Try evaluating code to catch errors
    try{ eval(codeBase) }catch(error){ console.log("Code error") }
    // Return output code
    return codeBase;
}

The syntax highlighting was accomplished by Prism JS, though using it required quite a few workarounds as any time the syntax highlighting was run it would reset the cursor position of the corresponding contenteditable div back to the first character, so I pieced together a way to detect then change the cursor position each time the syntax highlighting is run.

Audience View / Code Execution

Next step– get the outputs to combine together in interesting ways and form a live feed to the audience. The primary concern here is performance: whether they're combined through four+ individual video feeds or streams of code as it changes, it would likely end up taxing for the audience's devices. After going down a rabbit hole of figuring out web video streaming, I decided to go with the decidedly simpler code stream. The audience client basically listens for a SocketIO event indicating a new code for a certain id, then updates the performer's respective frame to evaluate that new code:

// Upon receiving new performer
socket.on('performerConnected', id => {
    performers.push({id: id, codeBase: ''})
    // Log table of current performers
    console.table(performers);
    // Add iframe for performer
    $("#performers").append(`
    <iframe class="output" id="${id.split("#")[1]}" 
    sandbox="allow-same-origin allow-scripts">
    </iframe>`);
    // Add code overlay for performer
    $("#performersCode").append(`
    <div class="codeTile ${id.split("#")[1]}">
        <div class="codeUpdate"></div>
        <div class="idTag">${id}</div>
    </div>`);
});

socket.on('codeChange', ({id, codeBase}) => {
    socket.emit('songStateRequest', {audienceId: socket.id , performerId: id, requestTime: Date.now()});
    // Log code change
    console.log(`📦 ${id} has changed their code`)
    // If performer does not yet exist
    if(!performers.some(performer => performer.id == id)){
        performers.push({id: id, codeBase: codeBase})
        // Log table of current performers
        console.table(performers);
        // Add iframe for performer
        $("#performers").append(`
        <iframe class="output" id="${id.split("#")[1]}" 
        sandbox="allow-same-origin allow-scripts"
        srcdoc="${returnOutput("1", false, 0, codeBase, id.split("#")[1])}">
        </iframe>`);
        // Add code overlay for performer
        $("#performersCode").append(`
        <div class="codeTile ${id.split("#")[1]}">
            <div class="codeUpdate">${codeBase}</div>
            <div class="idTag">${id}</div>
        </div>`);
        // Set timeout to remove code overlay
        setTimeout(() => {
            $(`.${id.split("#")[1]} .codeUpdate`).html("");
        }, 2000);
    // If it does exist
    } else {
        // Update array listing
        performers[performers.findIndex(performer => performer.id == id)]['codeBase'] = codeBase;
        // Change iFrame srcdoc property
        document.getElementById(id.split("#")[1]).srcdoc = returnOutput("1", false, 0, codeBase, id.split("#")[1]);
        // Add code overlay
        $(`.${id.split("#")[1]} .codeUpdate`).html(codeBase);
        // Set timeout to remove code overlay
        setTimeout(() => {
            $(`.${id.split("#")[1]} .codeUpdate`).html("");
        }, 2000);
    }
});

User Testing

Performer Testing

Luckily I had plenty of friends willing to user test the performance– both from the performer's side and the audience's side. The first round of testing was primarily focused on the nature of the programmer/performer's interface: did they have the tools they needed to perform live? I found that starting them with just a bare p5 document didn't inspire much in the way of live/reactive code, mostly some static shapes even when I explicitly wrote that the live amplitude variable for the music should be used. I decided to change the starter/default to a pulsating circle that I pre-programmed as a starting point for the performers.

In-Class Demo

I was given a chance to demo this project in class, with two classmates, my professor, and myself being the performers while the rest of the class was in the audience. Here are some screen shots from that performance: