Rock Paper Scissor ML

Background on Project

This week I decided to combine a couple of my courses and make a project that used both websockets (from Collective Play) and image classification. This ended up manifesting in a multiplayer Rock, Paper, Scissors game in which the two players can make hand gestures into their webcam which is picked up by ML5, translated into their sign, then sent to the server for distribution to the other player.

Model Training

The first step of the project was training the model to see if I could get it to properly recognize what most people would consider rock, paper, scissors gestures. I enlisted the help of a few friends in different environments with different skin tones and hand sizes to try to get the most accurate representation of the gestures. This is where I ran into one of two primary hiccups with Teachable Machine: the lack of any in-progress 'save' functionality meant that I had to keep the browser instance open the entire time for about a day. If I accidentally closed it or reloaded it (which happend once at the start) or if it happend to crash, then all progress was lost. It ended up working out fine in the end, but it made me quite nervous for most of the day!

This GIF shows it in the later environment of the game itself, only because I had forgotten to get a recording of it at the time. This does however show it working fairly well in a new environment with a very noisy image. Scissors works the best, followed by paper, followed by rock.

Client and Server Development

The setup of the model was fairly plain, however I did run into some issues. For Collective Play we are required to upload projects to Glitch.com to be able to run server code and rapidly create new instances of multiplayer setups. There was no way of getting it to work with Glitch, as Glitch wouldn't allow the weights bin to be loaded locally and ML5 kept changing the URL request for the model to lowercase letters. This gave me the option of re-recording the entire model (can't rename a Teachable Machine model after publishing) or just showing it being run locally.

// Setup function
    function setup() {
        // Remove P5 Canvas 
            noCanvas();
        // Create a camera input
            video = createCapture(VIDEO);
        // Initialize the Image Classifier method with MobileNet and the video as the second argument
            classifier = ml5.imageClassifier('model/model.json', video, modelReady);
    }
    
// Executed when ML model is loaded
    function modelReady() {
        console.log('Model Ready');
        classifyVideo();
    }

The logic of the image classification itself is relatively standard as well. It takes the image classifier, get the ordinary single 'most likely' result, stores it as a variable for later, and change the icon on screen to match.

// Get a prediction for the current video frame
    function classifyVideo() {
        classifier.classify(gotResult);
    }

// When we get a result
    function gotResult(err, results) {
        // Store current value
            currentThrow = results[0].label;
        // Change icon to represent current guess at symbol
        // The results are in an array ordered by confidence.
            $("#throwIcon").css("background-image","url('img/" + throwIcons[currentThrow] + "')");
        // Re-run classify function (infinite loop)
            classifyVideo();
    }

It does get rather interesting when integrated into sockets. First was the Socket.io handshake between the server and client to ensure that both were valid and log to both's console.

// Server
io.sockets.on('connection',
  // Callback function on connection
  // Comes back with a socket object
function (socket) {
    // Check to limit users to 2
    if(users < 2){
      // Log connection info
      console.log("We have a new client: " + socket.id);
      // Update user count and store id
      userIds[users] = socket.id;
      users += 1;

      // Listen for data from this client
      socket.on('message', function(message) {console.log("Received hello! " + message);});
      ...
    } 
  }
);

// Client
// Get socket connection
  socket.on('connect', function() {
    console.log("Connected");
    socket.emit('message', 'Hello server');
  });

Next was to transmit and receive the actual signs picked up by the classifer. This is done when a 'shoot' signal, timed by the server, is broadcast to both clients. It then start a three second countdown, and sends it back.

// Server
socket.on('throw', function(receivedThrow) {
  // Update user throw values
  userThrows[userIds.indexOf(socket.id)] = receivedThrow;
  console.log("THROW VALUE: " + userThrows[userIds.indexOf(socket.id)]);
});
...
setInterval(function(){
  // Emit 'shoot' command to get current user values
    io.sockets.emit('shoot', 'shoot');
    console.log("Shoot!")
  // Wait to check results - 3000 for countdown, 1000 for transit time
    setTimeout(function(){
      io.sockets.emit('result', userThrows);
    },4000);
},8000);

// Client
// Upon receiving shoot message from server
 socket.on('shoot', function(){
  setTimeout(function(){
    socket.emit('throw',currentThrow);
    console.log("Throw emitted: " + currentThrow);
  },3000);
 });

Finally, it receives the throws from the server and figures out who won. It appends the results to the table on the right hand side. This is done client side only because the programming for the server was easier and I was under a bit of a time crunch to get it done in time for the Collective Play class (which was a day after the assignment began for our class).

// Upon receiving result
  socket.on('result', function(result){
    let otherThrow;
    console.log(result);
    if(currentThrow!=result[0]){otherThrow==0}
    else if(currentThrow!=result[1]){otherThrow==1}
    // Draw
    else{$("#resultsTable  tr:last").after("<tr><td>" + currentThrow + "</td><td>" + currentThrow + "</td><td>Draw</td></tr>");}
    // Win
    if(currentThrow == 0 && result[otherThrow] == 2 || currentThrow == 1 && result[otherThrow] ==  0 || currentThrow == 2 && result[otherThrow] == 1){
        $("#resultsTable  tr:last").after("<tr><td>" + currentThrow + "</td><td>" + result[otherThrow] + "</td><td>Win</td></tr>");
    }
    // Loss
    else if(currentThrow == 2 && result[otherThrow] == 0 || currentThrow == 0 && result[otherThrow] ==  1 || currentThrow == 1 && result[otherThrow] == 2){
        $("#resultsTable  tr:last").after("<tr><td>" + currentThrow + "</td><td>" + result[otherThrow] + "</td><td>Win</td></tr>");
    }
  });