7:2 A guide to installing and using ML5js and PoseNet
ml5.js aims to make machine learning approachable creative coders. It provides easy access to machine learning algorithms in the browser. ml5js has great documentation and examples that work well with p5js (though it can work with any other library).
Getting started with ml5js
There is great documentation on the ml5js website, their getting started page is here: https://learn.ml5js.org/#/
If you're using the p5js editor we can look at a few steps to get started. Daniel Shiffman also has lots of great videos on this on the Coding Train, you might watch this one here first.
Installing ml5js in your project
If you look at the reference area for each of the models you'll find a p5js example - this will open in the editor and with a bit of luck everything will be working already. However some don't, others are a bit out of date - so we will look at how you install it all yourself into your own project.
We add ml5js just like you might add p5js to a webpage. Start by adding the below script file to the <head> of your html file:
<script src="https://unpkg.com/ml5@latest/dist/ml5.min.js"></script>Check ml5js is installed
To check we have ml5js installed in your p5js code lets add this single line to your sketch.js file:
console.log('ml5 version:', ml5.version);In the code example here we have installed the script in the html and put the above javascript in the sketch. Open the left hand draw to check the index.html and sketch.js.
Play with a machine learning model with ml5js
We've installed ml5js in the most simple way. Now we could either gradually add the rest of the code, or just go to something like the example below and Duplicate one of their examples in the p5js editor, its up to you.
https://learn.ml5js.org/#/reference/posenet
Here is a very simple posenet example with the minimal amount of code to get going.
Try this yourself in the editor:
https://editor.p5js.org/amcc/sketches/Cg6aUmphF
Read on to see a breakdown of how this code works
Lets break down how poseNet works with p5js
First of all we need some variables to store everything
let video;
let poseNet;
let poses = [];We will use the webcam video and store it as video
We will start up ml5js poseNet and store it as poseNet
We will then start to get results and need to store these in an array which we will call poses
Getting the video
We want the webcam. This is the standard code in p5js to start up the webcam, we then size it to the screen. p5js by default creates an html <video> element which appears beneath our canvas, we dont want this so the last line hides this html element.
video = createCapture(VIDEO);
video.size(width, height);
video.hide();Starting poseNet and detecting poses
We then start poseNet with the ml5.posenet() function, we need to give this function 2 things, an image or video for the first argument and a function to callback when posenet is ready.
// detect poses with posenet
poseNet = ml5.poseNet(video, modelReady);
// every time new poses are detected store them in 'poses'
poseNet.on("pose", function (results) {
poses = results;
});The last bit of the code above is an event that responds to new poses, this then makes our poses array equal to the results. Now we have an array of poses we can do stuff with them.
drawing the poses to the canvas
If we have some poses we can draw them to the canvas. First of all lets put the video on the canvas with this line:
image(video, 0, 0, width, height);
We could draw every pose to the canvas and draw 'skeletons' and much more. The p5js docs show how to do this. But lets keep things really simple.
First of all check we have at least one pose detected - otherwise we get an error. Then we can draw the nose!
if (poses.length > 0) {
let nose = poses[0].pose.nose;
fill(255, 0, 0);
circle(nose.x, nose.y, 50);
}This last bit of code creates a variable called nose which is equal to the nose of first person that poseNet can detect.
The first person is pose[0], this is an object that contains nose.
Its then quite simple to make a circle with the x and y position of the nose.
What else can poseNet do
console.log is your friend, this prints out the info so you can explore it. In the draw loop do this:
console.log(poses)Instead of using the p5js editor's console, use the Chrome/Safari/Edge/Firefox console. The information streams past too quick in the editor console, the browser lets you scroll up a bit in the console, open up the objects and arrays and have a peek!
Most applications let you open the console with command + shift + i on a mac or control + shift + i on a pc - you then need to select the console in the sidebar that appears.
The final code and more
Here's a working example, click on the link below for the full code. If you want to copy this make sure you're logged in then Duplicate the sketch in the editors File menu:
https://editor.p5js.org/amcc/sketches/Cg6aUmphF
Check out the docs on PoseNet here:
https://learn.ml5js.org/#/reference/posenet
Their example with a webcam is called PoseNet_webcam it uses two functions to draw keypoints and the skeleton. Because you get an array of poses, each pose being an object full of things it looks a little complicated as they loop through all these bits. Have a look at the objects and arrays section again if you need more help.