Went into the University for the first time

I went into the university for the first time today. It is less than a kilometer from the apartment. I had to go there to finalize certain documents regarding my accomodation. I arrived at the university during the lunch time and hence I had to wait for a couple of hours for the office staffs to return. During the time, I roamed around the campus. The place is aesthetically pleasing, however, it is pretty small, especially since where I did my bachelor's, IIT Patna, had a campus spread across over 500 acres. I also went to the nearby Lidl to get a few frozen pizzas. I had that for lunch, evening snack and dinner. I had thought about cooking something for dinner, but I felt a bit lazy to do the same. However, I did modify the last frozen pizza by adding some mushrooms, pizza sauce and pepper. I think this could be a good trick moving forward - buy a few frozen pizzas and chopped vegetables from Lidl and fuse them. I think this would make the pizzas a bit more healthier and the effor

Drawing in the Air

Drawing in the air


For someone who is completely unaware about image processing, this project would seem something out of the world. The people with some understanding about how to read an image pixel by pixel would have some understanding about what I am doing here. I am building a program in JavaScript that takes input from the computer's webcam and create a tool that lets you draw in the air and that gets displayed on the computer. For this, we will have a special pen to which we code our program to detect.

To get started, let's use the <video> tag to receive the video feed from the camera, which I will later make invisible.. We'll also have two <canvas> tags that would each hold the current video frame and the drawing output from the tool respectively. We shall also have a button that clears data from the second canvas that has the drawing output.

<video autoplay="true" id="videoElement"></video>
<canvas id="canvas"></canvas>
<canvas id="drawingCanvas"></canvas>
<button onclick="clearFrameMemory()">Clear Drawing</button>

Let us assume that the video frame is of the size 500x420. The array named 'frameMemory' would store the data that shall be rendered on the 'drawingCanvas'. Let's define the function that would clear this array.

const width = 500;
const height = 420;
var frameMemory = [];
function clearFrameMemory(){
    for(let i = 0; i < width*height*4; i++) {
        if(i%4 === 3){
            frameMemory[i] = 255;
        } else {
            frameMemory[i] = 0;
        }
    }
}
clearFrameMemory();

We make the video element start receiving data from the webcam using the following JavaScript code. Note that I have also made the video element display nothing as I am planning to make the first canvas display the same.


var video = document.querySelector("#videoElement");
video.style.display = "none";

if (navigator.mediaDevices.getUserMedia) {
  navigator.mediaDevices.getUserMedia({ video: true })
    .then(function (stream) {
      video.srcObject = stream;
    })
    .catch(function (error) {
      console.log("Something went wrong!");
    });
}

The following bunch of code sets up the two canvases with the previously mentioned dimensions. It also stores the context of the canvases in different variables.

var canvas = document.querySelector('#canvas');
canvas.width = width;
canvas.height = height;
var context = canvas.getContext('2d');

var drawingCanvas = document.querySelector('#drawingCanvas');
drawingCanvas.width = width;
drawingCanvas.height = height;
var drawingContext = drawingCanvas.getContext('2d');

Now, we need to get the contents from the video at every frame. To do that, we attach an event listener on 'play' to the video element. The function named 'timerCallback' gets invoked from the event listener, which get's executed at every instance in a recursive, yet asynchronous manner, through 'setTimeout'.

var timerCallback = function timerCallback() {
    if (this.video.paused || this.video.ended) {
      return;
    }
    computeFrame();
    setTimeout(function() {
        timerCallback();
    }, 0);
}

// video 'play' event listener
video.addEventListener('play', function() {
    timerCallback();
}, false);

As you can see, the function 'timerCallback' invokes 'computeFrame' function at every instance. This is the function in which the main logic for the drawing tool goes in. Let me break that down. The first portion of the code simply sets the current video frame to the first <canvas>. Following that it iterates over all the pixels in the frame and checks for certain colors falling within the range - red < 20, green < 40 and blue > 45. In the curent lighting condition, I manually observed that this is the color of the pen that I am trying to use. Whereven this color is detected, we mark the 'frameMemory' and that is equivalently being displayed in the second <canvas>. We never clear the frame memory unless the clear function is invoked.


var computeFrame = function computeFrame() {
    context.drawImage(video, 0, 0, canvas.width, canvas.height);
    let frame = context.getImageData(0, 0, canvas.width, canvas.height);

    // Lateral inversion (Mirror Image)
    for (i=0; i<frame.height; i++) {
        // We only need to do half of every row since we're flipping the halves
        for (j=0; j<frame.width/2; j++) {
            var index=(i*4)*frame.width+(j*4);
            var mirrorIndex=((i+1)*4)*frame.width-((j+1)*4);
            for (p=0; p<4; p++){
                var temp=frame.data[index+p];
                frame.data[index+p]=frame.data[mirrorIndex+p];
                frame.data[mirrorIndex+p]=temp;
            }
        }
    }
    context.putImageData(frame, 0, 0, 0, 0, frame.width, frame.height);

    // Getting the pixels for the pen
    let l = frame.data.length / 4;
    for (let i = 0; i < l; i++) {
        let r = frame.data[i * 4 + 0];
        let g = frame.data[i * 4 + 1];
        let b = frame.data[i * 4 + 2];

        if (r < 20 && g < 40 && b > 45) {
            frame.data[i * 4 + 0] = 255;
            frame.data[i * 4 + 1] = 255;
            frame.data[i * 4 + 2] = 255;
            frameMemory[i * 4 + 0] = 255;
            frameMemory[i * 4 + 1] = 255;
            frameMemory[i * 4 + 2] = 255;
        } else {
            frame.data[i * 4 + 0] = frameMemory[i * 4 + 0];
            frame.data[i * 4 + 1] = frameMemory[i * 4 + 1];
            frame.data[i * 4 + 2] = frameMemory[i * 4 + 2];
        }
    }

    drawingContext.putImageData(frame, 0, 0);

    return;
}



You can see how well this performs. To improve the performance, certain image processing techniques to filter the noise could be applied.

I hope you enjoyed this. I was originally intending on creating a three dimensional version of this, but due to COVID-19, I am unable to procure another webcam to perform this task. Maybe I will revisit this in the future.

Comments

Post a Comment

Popular posts from this blog

A decade of BuddyGo.net

Bought a new domain - chickfort.com

My attempt to create a Quine in MATLAB