Phone as Trackpad for your Computer

I am pretty sure that I am not the only one who crave for a mouse when I don't have one and crave for a trackpad when I don't have access to one. I have a few ideas on turning your phone into a mouse and a trackpad. Let us focus on the later in this blog post. There are two components to this experiment; the first is a server on the computer that you wish to control and the second is a web app that is opened up on your phone that will transmit coordinate data to the server. I think we'll create the server in Python and the web app in JavaScript.

Let's get started with creating the server. We'll use 'Flask' to handle API requests and 'PyAutoGUI' to control the mouse. We'll have APIs defined equivalent to cursor move and click for now.

import pyautogui

from flask import Flask, render_template

app = Flask(__name__)

# Apparently flask doesn't support negative numbers. So offsetting it.

coordinateShifter = 1000

@app.route('/')

def index():

f = open("index.html", "r")

return f.read()

@app.route('/move/<int:x>/<int:y>')

def move(x, y):

pyautogui.move(x - coordinateShifter, y - coordinateShifter)

return 'moving'

@app.route('/click')

def click():

pyautogui.click()

return 'clicking'

To run the python server, simply run the following commands on your terminal.

export FLASK_APP=server.py

flask run -h <your_ip_address>

You can experiment with the move and click APIs by opening the following links on your browser and observe the mouse moving and clicking.

http://<your_ip_address>:5000/move/200/100

http://<your_ip_address>:5000/click

Now, we need to create a Web App that can detect click and move using a touch screen and invoke the above URLs as API requests.

The HTML and CSS may designed to display a DIV that covers the entire screen. The following snippets respectively depict the same.

<div id="fullscreen"></div>

 

#fullscreen {

    background-color: #cccccc;

    text-align: center;

    position: absolute;

    top: 0;

    left: 0;

    right: 0;

    bottom: 0;

}

The JavaScript logic would be as follows:

 // Apparently flask doesn't support negative numbers. So offsetting it.

const coordinateShifter = 1000;

const elem = document.getElementById("fullscreen");

var prevX, prevY;

elem.addEventListener('mousedown', function (e) {

    console.log('click');

    var url = window.location.href + 'click';

    var xmlHttp = new XMLHttpRequest();

    xmlHttp.open("GET", url, false);

    xmlHttp.send(null);

}, false);

elem.addEventListener('touchstart', function (e) {

    e = e.originalEvent || e;

    e.stopPropagation();

}, false);

elem.addEventListener('touchmove', function (e) {

    e.preventDefault();

    if (e.targetTouches.length === 1) {

        console.log('move');

        if (typeof prevX === 'undefined') {

            prevX = e.targetTouches[0].pageX;

            prevY = e.targetTouches[0].pageY;

        } else {

            var diffX = e.targetTouches[0].pageX - prevX;

            var diffY = e.targetTouches[0].pageY - prevY;

            prevX = e.targetTouches[0].pageX;

            prevY = e.targetTouches[0].pageY;

            var url = window.location.href + 'move/' + parseInt(coordinateShifter + diffX) + '/' + parseInt(coordinateShifter + diffY);

            var xmlHttp = new XMLHttpRequest();

            xmlHttp.open("GET", url, false);

            xmlHttp.send(null);

        }

    }

    e.stopPropagation();

}, false);

elem.addEventListener('touchend', function (e) {

    console.log('move-done');

    prevX = undefined;

    prevY = undefined;

}, false);

Open up the web app on your phone using the URL http://<your_ip_address>:5000 and start controlling your mouse. You can observe that the controls are not very smooth, but neverthless, it works. The following is a video recording wherein I try to control the mouse using my phone. I did not have another camera to take the entire video, therefore I am just sharing the screencast. Note that only cursor movement and clicking have been implemented. You could also attempt to create a click and drag mechanism or multitouch scrolling as well.

Let me also attach the screenshot of the web app. Plain and simple, huh? If you are worried about the battery consumption, feel free to change the colout to black.

I hope this was inspiring. Feel free to modify the code and create something awesome.