<jupyter_start><jupyter_text>CS152 Final Project Gesture Controlled PC IntroductionThe main purpose of this assignment is to build a robot that (when connected to PC) enables you to control certain movement on your PC with your hands gestures. This feature has recently been introduced in some updated laptops and PC systems but its very costly and requires the user to understand and learn all the gestures if he wants to navigate through his computer efficiently. Moreover, this feature comes with some built in recognized gestures which may or may not have the option to customize and may be incovinient to use for a certain group of people (for example, I being a left handed person find it hard to play a car racing game with control keys in my right hand). This technology is named as "LEAP" and it was first introduced by HP into its laptops that use Leap Motion's gesture-based controller embedded directly inside the laptop to recognize the gestures. In this assignment, I am going to document my steps and show how can one transform any PC or laptop into a gesture controlled one in 1/100th the price of the original laptop with the built-in feature. Moreover, this gadget is totally customizeable so people with all demographics can choose the feature they want to control and how they want to control by just changing certain key association or the associated gesture. Although this model will require alot of training and trial and error in order to be as efficient as the in built leap laptops so for this assignment, we are only going to focus on how we can control some specific motions of our PC with gestures. These motions are described as follows: Objective 1: Controlling the motions of a playing video:For this feature we are going to customize our gadget so that it can control all the features of a video playing chrome applications. These features include1. Pause/Play video2. Increase/decrease volume3. Fast forward/ rewind the video 4. Exit the application Objective 2: Control a Scratch Based game with gestures:For this part of the project I will create a simple scratch game and try to control its motion using my gestures. Scratch basically is a block-based visual programming language and website designed for children learn code. By using this language, people can create projects on the web using a block-like interface. I will use its visual programming feature to showcase myself playing the game and interacting with it in real time. I will convert the inputs from my sensors into keyboard commands and associate them with my game controls. The rationale behind using scratch to build the game is that they are very less complicated. And implementation of this gadget on a complex game (Asphalt 8, Need for speed etc) would require us to build an AI model that would have to be trained according to the maps, noise in the environment, and more complex gaming controls in order to be efficient enough to recognize the gestures and play the game smoothly. This will require alot of time and skills that I will be focusing on for my intended capstone next year. So considering these constraints, I will have to build my own simple game (for now) to serve the purpose of this project.By the end of the paper I will also discuss how similar techniques can be extended and implemented on other PC controls and games. We will also discuss some alternative technologies that can be used to serve a similar purpose and their working such that you'll be able to differentiate between the approaches and their implementation. That being said, let's proceed towards the build up and circuit for the required robot.  Materials Used for building the Robot1. Arduino Uno2. (x2) Ultrasonic Sensors3. (x2) 220 Ohm Resistors4. Connecting wires5. Breadboard6. 9V Battery Circuit Diagrams![](Circuit_Diagram.png)In the circuit diagram seen above, I have connected two ultrasonic sensors to the Arduino boards. The pin description of this sensor is described below; Ultrasonic sensor Pin DescriptionVCC: This is the power supply for HC-SR04 Ultrasonic distance sensor which we connect the 5V pin on the Arduino.Trig: This name is the short form for Trigger. This pin is used to trigger the ultrasonic sound pulses. Echo: This pin produces a pulse when the reflected signal is received.Gnd: This pin is the grounded pin and is used as the power outflow for the sensor. It is very useful to complete the circuit Specifications of the Sensor UsedTrigger Input Signal: 10µS TTL pulseOperating Voltage: DC 5VOperating Frequency: 40KHz Explanation of the CircuitI have connected both the Vcc pins to the 5V power outlet of the Arduino board I have used 2 resistors to control the power inflow for our sensor. Although it is not necessay but this will be helpfull in case we decide to power our gadget through a 9v batter during the circuit testing phase. Both the Gnd pins are grounded. The Trig and echo pins of the sensors occupy pins 7,5 and 6,4 respectively. These will be used to send inputs to the Arduino board which will then be interpretted by python running locally on the machine. The implementation of this circuit diagram can be seen in the figure below which is created using a simulation software Simulated Circuit Diagram![](Implemented_circuit.png) Implemented CIrcuit and Building of the Robot![](a.png)![](b.png)![](c.png) Objective 1: controlling a video with gestures Working of the Prototype Recognizeable gesturesGesture 1: When both hands are placed in front of the sensor at a particular distance then the videoin VLC player should Play/Pause.Gesture 2: When the right hand is placed in front of the sensor at a particular distance then thevideo should Fast Forward.Gesture 3: When left hand is placed in front of the sensor at a particular far distance then the videoshould Rewind one step.Gesture 4: When right hand is in front of the sensor at a near distance and is moved towards thesensor the video should fast forward and if moved away the video should Rewind.Gesture 5: When left hand is in front of the sensor at a near distance and is moved towards thesensor the volume of video should increase and if moved away the volume should Decrease. Arduino Code<jupyter_code># Arduino Code to get inputs from the sensors and associating different inputs with different recognizeable gestures
const int trigger1 = 6; //Trigger pin of 1st Sesnor
const int echo1 = 7; //Echo pin of 1st Sesnor
const int trigger2 = 4; //Trigger pin of 2nd Sesnor
const int echo2 = 5;//Echo pin of 2nd Sesnor

long time_taken;
int dist,distL,distR;

void setup() {
Serial.begin(9600); 
  
pinMode(trigger1, OUTPUT); 
pinMode(echo1, INPUT); 
pinMode(trigger2, OUTPUT); 
pinMode(echo2, INPUT); 
}

/*###Function to calculate distance###*/
void calculate_distance(int trigger, int echo)
{
digitalWrite(trigger, LOW);
delayMicroseconds(2);
digitalWrite(trigger, HIGH);
delayMicroseconds(10);
digitalWrite(trigger, LOW);

time_taken = pulseIn(echo, HIGH);
dist= time_taken*0.034/2;
if (dist>50)
dist = 50;
}

void loop() {
calculate_distance(trigger1,echo1);
distL =dist; #get distance of left sensor

calculate_distance(trigger2,echo2);
distR =dist; #get distance of right sensor


#Pause Modes -Hold
if ((distL >40 && distR>40) && (distL <50 && distR<50)) //Detect both hands
{Serial.println("Play/Pause"); delay (500);}

calculate_distance(trigger1,echo1);
distL =dist;

calculate_distance(trigger2,echo2);
distR =dist;

#Control Modes
#Lock Left - Control Mode
if (distL>=13 && distL<=17)
{
  delay(100); //Hand Hold Time
  calculate_distance(trigger1,echo1);
  distL =dist;
  if (distL>=13 && distL<=17)
  {
    Serial.println("Left Locked");
    while(distL<=40)
    {
      calculate_distance(trigger1,echo1);
      distL =dist;
      if (distL<10) //Hand pushed in 
      {Serial.println ("Vup"); delay (300);}
      if (distL>20) //Hand pulled out
      {Serial.println ("Vdown"); delay (300);}
    }
  }
}

#Lock Right - Control Mode
if (distR>=13 && distR<=17)
{
  delay(100); //Hand Hold Time
  calculate_distance(trigger2,echo2);
  distR =dist;
  if (distR>=13 && distR<=17)
  {
    Serial.println("Right Locked");
    while(distR<=40)
    {
      calculate_distance(trigger2,echo2);
      distR =dist;
      if (distR<10) //Right hand pushed in
      {Serial.println ("Rewind"); delay (300);}
      if (distR>20) //Right hand pulled out
      {Serial.println ("Forward"); delay (300);}
  }
}
}

delay(200);
}<jupyter_output><empty_output><jupyter_text>Explanation of Arduino CodeThe above code is the C++ implementation for the working prototype. In this code I first configured all the pins on the arduino board that have a particular sensor pin associated with it. Once the configuration is done, I specify that the trig pins of both the sensors are responsible for the outputs where as both the echo pins are associated with the sensor inputs. Once we have our inputs and outputs configured, I tell the sensor to generate ultrasonic signals for 2 microseconds and then turn on the output interface to receive these signals back. The total time used for these signals to leave the sensor and comeback is then used to calculate the total distance of the object placed infront of the sensors. Once the distance is calculated, it is stored in two variables distL and distR.Here I would like to mention my code calculates this distance for everytime a stimulus is sensed by the sensors, and updates the left and right distance variables. This is why I created the distance measuring function which can be called everytime the sensors sense a stimulusOnce we know the distance, we now compare it with predefined values to specify certain metrics that will help us specify certain gestures. As for this identification, instead of using a particular distance value, I used a range between the values to increase the overall time taken to get the inputs. Once I have configured the inputs, I use them in the next section where I use python to associate these inputs with prefered outputs Python Code<jupyter_code>import serial  #Serial imported for Serial communication
import time #Required to use delay functions
import pyautogui

ArduinoSerial = serial.Serial('COM5',9600) #Create Serial port object called arduinoSerialData
time.sleep(2) #wait for 2 seconds for the communication to get established

while 1:
    incoming = str (ArduinoSerial.readline()) #read the serial data and print it as line
    print(incoming)
    
    if 'Play/Pause' in incoming:
        pyautogui.typewrite(['space'], 0.2)
        print('Play/Pause')

    if 'Rewind' in incoming:
        pyautogui.hotkey('ctrl', 'left')
        print('Rewind')

    if 'Forward' in incoming:
        pyautogui.hotkey('ctrl', 'right')
        print('Forward')

    if 'Vup' in incoming:
        pyautogui.hotkey('ctrl', 'down')
        print('Volume up')
        

    if 'Vdown' in incoming:
        pyautogui.hotkey('ctrl', 'up')
        print('Volume Down')

    incoming = "";<jupyter_output><empty_output><jupyter_text>Explanation of my Python CodeThe above code is the python implementation for the working prototype that is used to get the end results and perform actions. In simple robotics terms, I can term this part of my implementation to be the end effector that reacts to the stimulus in the environment and generates outputs which in this case are associations with keys on the key board. The stimulus , or in my case hands, are identified by the sensors according to the associated gesture. The actions are performed by creating a serial communication with arduino and then associate certain keyboard shortcuts with the incoming inputs. This is done by using python's pyautogyi module which lets our Python script to control the mouse and keyboard to automate interactions with other applications. Once the actions are performed, it prints out a list of actions performed which can bee seen in the test output below Test controlling a video with gestures<jupyter_code>from IPython.display import Video

Video("screen-capture (1)_Trim.mp4")<jupyter_output><empty_output><jupyter_text>Python Generated Outputs During Prototype Testing![](d.png)  Objective 2: Controlling a Scratch based game with Gestures Arduino Code<jupyter_code># Arduino Code to get inputs from the sensors and associating different inputs with different recognizeable gestures
const int trigger1 = 6; //Trigger pin of 1st Sesnor
const int echo1 = 7; //Echo pin of 1st Sesnor
const int trigger2 = 4; //Trigger pin of 2nd Sesnor
const int echo2 = 5;//Echo pin of 2nd Sesnor

long time_taken;
int dist,distL,distR;

void setup() {
Serial.begin(9600); 
  
pinMode(trigger1, OUTPUT); 
pinMode(echo1, INPUT); 
pinMode(trigger2, OUTPUT); 
pinMode(echo2, INPUT); 
}

/*###Function to calculate distance###*/
void calculate_distance(int trigger, int echo)
{
digitalWrite(trigger, LOW);
delayMicroseconds(2);
digitalWrite(trigger, HIGH);
delayMicroseconds(10);
digitalWrite(trigger, LOW);

time_taken = pulseIn(echo, HIGH);
dist= time_taken*0.034/2;
if (dist>50)
dist = 50;
}

void loop() {
calculate_distance(trigger1,echo1);
distL =dist; #get distance of left sensor

calculate_distance(trigger2,echo2);
distR =dist; #get distance of right sensor

#Control Modes
#Lock Left - Control Mode
if (distL>=13 && distL<=17)
{
  delay(100); //Hand Hold Time
  calculate_distance(trigger1,echo1);
  distL =dist;
  if (distL>=13 && distL<=17)
  {
    Serial.println("Left Locked");
    while(distL<=40)
    {
      calculate_distance(trigger1,echo1);
      distL =dist;
      if (distL<10) //Hand pushed in 
      {Serial.println ("MoveLeft"); delay (300);}
    }
  }
}

#Lock Right - Control Mode
if (distR>=13 && distR<=17)
{
  delay(100); //Hand Hold Time
  calculate_distance(trigger2,echo2);
  distR =dist;
  if (distR>=13 && distR<=17)
  {
    Serial.println("Right Locked");
    while(distR<=40)
    {
      calculate_distance(trigger2,echo2);
      distR =dist;
      if (distR<10) //Right hand pushed in
      {Serial.println ("MoveRight"); delay (300);}
  }
}
}

delay(200);
}<jupyter_output><empty_output><jupyter_text>Python Code<jupyter_code>import serial  #Serial imported for Serial communication
import time #Required to use delay functions
import pyautogui

ArduinoSerial = serial.Serial('COM5',9600) #Create Serial port object called arduinoSerialData
time.sleep(2) #wait for 2 seconds for the communication to get established

while 1:
    incoming = str (ArduinoSerial.readline()) #read the serial data and print it as line
    print(incoming)

    if 'MoveLeft' in incoming:
        pyautogui.hotkey('ctrl','left')
        print('Moving Left')

    if 'MoveRight' in incoming:
        pyautogui.hotkey('ctrl','right')
        print('MoveRight')

    incoming = "";<jupyter_output><empty_output><jupyter_text>Scratch Game Discription and Associated GesturesGame link: https://scratch.mit.edu/projects/466143079Here I created this simple scratch game where a bunch of aliens appear on the top of the screen and you have to bounce the ball, in my case planet, over a flat surface to kill as many of them possible in order to save the galaxy. The code for the game can be seen by clicking at the link above. Now I am goint to test run this game using my updated arduino code and associated python code. Here I just used two gesturesGesture 1: When the left hand is placed in front of the left sensor at a particular distance and then moved closer to it then the flat surface should head to leftGesture 2: When the right hand is placed in front of the left sensor at a particular distance then the flat surface should head to Right Test Controlling the Scratch Based game<jupyter_code>from IPython.display import Video

Video("screen-capture (2).mp4")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06305963040579157, 'fraction_numerical': 0.017336635549628502, 'mean_word_length': 4.468055555555556, 'pattern_counts': {'":': 0, '<': 43, '<?xml version=': 0, '>': 39, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '136085', 'license': 'MIT', 'n_tokens_mistral': 4409, 'n_tokens_neox': 4090, 'n_words': 2180, 'path': 'CS152 FInal Project/CS152; Final Project.ipynb', 'repo_name': 'Usamapuri/Gesture-Controlled-Computer', 'size': '15747'}