Hand Gesture based Simple Jump unity game

-


This game was developed using unity and python opencv using simple hand recognition using contour.

To learn more on unity visit:

You  should know how to use Hand Gesture Recognition in this tutorial, as well as how to communicate and make use of the data generated by the Hand Gesture Recognition in order to control a player in your Unity game.

Specifially, you’ll learn:

  • Basics of communication protocols in Networking.
  • What is OpenCV and some of its applications.
  • How to install, set up and run the Python version of OpenCV on your system.
  • How to send data to a specific port via Sockets in Python.
  • How to read data from a Socket with the help of UDP Client in Unity.

Setting Up Your Scene

Open the Main scene inside the Scenes folder.

You can create your own game my suggestion is watch

Brackeys tutorial

https://www.youtube.com/channel/UCYbK_tjZ2OrIZFBvU6CCMiA

If you click the Play button at this moment, there should be background music playing. Additionally, you’ll see the player in his Idle animation, and not much else going on at this moment.

You should also see the PlayerController under Managers, with a PlayerController script attached to it. This is the file you’ll add all your code to later on.

In this tutorial, you’ll use Python to create a virtual server and use OpenCV to detect the number of fingers of your hand as recorded by the webcam, and you’ll use Sockets to send that information to a predefined ‘port’.

Once that is working, you’ll use an UDP Client, which is part of the System.Net.Sockets class library, to receive that data in the Unity project.

Understanding Key Concepts

Before moving on to the actual coding, take a look at the underlying concepts that will be applied throughout the rest of this tutorial.

Communication Protocols and UDP

In order to understand how the Python-OpenCV script and your Unity instance will communicate with each other, it’s important to know about the underlying principles of how data is shared between two or more applications in computer systems.

Communication protocols are formal descriptions of digital message formats and rules. They are required to exchange messages in or between computing systems and are required in telecommunications.

Two commonly used Protocols are TCP (Transmission Control Protocol) and UDP (User Datagram Protocol).

TCP is the dominant protocol used for the bulk of internet connectivity owing to the services it provides for breaking large data sets into individual packets, checking for and resending lost packets, and reassembling packets into the correct sequence — sending and receiving email (SMTP) and browsing the web, for example.

In contrast, applications that communicate using the UDP protocol just send the packets and don’t check or wait for an acknowledgement before sending the next packet, which means that they have much lower bandwidth overhead and latency.

Now that you understand a little bit about how the data will be communicated between Unity and our Python instance, here’s a little bit about OpenCV and how it will help in detecting the fingers of your hand.

OpenCV

OpenCV is a library of programming functions mainly aimed at real-time computer vision. 

A broad overview of the steps performed by OpenCV to detect the number of fingers:

  1. Frames are captured from the camera (webcam) as images.
  2. Gaussian Blur is applied to the image to blur the image. This is done to reduce the noise in the image and to separate out the outlines of the fingers for the later processing stages. (Left : Normal Image, Right : Blurred Image)
  3. A binary image (by removing all but two colors from the image) is created, wherein the skin color is replaced with white and everything else is replaced with black.
  4. Contour detection is applied to find the defective areas of the hand.
  5. Based on the number of contours detected the number of fingers is calculated.

You can read through the gesture_recognition.py script to get an idea of what each line of code is doing.

Once the number of fingers is recognized, the Socket library is used to send the relevant data via UDP to port number 5065 



Download code :Github
 

Comments

Post a Comment

Popular posts from this blog

Lane Detection

-
Image
In a nutshell, I’ve taken the following steps to accomplish this: Correct for Camera Distortion Filter Lane Pixels Get the“Sky View” Fit the “Best-Fit” Curve Project the Lane back to the “Vehicle-View” Image 1. CAMERA CALIBRATION Because of the physical properties of a camera lens, the captured two-dimensional image isn’t perfect. There are image distortions that change the apparent size and shape of an object. More importantly it makes some objects appear closer or farther away than they actually are. Fortunately, we can measure this distortions and correct them. We can extract all the distortion information we need by having pictures of objects which we know where the certain points should be theoretically. Commonly used are chessboards on a flat surface because chessboards have regular high contrast patterns. It’s easy to imagine what an undistorted chessboard looks like. 2. PERSPECTIVE TRANSFORMATION After having the image corrected, we’d have an undistorted image of a road from th...
Read more

Gender Age Facial prediction

-
Image
Gender Age Facial prediction The project idea was implemented using OpenCV-Python and aws. The aim was to predict the gender,age and other facial features of the image or frame in a video. Requirements:  Python3 library: opencv-python  https://pypi.org/project/opencv-python/ AWS account Step 1:            Sign up an aws account and create iam user service (amazon rekogmition and amazon bucket s3) enabled.          Once created iam user account download credential.csv file and copy the file to your folder location.( Note don't share your credential to anyone ) Step 2:            write a  python program to capture the video using webcam.                        Read the credential file to your program and read  aws_access_key_id  and aws_secret_access_key. with open ( "new_user_credentials.csv" , 'r' ) as in...
Read more