Monday, April 2, 2012

Accelerometer Based Gesture Recognition - Integration With Windows Paint

This article delineates, yet another application of Gesture Recognition Framework by integrating it with Windows Paint application. The idea behind this work is to provide economic solutions for smart classes which makes interactive teaching very imminent. The application is just not confined by few use cases, but further enhancements will lead to  opportunities for wider applications like gesture to text conversion, presentations involving multiple participants, image viewer control etc. 

Project Outline

Paint Brush is an accelerometer controlled painting application, which enables user to draw basic line diagrams in 2D space . Inputs for this framework are the gestures generated from user's handheld device, as the user tilts/waves his cell phone in the air.
The project currently is in it's primitive stage of development which renders user's gestures only as horizontal and vertical lines, but further enhancements makes it feasible to draw various geometrical shapes, curves and free hand drawing as well.


Domain Model



Accelerometer based Paint Brush application consists four modules,
  • Accelerometer module on mobile device
The accelerometer module running on user’s mobile will initiate Accelerometer hardware and start listening to accelerometer output data which returns the acceleration due to gravity of the device in three dimensions. This is helpful in understanding the inclinations of the device due to tilting gestures and also the direction in which device is moving. This information is fed to Gesture Recognition Algorithm which processes raw accelerometer data values into useful line patterns which is drawn within Paint Brush window.
  • Bluetooth server running on user's PC
Bluetooth server running on user’s PC will publish text/file transfer services, and these services are discovered by Bluetooth client application running on user’s device. Once the service is discovered the client will establish a connection with server and start pushing accelerometer data to server socket. Server will continue to receive Bluetooth client data using a series of “Receive” function calls.
  • Bluetooth client running on mobile device
Bluetooth client running on mobile device will search for peer Bluetooth servers publishing various services, In this case the client will search for file/text transfer services. After discovering the service client makes a connection with Bluetooth server and starts sending accelerometer data to server using series of “send” function calls.
  • Gesture Recognition Framework for Painting
Bluetooth server module feeds painting framework with Gestures pumped continuously from user's handheld device. The framework gets hold of target painting application on which these gestures are executed to achieve required functionality. Each gesture is translated distinct draw event  which ultimately renders desired shapes on the screen.

Future Enhancements
  • Enable drawing of curves, and other geometrical shapes.
  • Text rendering
  • Multicolor support
  • Adjustable brush/pencil size.
Suggestions and feedback on this work is most appreciated. I'm reachable at

Share This Article