Sikuli Bot

SikuliBot is an image-based approach to automating user interface. SikuliBot extends the visual programming concept of Sikuli Script from the graphical UIs to the real world of physical UIs, such as mobile devices’ touch-screens and hardware buttons. The key to our approach is using a physical robot to see an interface, identify a target, and perform an action on the target using the robot’s actuators. We demonstrate working examples on MakerBot 3D printer that could move a stylus to perform multi-touch gestures on touchscreen to automate tasks such as swipe-to unlock, playing a virtual piano, and playing the Angry Bird game. A wide range of automation possibilities are made viable using a simple scripting language based on images of UI components. The benefits of our approach are: generalizability, instrumentation-free, and high-levelprogramming abstraction.

Applications
Playing Piano Playing Games Typing a Keyboard