Draw-Detect is an interactive machine learning tool that allows end users to create a model of n-shape classes by graphically drawing example shapes. The system can then predict the class of new shapes, and indicate the proximity of the shape to all user-defined classes.The goal of this system was to explore initial methods of incorporating machine learning into a digital illustration tool. The working system currently classifies end user-created vector line drawings based on a small sample set of user-defined classes.
Input: User draw shapes (square, spiral, diamond, squiggle etc.) and plain text class labels corresponding to different categories of shapes.
Output: Prediction of which class a new user drawn shape belongs to (e.g. squiggle), along with a visualization of the prediction metrics of the system (class likelihood, distance to other classes etc.). Draw-Detect scales the points and edges of each user-defined shape within a target area and translates them to the origin. Following this transformation, Draw-Detect calculates the following features for each shape instance: accumulated edge length (sum of all of the edges), accumulated angle (calculating angles between -180 and 180),accumulated relative angle (calculated after normalizing the length of each edge), accumulated slope, total number of points, distance between the start point and end point, the scaled width, height and area of the shape. Because of the transformation and scaling, the system is invariant to scale and position, and generally able to recognize similar shapes that differ in size and location. During both training and prediction phases, Draw-Detect also displays a visualization of the features of a shape to the user.
Back-end: Draw-Detect uses the min-dist algorithm in the Gesture Recognition Toolkit. Min-dist is a supervised learning algorithm that works by independently fitting a user-defined number of clusters to each class. New instances are classified by determining the class which contains the cluster with the minimum euclidean distance to the new instance. Draw-Detect uses min-dist with a cluster number of 2, and scales the features using min-max scaling. The system also can reject instances as not belonging to any existing class, and has a null-rejection coefficient of 2.
Interface: The Draw-Detect interface was designed to combine an intuitive drawing environment with high degrees of visual feedback on the performance and response of the machine learning system, and is divided between a set of buttons and visualizations on the left and a drawing canvas on the right. The interface has two modes, train and test, which can be toggled between with a set of buttons at the top of the window.
Drawing Tools: Both modes contain a set of drawing tools that allow the user to create vector shapes. The draw tool allows the user can draw line segments by clicking to place individual points, or dragging to create many points at once. The move and delete buttons allow users to move individual points by clicking and dragging them with the mouse, or to delete individual points by clicking on them. The clear button clears the canvas of all points.
Training Mode: The training mode allows a user to draw, label and save example shapes. When the user creates a shape the feature section shows a set of bar graphs which update in real-time to display the current features for the shape as it is drawn and modified. The color of any single bar varies between shades of white to red, to indicate the degree to which this feature differs from the same feature of the prior shape (Red being extremely different, white being minimally different). For any shape, the user can select a training label number, and define a plain text label for it by entering a name in a text box. Once a shape is on the canvas (with a minimum of two points), it can be saved as a labeled example, by selecting the “save example button”. Mistakenly saved examples can be removed by hitting the “delete last” button. The user can save as many examples in as many different categories as they choose. As each example is saved, the interface displays an updated listing of the number of examples saved to each class.
Testing Mode: When the user switches to the testing mode, the system automatically trains the model with the current examples. In the testing mode, any shape drawn by the user is automatically classified in real time. The display in the testing mode lists the current class label predicted for the user’s shape, the likelihood associated with that class, and the distances and likelihoods for all other classes in the form of bar graphs and numerical readouts. If the shape is not recognized correctly, the user can toggle back to the training mode and save it as an example under the correct label.
Future Directions:The system can recognize fairly sophisticated drawings with a high degree of variation, given the simple drawing interface. The more classes that are defined however; the more effort a user may have to spend in refining their original classes so that they can be effectively distinguished. Because Draw-Detect is a drawing tool, the interface is designed to encourage a natural and methodic flow between drawing, training, and testing. I used the tool with a stylus and tablet, instead of a mouse, which allowed for more control and variation in the drawings. There are two possible directions to develop this tool in. The first is as a generative assistant wherein the user trains the tool in a class, and then can auto-generate drawings that correspond to the aesthetics and features of the original class, but are unique in themselves. This direction would probably require using an alternative technique such as an evolutionary algorithm. The other direction would be to develop the tool as an evaluative assistant for a digital artist. The tool could “learn” the artist’s style over time, and provide a form feedback on visual qualities and patterns in their work, and suggest alternative directions or techniques. Both of these directions would be challenging to pursue and would push the limits of interactive machine learning. They would also require developing more sophisticated drawing tools, and better forms of visualization to inform the user of how their drawings change over time.
Draw-Detect was created using Processing. The machine learning was implemented using the GUI version of the Gesture Recognition Toolkit, and communicating via the OSC Processing library. The interface was created using a combination of the ControlP5 Processing library and custom code. All other aspects of the tool I developed myself. The source is available at:
To operate the tool, you must install Processing, and download the GRT GUI using the above link. Draw-Detect is packaged as a processing library. Download the library here, unzip it and move it to your Processing libraries folder. You must also install the OSC and ControlP5 libraries if you do not already have them. Restart Processing if it is already open, and then go to File->Examples. Scroll down to the GRT folder, and open the draw-detect sketch. Open the GRT GUI and then run Draw-Detect from processing. Everything else can be controlled directly through Draw-Detect. The source of the Draw-Detect library is available here: http://github.com/pixelmaid/DrawDetect_Processing