LADDER, a sketching language for user interface develoeprs

Bibliography Information:

Hammond, Davis. "LADDER, a sketching language for user interface developers". Computers & Graphics 29.  Pages 518-532. 2005.

URL:

http://www.sciencedirect.com/science/article/pii/S0097849305000865

LADDER was designed around the increasing ubiquity and extended use cases for applied sketch recognition. In areas for sketch recognition where the user and creator are both expected to share some knowledge of a particular domain (such as a characters in a new language, mathematical symbols, or free-hand physics diagrams), the requirement for someone to create a sketch recognition system in the past would require the "instructor" (that is, the individual providing the domain-specific materials) to have additional significant experience in the concepts behind sketch recognition. LADDER was built specifically for user-interface developers to more easily create sketch recognition domains without the need for expertise in the field of recognition. Instead they would be asked to describe how the diagrams in the domain are drawn, displayed and edited. To this end, the LADDER system handles shape and editing recognizers, shape exhibitors (that is, rules in how shapes are displayed to the screen), and an extensive sketch recognition system. A code generator was built so these descriptions can be input into the base recognition system.

LADDER's system relies mostly on recognizing as few strokes as possible (such that an "artistic rendition" of a cat, complete with thousands of small strokes for added facial and shadow detail, etc., would not be considered), relying instead on the most essential geometric rules of a shape to help craft the definition of a shape. Shapes are identified as a list of components (for instance, an arrow is identified as being "built from three lines"), constraints that define the relationships between the components, a set of aliases that simply elements in the description, editing behaviors that differentiate user input between "drawing" and "editing" actions (cut and paste, translation, rubber-band resizing, etc.), and display methods such as beautification of input strokes (strokes intended by the user to be lines but exhibiting minor hand tremors are "redrawn" as perfectly straight lines).

These recognized shapes are then identified into shape "groups" that make up more complex shapes, usually these consisting of domain-specific shapes unique only to one particular field (such as shapes depicting chemistry diagrams, Kanji characters, etc.) The language that describes LADDER shapes is based on a hierarchical system, providing descriptors such as the bounding box, center point, width and height, etc. LADDER is has several predefined constraints built into it, such as perpendicular, parallel, sameSide, oppositeSide, etc. Some constraints are valid only in a particular direction, and some directions can be specified in degrees relative to other shapes. The researchers found that a description system based on orientation is most useful to content providers that would use a system like LADDER.

Recognition of domain shapes happens over the course of several bottom-up data triggers where a recognized shape in the drawing can represent a particular "fact about the world". This shape recognition uses the Jess rule-based system that performs "clean up" on these shapes. These recognized shapes are added into the Jess system for later recognition. Additional similar rules exist on user input intended to perform edits on existing shapes, except these rules are applied to the user input rather than the resulting sketch.

I believe the work provided in LADDER is an extremely important advancement on the field of sketch recognition, since it works a great deal toward 1) contextualizing sketch input that makes this technology useful for real-world applications and 2) making it easier for content providers to create and support their own domains without expertise required in the field of sketch recognition. This system and its underlying principles have been used as the driving force behind much of the projects in the Sketch Recognition Lab.