What is OmniParser?

OmniParser is a cutting-edge approach that transforms user interface screenshots into organized components, significantly enhancing the precision of multimodal models such as GPT-4 in performing actions that correspond accurately to designated areas of the interface. This technique is particularly adept at identifying interactive icons within user interfaces and understanding the significance of various elements captured in a screenshot, thus connecting desired actions with the correct on-screen locations. To support this operation, OmniParser curates a dataset for the detection of interactable icons, consisting of 67,000 unique screenshot images, each meticulously annotated with bounding boxes around the interactable icons derived from DOM trees. In addition, it employs a collection of 7,000 icon-description pairs to fine-tune a captioning model aimed at extracting the functional meanings of the recognized elements. Evaluation against a range of benchmarks, including SeeClick, Mind2Web, and AITW, indicates that OmniParser outperforms the GPT-4V baselines, showcasing its efficacy even when relying exclusively on screenshot data without additional context. This significant progression not only boosts the interaction capabilities of AI models but also fosters the development of more seamless and intuitive user experiences across digital platforms. As a result, OmniParser stands to redefine the way users engage with technology, making interactions simpler and more efficient.

Screenshots and Video

OmniParser Screenshot 1

Company Facts

Company Name:
Microsoft
Date Founded:
1975
Company Location:
United States
Company Website:
microsoft.github.io/OmniParser/

Product Details

Deployment
SaaS
Training Options
Documentation Hub
Video Library
Support
Web-Based Support

Product Details

Target Company Sizes
Individual
1-10
11-50
51-200
201-500
501-1000
1001-5000
5001-10000
10001+
Target Organization Types
Mid Size Business
Small Business
Enterprise
Freelance
Nonprofit
Government
Startup
Supported Languages
English

OmniParser Categories and Features

More OmniParser Categories