Recent research has investigated the use of Large Language Models (LLMs) to engineer Graphical User Interfaces (GUIs) using Generative Artificial Intelligence (GenAI). We have developed an LLM-based tool, called Instigator, for the purpose of facilitating the search for GUI layouts in this study. The purpose of our study was to determine whether such a tool would enhance serendipity in the design process without introducing irrelevant layouts.
An evaluation of Instigator was conducted under controlled conditions and indicated a high correlation between the rankings of GUI layouts produced by Instigator and the rankings given by the participants for the same layouts. The study involved 34 practitioners who evaluated various scales according to their perceived quality of the tool. This feedback offered insightful perspectives for enhancing future tools like Instigator.
In the course of their design process, GUI practitioners seek out proven or recommended GUI layouts from various sources. With our tool, practitioners can generate GUI layouts from a textual instruction, edit and combine these layouts to produce a final GUI that can be rendered in high fidelity and converted into code. The tool is powered by GenAI and LLMs, showcasing the potential of these technologies in software engineering, especially in GUI design.
Our approach relied on a Web Parser that was built using Puppeteer and AWS Cloud Services to parse up to 4.5k websites per minute. This provided high-level information about the nodes of each website, allowing us to segment the GUI layout into elements and identify the GUI elements using computer-vision software libraries. In order to determine the probability of a high-quality website, we extracted metadata from each UI and built a cleaner dataset with the assistance of a machine learning model.
Instigator’s implementation and evaluation in the GUI layout design process showed promising results. However, we also identified areas for improvement. In the future, it may be possible to incorporate context-dependent constraints, demonstrate examples progressively from novice to expert levels, and employ reinforcement learning for layout suggestions. Moreover, instigator’s ability to provide suitable GUI suggestions could be enhanced by the introduction of diversification metrics.
In conclusion, our research demonstrates the potential of LLMs in the field of GUI design. The Instigator tool showcases the effective generation and composition of GUI layouts from textual instructions, a significant contribution to this field. This opens up exciting future research avenues and the potential for more advanced tools inspired by our initial study.