ACM UIST 2021 “Trip” Report

Latest user Interface technologies for programming support

I attended the 34th ACM Symposium on User Interface Software and Technology (UIST 2021), the premier forum for innovations in human-computer interfaces, in the week of October 11th, 2021. It was a fully virtual conference due to COVID. UIST covers many topics, but this “trip” report is focused on the latest research related to computer programming I saw and read at the conference.

General impressions

Before I dive into specific papers, I’d like to offer my general impressions of this year’s UIST.

The virtual conference experience

I thought the virtual conference worked out well. One unique thing the conference organizers did was a 24-hour schedule with each paper presented twice to accommodate attendees from different time zones. Having learned lessons from attending virtual events in the past, I also cleared my work schedule for the duration of the conference as if I was traveling for an in-person event. As a result, I was able to see almost everything I was interested in at UIST 2021, and I even had quite a few live interactions with presenters and other attendees.

Four different communication tools were used to support the conference. Zoom was the main live presentation system, while Discord served as both the live communication and asynchronous communication platform for attendees and presenters (as shown in the screenshot below). Since I no longer need to line up behind a microphone quickly after a talk to ask a question, I actually engaged more with presenters during live sessions this time than many off-line conferences I attended in the past.

Asking live questions through the conference’s Discord server

Coffee breaks and social hours were conducted over another platform called Ohyay which allows you to “sit down” in a virtual lounge (see the picture below). I found it a little awkward but probably no more awkward than walking up to a group of strangers at a physical conference. So it’s a win in general. The last social technology the conference used was who2chat, a program that matches attendees based on keywords mined from their research profiles. It helped me set up a 1:1 chat with someone who shared my research interest.

People “sit down” and chat during social hours on the Ohyay platform. Permissions have been obtained to show people’s names and faces in the photo.

The big ideas that keep coming back

Due to my area of focus, I saw about 30% of the conference. A couple of old but powerful ideas in Human-Computer Interaction kept showing up in this latest batch of UIST papers. I thought it might be useful to give them a very quick introduction here.

• Programming by demonstration (PbD): According to Wikipedia, “programming by demonstration (PbD) is an end-user development technique for teaching a computer or a robot new behaviors by demonstrating the task to transfer directly instead of programming it through machine commands.” PbD sometimes is also referred to as programming by example. A PbD system can learn program behavior through a rule-based system or an ML-based system. A number of papers presented at UIST this year, including reCode and Rapido, describe new applications of this approach.
• Mixed-initiative system: This is a quite broad approach used in designing workflows where both the human operator and the intelligent agent take initiatives as appropriate to carry out a usually complex task. A mixed-initiative system often requires a careful choreography of direct manipulation and automation. In 2016, Microsoft Research published some useful guidelines on designing such systems.

Next, I will give a quick tour of the programming-related papers I saw at UIST 2021. The screenshots and everything in quotation marks below comes from the paper being described unless identified otherwise.

Papers about programming & prototyping support

reCode: A Lightweight Find-and-Replace Interaction in the IDE for Transforming Code by Example

📺 Presentation | 📄 Paper

Problem: “Software developers frequently confront a recurring challenge of making code transformations — similar but not entirely identical code changes in many places — in their integrated development environments.”

Solution: The authors “built an extension for Visual Studio Code, called reCode. reCode improves the familiar find-and-replace experience by allowing the developer to specify a straightforward search term to identify relevant locations, and then demonstrate their intended changes by simply typing a change directly in the editor. Using programming by example, reCode automatically learns a more general code transformation and displays these transformations as before-and-after differences inline, with clickable actions to interactively accept, reject, or refine the proposed changes.”

Key idea: reCode is a mixed-initiative system that leverages both user demonstrations of desired code transformations and a code synthesis engine called ReFazer, which is capable of generating program transformations from examples. reCode provides a workflow to provide examples for the synthesis engine and apply the generated changes back to the source code.

My thought bubble:

• Can we use reCode and Refazer for migrating API and language breaking changes?
• Can we use these techniques to expand editing assists offered by IDEs?

Unravel: A Fluent Code Explorer for Data Wrangling

📺 Presentation | 📄 Paper

Problem: Data scientists have widely adopted the fluent interface (i.e., method chaining) for composing data wrangling code. However, debugging such code requires unraveling the chain in order to inspect intermediate results.

Solution: The authors “designed a tool called Unravel that enables structural edits via drag-and-drop and toggle switch interactions to help data scientists explore and understand fluent code.” Unravel is specific for the RStudio statistical computing environment.

Key ideas:

• Color coding different types of changes each line leads to: It’s easy to tell the effect of each method on the final results.
• Describing intermediate results: There are many different ways an intermediate result, which is a data frame in this case, can be described. The authors leveraged domain knowledge to focus on three things: 1) dimensional changes, 2) important column variables, and 3) subtle changes made by the function.
• Live re-evaluation of code after toggling specific lines on/off: The user can technically do this by hand, but Unravel made it both more convenient and fast. Behind the scenes, the system split the code into multiple code snippets that represent each of the chains and went through different combinations in advance.

My thought bubble:

• The fluent interface used in data science has a lot of similarities with UI code written with Flutter or Jetpack Compose. How can we apply this technique to debugging UI code? One possibility is using something like this for layout debugging. The tool can surface how each widget in the tree changes the layout constraints and allow the user to explore the effect of turning off widgets that might be causing problems and provisionally inserting widgets that can potentially fix the layout problem (e.g., an Expanded widget in Flutter).
• Another possibility is to use this technique for performance debugging. We’ve seen experienced developers use a similar but manual process of elimination to find the root cause of a performance problem.

Weaving Schematics and Code: Interactive Visual Editing for Hardware Description Languages

📺 Presentation | 📄 Paper

Problem: A hardware description language (HDL) provides benefits such as abstraction and automation, but it’s not friendly to hardware designers who’re used to graphical workflows.

Solution: In this work, the authors “investigate an IDE approach to provide a graphical editor for a board-level circuit design HDL.”

Key ideas:

• Code is the source of the truth to preserve its full power, but edits to the code and to its graphical representation (i.e., the block diagram visualization) are synchronized both ways.
• User study shows that the system supports diverse workflows: half of the participants favored the code-based workflow, while the other half favored the GUI-based workflow

My thought bubble: To everyone following the evolution of GUI toolkits, it’s unsurprising to see circuit and chip design tools are moving towards a code-first approach. This has happened with the rise of declarative UI frameworks such as Flutter, SwiftUI, and Jetpack Compose. However, what’s often missing is a complementary visual programming workflow that a design-oriented audience is more accustomed to.

Umitation: Retargeting UI Behavior Examples for Website Design

📺 Presentation | 📄 Paper

Problem: “Good artists copy, great artists steal.” However, it’s difficult to figure out how a particular UI behavior is implemented on the web and adapt it for a different website.

Solution: The authors proposed “Umitation, a system that helps designers extract, edit, and adapt example front-end UI behaviors to target websites.” In my opinion, it’s kind of like a “paint format” button for retargeting UI behaviors.

Key ideas:

• Extracting UI behavior by demonstration: Umitation provides a visual workflow to specify the behavior of interest and provides useful abstractions to guide the user through the selection and retargeting process. Under the hood, it uses web APIs such as MutationObservers to capture the resulting UI changes.
• Visualizing UI behavior: Umitation features a minimalist but useful visualization of the captured UI behavior. It uses a square to represent the response object, and visualizes changes to its attributes in a 2D diagram (see the figure on the side).

My thought bubble: This technique has the potential to bring example-centric programming to the next level by reducing the need of seeing and understanding the example code being borrowed and supporting the adaptation of the example. It’s not there yet, because the current implementation of Umitation doesn’t trace the source code of the original website nor synthesize new code for the target website.

Ten Million Users and Ten Years Later: Python Tutor’s Design Guidelines for Building Scalable and Sustainable Research Software in Academia

📺 Presentation | 📄 Paper

Problem: Python Tutor was built and sustained by a single academic over the last 10 years. This paper addresses the problem of building sustainable research software with widespread usage in academia. While not directly related to folks working on products in the industry, lessons learned from Python Tutor helps answer a similar question many of us run into: how to sustain a large-scale but underfunded and understaffed software project?

Solution: The author proposed 10 design guidelines as his answer to the above question. Several guidelines are related to preventing feature creeps, including a controversial statement of “Ignore most users to sustain your project long-term.” There is also some technical advice such as “Be Stateless” and avoiding hosting and moderating user-generated content.

My thought bubble: The author accepted very few feature requests over ten years. While his advice of “ignoring most users” might be out of pragmatic constraints, I found it agreeable to some extent that product teams need to be selective when considering user feedback.

Sporq: An Interactive Environment for Exploring Code Using Query-by-Example

📺 Presentation | 📄 Paper

Problem: “Programmers still find it difficult to conveniently analyze their code for custom patterns”

Solution: The paper presents “Sporq, a tool that allows developers to mine their codebases for a range of patterns, including bugs, code smells, and violations of coding standards. Sporq offers an interactive environment in which the user highlights program elements, and the system responds by identifying other parts of the codebase with similar patterns.”

Key ideas:

• Searching code as querying a relational database: internally, the system builds a relational database to represent the facts in the user’s code base.
• Synthesizing queries by examples: the user usually starts with positive examples and then refine the query by identifying negative examples from the predicted results.

GenLine and GenForm: Two Tools for Interacting with Generative Language Models in a Code Editor

📺 Presentation | 📄 Paper

Problem: The paper tackles the problem of authoring generative language macros (“genmacros”) to guide a large language model for the purpose of generating source code from a natural language description of programming intent.

Solution: The authors proposed two GUI tools for this task: 1) GenLine, which “allows users to pass inline requests to a genmacro using a double-bracket notion (e.g., “[[html: Make an OK button]]”); and 2) GenForm: a form-based UI.

My thought bubble:

• What I found really cool was that GenLine allows the user to wrap regular source code within natural language macros (see the image above). The system then replaces the macro with generated source code but preserving the wrapping relationship.
• The macro in GenLine appears to be ephemeral. The user cannot revisit a macro after it’s been replaced by the source code it generates. This workflow is very different from macros in static metaprogramming, where the macros are often persistent and get checked into version control. The generated code is more often to be ephemeral.

Idyll Studio: A Structured Editor for Authoring Interactive & Data-Driven Articles

📺 Presentation | 📄 Paper

Problem: “Interactive articles are an effective medium of communication in education, journalism, and scientific publishing, yet are created using complex general-purpose programming tools.”

Solution: The authors “present Idyll Studio, a structured editor for authoring and publishing interactive and data-driven articles”

Key ideas:

• Structured editing: instead of modifying a program directly, all edits are done through the affordances in the GUI (e.g., insertion points) to ensure every change is syntactically valid.
• Domain-specific programming model: the tool is designed around four core concepts specific for producing interactive documents: components, text, state, and styles.

My thought bubble:

• Interactive documents represent increasingly blurring boundaries across documents, applications, and datasets. How should UI toolkit developers prepare for these kinds of mashed-up experiences?
• The tool reminds me of Webstrates, a more general tool for mashing up different kinds of media into interactive documents.

Rapido: Prototyping Interactive AR Experiences through Programming by Demonstration

📺 Presentation | 📄 Paper

Problem: Creating fully interactive mobile AR prototypes is hard.

Solution: The authors present “Rapido, a novel early-stage prototyping tool to create fully interactive mobile AR prototypes from non-interactive video prototypes using Programming by Demonstration (PoD).”

Key idea: Rapido infers an initial state machine from the metadata captured in the process of creating a video prototype. It then allows the designer to test, refine, and tweak the interaction rules.

My thought bubble:

• Rapido’s design leverages the workflow of creating video prototypes, which is already familiar to AR designers and then gently introduces them to a form of action trigger programming.
• In the context of conventional GUI programming, the PoD technique used in this tool could potentially be useful for authoring app navigation and animation.

Other programming-related papers

There are a few other papers about programming at UIST 2021, but I took less notes for them. Nonetheless, I list them below for completeness:

• Marcelle: Composing Interactive Machine Learning Workflows and Interfaces (📺 Presentation, 📄 Paper)
• Situated Live Programming for Human-Robot Collaboration (📺 Presentation, 📄 Paper)
• ETNA: Harvesting Action Graphs from Websites (📺 Presentation, 📄 Paper)
• Shapir: Standardizing and Democratizing Access to Web APIs (📺 Presentation, 📄 Paper)
• Assuage: Assembly Synthesis Using A Guided Exploration (📺 Presentation, 📄 Paper)

Final thoughts

The virtual conference experience has come a long way since the start of the pandemic. Attending UIST 2021 from my desk was a fun, engaging, and adequately social experience. Was it as good as an in-person conference? No, but I hope some of the best aspects of virtual conferences, such as convenience and low cost, can continue as many events start going back to their “normal” setups in the near future.