Most interactive UI applications today are based on an event-based programming model in which user interactions are represented as events. The main event loop then dispatches these events to registered event handlers that change both the internal application state as well as the visible user interface. However, imperative updates to a stateful user interface can results in data dependencies that are difficult to keep consistent. Therefore, patterns such as Reactive Programming  and Model-View-Controller  have been adopted that separate the handle code for updating the application state from the render code for generating the output. Figure 1 illustrates how this programming model processes user interactions.
While common user interactions only affect the application state and output, live programming environments like Smalltalk  and TouchDevelop  also allow changes to the code of a running application in order to provide earlier feedback to the programmer for both debugging and development. Large code updates may require a restart or manual data migration but smaller code updates can often be applied automatically while retaining the current application state. In contrast to regular user interactions performed through the application’s user interface, these live code updates are performed in the source code editor of the development environment (see Figure 2).
In this paper, we introduce a new kind of user/developer interaction that extends live programming with UI interactions. Instead of supplying a change to the source code, the current user interface can modified through direct manipulation  to change the running code. The changed user interface serves as example for the intended output of rendering the current state. Depending on the domain and programming language, there are various different programming-by-example techniques [12, 7] to ‘repair’ or ‘synthesize’ a new program whose rendered output partially or fully conforms to the provided output example. To better describe the general design space of these systems, as illustrated in Figure 3, we also include a minimal formalization.
To summarize, the contributions of this paper are
- a new kind of user interaction that combines live programming of stateful event-based applications with programming by example based on direct manipulation of the user interface,
- a formal definition of such systems, and
2 Related Work
The term live programming is used in different contexts and can denote the act of programming as part of a live art or music performance or the feature of programming environment to continuously evaluate expressions and displaying results alongside code. However, this research focusses on live programming as the ability to change the code of a running stateful application without restarting it, also known as hot-swapping or dynamic software update , while providing immediate and continuous feedback about these code changes. Such live programming systems were described and motivated by Hancock  and further explored by systems like SuperGlue, which uses dynamic inheritance and explicit FRP signals , and Elm, which demonstrates live programming and time traveling with first-order FRP .
The solution described in this paper is based on earlier work on live programming for event-based systems  and generally follows the live programming technique for TouchDevelop which requires UI rendering and stateful computation to be separated and prohibits function values (closures) in the application state to allow the state and the code to be updated independently . More recent work outlined the possible design space between live programming systems that resume computation (with a possibly inconsistent state) and systems that record and replay execution , as well as introducing managed time as concept for supporting both live programming and time travel .
Programming-by-example allows users to author programs by providing examples instead of writing source code. Prior work on programming-by-example and programming-by-demonstration ranges from domain-specific macro systems, visual programming languages and inference of string processing rules [12, 7] to depth-limited generate-and-test approaches for general-purpose programming languages. Most noteworthy, CodeHint  synthesizes short Java code snippets at runtime based on user-provided queries. In order to synthesize larger code snippets, SMT solver-aided approaches may be a promising alternative .
Direct Manipulation  of the graphical user interface is a well-known form of user interaction with popular applications in Smalltalk , Morphic  and others. However, direct manipulation usually only affects the current state of visible objects. Recent work on Prodirect manipulation  shows how direct manipulation of SVG vector graphics can be used to automatically modify the SVG rendering code. The same idea has also been applied to the manipulation of string constants in PHP web applications . These two projects are most closely related to this paper but do not support live code updates of stateful applications without restarting the execution.
3 Live Programming for Event-based Languages
Live programming enables changes to the source code of running programs without restarting its execution. The inherent entanglement of state and code poses a challenge for live programming but approaches like functional reactive programming or a separation of rendering and event handling enable programmers to perform many live updates without manual data migration or restarting the execution.
To illustrate the challenges of live programming for general imperative applications, we
consider a simple interactive application that replaces keywords in a text according to a
Figure 4 shows a ‘traditional’ way of implementing this application with
imperative updates to register and modify event handlers (
In the context of live programming, changing
3.2 Separating Rendering from Event Handling
In the example shown in Figure 4, any change in the application state involves
corresponding updates in the output to ensure consistency. Programming models
such as Reactive Programming  and Model-View-Controller  solve this issue by
separating the rendering code from other stateful computation. Figure 5 shows an
implementation of the same application but the output will be generated by a pure
3.3 Live Code Updates
Separating rendering from event handling has the additional advantage that both
can be updated independently. Following the example from Section 3.1, changing
In contrast to changes to the rendering code, changes to the
event handling will not immediately be visible in the output.
Instead, the updated code will be used to handle all subsequent
In addition to live programming, separating rendering from event handling also has advantages for back-in-time debugging. By either recording events or snapshotting the application state between events, it is possible to travel back in time and visualize past execution states using the pure rendering mechanism. Combining back-in-time debugging with live programming enables the programmer to navigate both execution as well as the version history simultaneously.
Changes to the program code may involve added, modified or removed function definitions. Unfortunately, function values/closures in the application state cannot always be automatically transformed to match the new code, therefore closures are currently not allowed in the application state.
Additionally, an application that resumed execution with an updated event handler may behave different than an application that has been restarted. Depending on the development or debugging context, this may or may not be desirable.
Finally, changes to the initialization code of the program or the data type of the application state will likely result in an incompatibility and require either manual data migration or a restart.
4 Live Programming by Example
Live programming systems as described in the previous section enable program updates at runtime for changes made to the program source code. By separating rendering from event handling, the application output can be updated in reaction to code changes to provide visual feedback to the developer. Extending on this idea, the output of the program can also be used by the developer to make changes to the program — in particular in cases where direct manipulation is more convenient or intuitive than edits to the source code. This idea of live programming by example is also illustrated in Figure 3.
4.1 Formal Definition
Live programming by example is applicable to a wide range of applications and programming languages. To clarify the semantics and system requirements for the approach outlined in this paper, we model the system configuration and its transitions on a higher level of abstraction.
Figure 6 shows a formal definition of a system that supports live programming by
example. The input events
Following the programming model described in Section 3.2, the program code is
partitioned into event handling
There are three different high-level interactions with the system. Regular input
events are processed by evaluating the event handler and rendering the potentially
modified state (
Apart from language details of the implementation and the concrete UI
representation of the output, this definition highlights the design space for live
programming by example. Both the program synthesis technique (
Based on prior work on live programming environments for event-based languages ,
we implement a first prototype that supports live programming by example for
5.1 Example Interaction
Given an example application for replacing keywords in a text according to a
static replacement rule (see Section 3.1), this rule in the source code can be modified
through UI manipulation. Our approach depends on a separation of rendering
and event handling (see Section 3.2), so that changing the string constant
Interactions with the application’s user interface are usually handled by the application itself. In order to support live programming by example, the programming environment provides a way to halt execution 1 and enable a special interaction mode for the application’s user interface. Alternatively, it may be possible to reserve certain controls for this purpose, e.g. reserving a special meta key for direct manipulation or using halo controls .
Different forms of direct manipulation may be available (e.g. resizing or
reordering via drag and drop). In this example, the text of the static label (
Parts of the label text
5.2 String-origin Analysis
In order to support the interaction outlined in the previous section, the environment needs a mapping of strings in the output to their generating string literals in the source code. This mapping can be obtained by instrumenting the execution such that string values are tagged with provenance information. This form of dynamic analysis is closely related to dynamic taint analysis for information flow security.
As a first step, all string literals/constants in the source code are assigned a unique identifier by replacing them with constructor calls, yielding tagged string values with origin information.
Additionally, built-in unary and binary operations which cannot be instrumented are replaced with calls to custom functions implementing these operators.
Tagged string objects include a custom implementation of concatenation,
substring extraction, replacement and other common string operators (e.g.
5.3 Programming Environment Integration
The programming environment shown in Figure 7 supports two ways of manipulating the output for the purpose of live program synthesis. The developer can either edit the raw HTML code or manipulate the graphical user interface. The HTML representation has the advantage that all parts of the output including element names and attribute keys can easily be modified textually. Manipulation of the actual UI is limited to the plain text content of visible HTML elements but is highly intuitive and immediate.
Given a modified DOM tree, the program synthesis generally follows the following informal algorithm:
- Determine the previous output based on the current rendering code and application state.
- Compute the difference between the provided example and previous output.
(The modified DOM tree either has new characters inserted, existing
characters removed or both
5 5 Changes to the tree structure of the DOM output are not currently supported and remain future work..)
- Check origin information of the modified characters. Modifications to string parts that do not have origin information cannot be handled and will be suppressed.
- Determine source code location of the generating string literals using string origin information and a mapping from string literals to AST nodes.
- Use the source code location and computed offsets to insert or delete characters in the program source code.
- Recompile code and obtain a new rendering method.
- Render output using the modified rendering code and the current program state.
6 Discussion and Conclusions
The approach presented in this paper requires applications to separate rendering from event handling. Thereby, any updates to the rendering code can be synthesized and applied immediately. Updates to the event handling code will only affect subsequent events and therefore live programming by example is not directly applicable to the event handling code. A possible solution is to replay past events instead of resuming execution with the existing state. However, it is not always clear how many events have to be replayed as replaying all events may not be practical or desirable for long-running applications and replaying just the last event may not suffice.
Changing string literals in the program by manipulating text in the
output is a very simple implementation of live programming by example
and thereby avoids ambiguities that are common in program synthesis
Finally, the program environment presented in this paper mainly serves as a precursor for future systems that support more sophisticated program synthesis guided by more flexible forms of direct manipulation. Moreover, the approach still needs to be evaluated for larger applications and development tasks — potentially as part of a user study.
 Sebastian Burckhardt, Manuel Fahndrich, Peli de Halleux, Sean
McDirmid, Michal Moskal, Nikolai Tillmann, and Jun Kato. It’s
alive! continuous feedback in ui programming. In
 Ravi Chugh, Brian Hempel, Mitchell Spradlin, and Jacob Albers.
Programmatic and direct manipulation, together at last. In
 Evan Czaplicki and Stephen Chong. Asynchronous functional reactive
programming for guis. In
 Joel Galenson, Philip Reames, Rastislav Bodik, Björn Hartmann,
and Koushik Sen. Codehint: Dynamic and interactive synthesis of code
 Sumit Gulwani. Automating string processing in spreadsheets using
input-output examples. In
 Michael Hicks, Jonathan T. Moore, and Scott Nettles. Dynamic
software updating. In
 John H. Maloney and Randall B. Smith. Directness and liveness in
the morphic user interface construction environment. In
 Sean McDirmid. Living it up with a live programming language.
 Sean McDirmid. Usable live programming. In
 Sean McDirmid and Jonathan Edwards. Programming with managed
 Emina Torlak and Rastislav Bodik. Growing solver-aided languages
with rosette. In
 Tom Van Cutsem and Mark S. Miller. Proxies: Design principles
for robust object-oriented intercession apis. In
 Xiaoyin Wang, Lu Zhang, Tao Xie, Yingfei Xiong, and Hong Mei.
Automating presentation changes in dynamic web applications via
collaborative hybrid analysis. In