The thing on the screen

is supposed to be

the actual thing


by David Ungar and Randall B. Smith

We were delighted to see a workshop dedicated to Live Programming, as we have been fortunate to have been programming in live environments since 1969, and building them since 1985. Since those early days, many have created environments that rapidly respond to what people do, a tremendous benefit. Systems do indeed feel live when they react on their own to stay a consistent part of the world. We propose here a broader definition of liveness, one that emerges from the design principle of making everything seem real. (After all, reality itself is live.) In any such system, the user must always see the true state of the actual thing, and so the reactive character of liveness emerges naturally. But a deeper feeling of liveness emerges when the user is immersed in an entire world that maintains a consistent physics., when the thing on the screen is supposed to be the actual thing.

Any good user interface offloads the cognitive burden of the user, by harnessing his precognitive abilities, and passes the squint test, the notion that what is most visible when you squint should be communicating the most important information about the system. Physical reality passes both tests: by the time we reach the age of three, we are well wired to process the visual information that real objects around us supply. An oncoming car is visible even when you squint! But many “live environments” around today fail this test: they show indirect tools such as browsers and inspectors. As a consequence of indirection, they fail to respect object identity, forcing the user to keep track of complex data structures with pencil and paper.

We propose to illustrate these points by demonstrating three phases in the evolution of our work: the Alternate Reality Kit, the Self UI1 IDE, and the Self UI2 IDE. First, the inspiration for reality in live environments came to Ungar from Smith’s Alternate Reality Kit, which Smith was inspired to build after moving to the Smalltalk group from a computer and video game company, Atari. ARK was a novel programming system: everything was represented as a physical object with mass and that cast a shadow when picked up, causal connections were represented as real wires, and signals as animations. Here is a brief portion of the video that Smith made:

Video clip of The Alternate Reality Kit

When Smith wrote ARK, he had to disguise the abstract nature of Smalltalk classes in order to present a concrete, physical manifestation of programs. When designing the object-oriented language Self, we had the opportunity to create a complete programming experience, from the language to the user interface, and so could carry the principles of concreteness and physicality throughout. This unity of design allowed the entities depicted in Self’s first interface and IDE, called UI1, to closely correspond with the elements in a Self program.

With Ungar at Stanford, Chang, Chambers, Lee, and Hölzle built a virtual machine and the UI1 IDE for Self. A notable innovation emerged as a consequence of UI1‘s concreteness principle: direct manipulation replaced the need for tools. An object was depicted as a real thing, not as a hidden abstraction revealed by some “inspector.” The contents of an object (including code) were shown as living inside of the object, and object identity was respected. That is, when calling for an object that was already on the screen, instead of drawing a new picture of that object, as Smalltalk had done with its inspectors, UI1 drew an arrow to the existing object. The picture on the screen was supposed to be the actual object. In service of the creation of the illusion of reality, we had to depict motion of solid objects in a realistic and legible way. So UI1 was one of, if not the first, IDE to exploit the principles of cartoon animation: motion blur, anticipation, follow-through, and slow-in-and-slow-out. We can do a live demo at the workshop. Our second video was done by Bay-Wei Chang for an award-winning UIST paper:

Self’S UI 1

Thanks to Sun Microsystems Laboratories, Randy and Dave reunited, Lars Bak, John Maloney, and Mario Wolczko joined the group, and together with student Ole Agesen we took Self from an experimental system into one capable of supporting large-scale programming. Self 4.0 featured a new IDE based on a new UI architecture, Morphic.

Smith took the principle of liveness even further when he designed and (with Maloney) implemented Morphic. Every graphical element in the interface became not merely a picture on the screen, but an actual thing that the user could manipulate directly. For example, you can directly yank a button out of a menu and reattach it anywhere.  We are pleased that the Morphic ideas have been ported to several other languages, including Smalltalk [], Python [], Ruby on Rails [], and JavaScript[ and]. Morphic plays a key role in the liveness of Self’s UI2, so as other languages’ IDEs incorporate Morphic they take a big step towards the goals we set forth. However, almost all of these systems (except Avocado []) remain browser-based. Smith demonstrates Morphic and UI2 in our last video:

Self’s UI 2

In conclusion, although lessons of Smalltalk have been widely learned, we feel that the physical reality lessons of ARK, UI1, and UI2 remain under appreciated in light of the vast capabilities of today’s systems, which put a brain the size of a planet in your pocket. The thing on the screen is supposed to be the actual thing.


The History of Self, David Ungar, Randall B. Smith. ACM HOPL-III, 2007. Proc. of the Third ACM SIGPLAN History of Programming Languages Conference (HOPL-III), B. Ryder, B, Hailpern (Eds.), San Diego, California, USA, 9-10 June 2007. Paper and video at

Dynamic Languages (in Reactive Environments) Unleash Creativity, D Ungar, E Ernst. Point/counterpoint essay in IEEE Software, 2007.

From Kansas to Oz: Collaborative Debugging When a Shared World Breaks, Randall B. Smith, Mario Wolczko, David Ungar,. CACM April 1997.

Debugging and the Experience of Immediacy, David Ungar, Henry Lieberman, Christopher Fry. CACM April 1997.

Annotating Objects for Transport to Other Worlds. David Ungar,  OOPSLA’95.

Programming as an Experience: The Inspiration for Self. Randall B. Smith and David Ungar,  Invited paper for ECOOP’95. Reprinted in Prototype-Based Programming: Concepts, Languages and Applications, James Noble, I. Moore, A. Taivalsaari, eds. Springer-Verlag.

Self: The Video. Randall B. Smith. (Special appearances by: David Ungar, Mario Wolczko). Sun Microsystems Laboratories, 1995. Available at

Self 4.0 User Interface: Manifesting a System-wide Vision of Concreteness, Uniformity, and Flexibility. Randall B. Smith, John Maloney, and David Ungar,  OOPSLA’95.

Getting close to objects: Object-focused programming environments, Bay-Wei Chang, David Ungar, Randy Smith. Visual Object Oriented Programming, Margaret Burnett, Adele Goldberg, & Ted Lewis, eds., Prentice-Hall, 1995.

Animation: From Cartoons to the User Interface, Bay-Wei Chang and David Ungar. UIST’93. Lasting Impact Award, UIST’04.

Directly Experiencing Objects, Bay-Wei Chang and David Ungar. OOPSLA’93 Workshop on Visual Object-Oriented Programming.

The Use-Mention Perspective on Programming for the Interface, Randall B. Smith, David Ungar, and Bay-Wei Chang. In Computer Languages for Programming User Interface Software, a workshop at the ACM SIGCHI’91 conference, Brad A. Myers, organizer and editor. Reprinted in Languages for Developing User Interfaces, Brad Myers, ed., Jones and Bartlett, London, 1992.

SELF: The Power of Simplicity, David Ungar and Randall B. Smith. OOPSLA’87. A revised version appeared in the Journal of Lisp and Symbolic Computation, 4(3), Kluwer Academic Publishers, June, 1991, pp. 187-206. Also IEEE COMPCON’88, San Francisco, CA, 3/88. Presented along with an introduction to object-oriented programming at IBM internal programming languages conference, Almaden, CA, 1988, and at IBM Santa Theresa, CA, 1988.