Nathan Koch

Major Studio I: Final Project – Design Proposal

Monday, November 6th 2017

Background and Purpose

The stories I love were written, designed, and coded by people, but many of my most memorable experiences emerged out of the fabric of city-life.

Stories and imaginary worlds, as we traditionally imagine them, are hand-crafted, populated painstakingly by authors or designers. What happens if, instead, we observe narrative agents in a persistent virtual space? Can happy accidents arrive through emergent systems?

I'm exploring and studying these techniques and associated algorithms in order to create and populate "generative worlds." My hope is that the combination of procedural content, algorithmically simulated behavior, and programmatic world-building can create something that is both like and entirely unlike a game; although it is built on game technologies and has the look and feel of a game, it isn't playable. There's a sense of place and users may infer their own story, but it will resist conventional narrative readings.

Context and Inspiration

Procedural content in games in particular can be defined as "algorithmic creation of game content with limited user input,” [1] and to date, it is an area of interest that has been driven primarily by the economics of large-scale game creation. For example, if game designers and artists mediate computer-generated work rather than create it by hand, they can produce more work in less time.

There's more to procedural content than efficiency. I'm not looking to speed up the traditional level design and AI behavior tasks that go into game creation; I want instead to create an observable bubble of simulated reality in which a set of pre-established parameters drive the motivations of virtual actors in unpredictable ways.

My approach has more in common with Brian Eno’s rules-based works like Discreet Music or Music for Airports than commercial game development. My hope is that I can create a piece that is effectively ambient storytelling instead of ambient music. These techniques have long been used in new media art, but usually the system is fully exposed for the viewer – it is the piece. By comparison, experimental music is rules-based, but those rules are a tool through which a composer creates new work that can exist on its own merits.

Emergent systems are larger than either music or the visual arts, however. Mitchel Resnick's book "Turtles, Termites, and Traffic Jams: Explorations in Massively Parallel Microworlds" shows real-world examples of emergent behavior. “Ant colonies, highway traffic, market economies, immune systems–in all of these systems, patterns are determined not by some centralized authority but by local interactions among decentralized components.” [2]

Ian Cheng – Emissaries. Image courtesy MoMA.

One particular precedent that steered me in the direction of simulation-as-storytelling-tools was Ian Cheng's Emissaries exhibition at MoMA PS1, which he described as “video games that play themselves.” [3] I found Emissaries striking and endlessly watchable, as a cast of AI characters played out their programmatic scripts in unpredictable ways.

Catching Cheng's exhibition before it closed in September inspired me to think about systems and simulations. What he and his team were able to accomplish was infinite, storytelling through winding something up and letting it go.

His work was steeped in video game tropes while living completely outside the context of games as we know them – narratives emerged secondarily out of a collection of colliding systems and algorithms. "Unlike conventional animations, whose story arcs are predetermined, Cheng’s digital worlds are driven by a technology in which narrative agents behave within coded parameters, but on their own terms. This means that the action is unpredictable and unfolds at a lifelike pace, which is to say it’s both absorbing and tedious.” [4]

Luigi Serafini – Codex Seraphinianus. Image courtesy Dimka Daily.

In an entirely different direction, a very different precedent for this project is the Codex Seraphinianus. [5] The Codex is a profoundly strange book dating back to the late 1970’s, an encyclopedia of an imaginary world documented extensively in an artificial language. It's filled with grotesque organic imagery, utterly unlike our own world but similar enough to evoke emotions ranging from disgust to wonder. Serafini effectively created a virtual world using entirely personal, analog methods, but we can see parallels with both Cheng's work and modern video games like Skyrim.

Alan Warburton – Primitives. Image courtesy Alan Warburton.

Alan Warburton’s Primitives series [6] explores the possibilities of programmatic, AI-generated behaviors applied to 3D forms and animation. He uses crowd-simulation techniques applied to modeled human figures to achieve results that are both grotesque and affecting.

Much of Warburton's work is critical of how the medium of 3D animation is driven almost entirely by commerce, reflecting on how Hollywood continues to push the boundaries of photorealism in the service of producing massive spectacles for crowd-pleasing movies. He uses the same tools as big-budget game and movie studios but with an avant-garde, subversive bent.

Intention and Creation

I'll be creating a persistent, ambient game world (or, as described above, an observable bubble of simulated reality) using behavioral simulation and rules-based AI techniques to create an explorable virtual space where an observer can watch these algorithms unfold in real-time.

I'm interested in whether as a designer I can effectively back into storytelling, by setting up a lightweight framework of space and character and letting the story emerge organically out of multiple systems interacting in unexpected ways. My intent is to observe the possibilities of story and setting derived from emergent systems, and expose my audience to this "ambient storytelling" as well.

The technology of games, much like the technology of 3D animation, has long been driven more by the commercial realities of large scale “AAA” titles than individual artists. I'm interested in reclaiming these tools for my own use, as an individual creator working apart from the goals of either large scale studios or solo indie game developers, to create a game-like object, or piece of toy software, completely separate from the usual sales-driven ecosystems like the PSN Store.

Over the course of this project I've read about and recreated a number of algorithms that may find a place in my project, found in sources ranging from Daniel Shiffman's "The Nature of Code" [7] to A.K. Dewdney's "The Magic Machine." [8]

Dinner Party Study. Image courtesy author.

I started with an initial study which recreated A.K. Dewdney's “Programmed Parties” (an algorithm from his column in Scientific American) in Python. "Programmed Parties" [9] simulates the guests at a dinner party, with each one being motivated to get closer or farther away from either each other or the food table. Although a fairly simple AI algorithm by today's standards, it created a seemingly ever-changing dynamic system.

Another study used Craig Reynolds' "flocking" algorithm [10] to drive a set of bird-like creatures around a screen according to a simple set of steering rules. When the birds intersect with sound-emitting clouds, they produce different notes, and, at times, harmonies.

From there I attempted to add more complexity, and created a prototype that used one of Reynolds' steering examples, [11] combined with a naive resource-gathering algorithm of my own creation. This prototype also contained a few different modes, including icon-based graphics, a visible text-based console exposing the motivations of the agents on-screen, and even simple text-to-speech audio.

Testing this prototype in-class, it became clear how powerful sound could be at grabbing observer attention. Moving forward I'm going to experiment with giving agents in this virtual world distinct voices and trigger audio specific to their individual interactions.

There is still more game AI and behavioral algorithm research to be done. I'm looking at A.K. Dewdney's “Palmiter Protozoa" [12] article in Scientific American, as well as the typical pathfinding and state machine algorithms found in modern games. [13] I'll also be reviewing “Utility AI,” known mostly from The Sims and other simulation-heavy games. [14] Depending on my choice of engine, I may also explore the Unity3D game engine’s own navigation [15] and machine learning [16] tools.

8-bit Look and Feel Study. Image courtesy author.

I've also begun some early studies into look-and-feel, including both 2D and 3D executions. I believe a compelling scenario can emerge from interesting rules and algorithms, but in the final work I hope to combine behavior and aesthetics in a unified whole.

My studies to date have each demonstrated the challenges and opportunities of implementing a single behavioral algorithm. Ultimately I'll be combining multiple algorithms and multiple agents in service to an unpredictable and emergent experience. I've also experimented with both 8-bit and 3D aesthetics, and multiple languages and creative coding frameworks, ranging from pure Python, to Processing, to openFrameworks.

I still have some questions around the final artifact I'll be creating, but I'm beginning to dial in a vision for a project with a deliberately low-fi aesthetic built in Unity, a tool with a long history in indie game development particularly.

I believe that by creating this simulation within a game engine, I can expand the boundaries of the medium, or at least expand audience perceptions of what is possible. Along the way, I'll learn about the algorithms and techniques that lie behind the curtain of AI programming in modern game development.

Values and Goals

Some design values I'll be bringing to this project, beyond the larger domains of systems and simulations:

No human interaction

This project should be completely unmediated by human hands. I want to see what's possible at the bounds of pure procedural design, rather than merely parametric.

Affecting ambience

The world should be able to create a sense of space that encourages exploration or restful observation.

Aesthetic poetry

The look and feel of this simulation should be aesthetically coherent and finished. It should feel less like a technical demo or prototype than a finished work.

Game-like objects

This world should be playable to the extent that a user who doesn’t understand the underlying concept can still be engaged by exploring the experience.

I believe that AI-assisted world-building and generative stories may be able to transform the roles of game designers, advertisers, filmmakers, and even authors. My hope is that this project can demonstrate what's possible in algorithmic storytelling, and inspire additional work in the area of simulated virtual worlds.

Footnotes and Bibliography

  • [1] Shaker, Noor, Julian Togelius, and Mark J. Nelson. Procedural content generation in games. Cham, Switzerland: Springer, 2016: 3.
  • [2] Resnick, Mitchel. Turtles, termites, and traffic jams: Explorations in massively parallel microworlds. Cambridge, MA: Bradford Book, 1997: 1
  • [3] Cheng, Ian. Emissaries. 2016. New York, NY. https://www.moma.org/calendar/exhibitions/3656
  • [4] Scott, Andrea K. "Watch the Absorbing and Tedious Simulations of Ian Cheng." The New Yorker, May 16, 2017
  • [5] Serafini, Luigi. Codex Seraphinianus. New York, NY: Rizzoli, 2013
  • [6] Warburton, Alan. Primitives. 2016. Laboral, Gijon, Spain. http://alanwarburton.co.uk
  • [7] Shiffman, Daniel. The nature of code. Self-published, 2012
  • [8] Dewdney, A. K. The magic machine: more computer recreations columns from Scientific American. W.H. Freeman, 1990: 151-160
  • [9] Dewdney, A. K. The magic machine: more computer recreations columns from Scientific American. W.H. Freeman, 1990: 151-160
  • [10] Shiffman, Daniel. The nature of code. Self-published, 2012: 308-314
  • [11] Shiffman, Daniel. The nature of code. Self-published, 2012: 262-269
  • [12] Dewdney, A. K. The magic machine: more computer recreations columns from Scientific American. W.H. Freeman, 1990: 161-178
  • [13] Mahdav, Sanjay. Game Programming Algorithms and Techniques. Addison-Wesley, 2014: 180-202
  • [14] https://alastaira.wordpress.com/2013/01/25/at-a-glance-functions-for-modelling-utility-based-game-ai/
  • [15] https://docs.unity3d.com/Manual/nav-BuildingNavMesh.html
  • [16] https://blogs.unity3d.com/2017/09/19/introducing-unity-machine-learning-agents/

Additional editing courtesy of Ting Cheng, Wen-Chi Huang, Yujie Jiang and Terry Koch.