Video details

"Go Circuit: Distributing the Go Language and Runtime" by Petar Maymounkov


The Go Circuit Project
The circuit reduces the human development and sustenance costs of complex massively-scaled systems nearly to the level of their single-process counterparts. It is a combination of proven ideas from the Erlang ecosystem of distributed embedded devices and Go's ecosystem of Internet application development.
The circuit is an extension to the Go Language and Runtime. The circuit language is a superset of the Go language in that it includes one additional operator "spawn", designated for executing functions in newly spawned goroutines (Go's light-weight threads) on a specified remote, out-of-process runtime.
In the sense that remote control is available within the source environment, the circuit can be likened to an imperative version of Erlang. Using the circuit one is able to write complex distributed applications — involving multiple types of collaborating processes — within a single circuit program (even within a single file). The circuit language used therein is syntactically identical to Go while also:

Providing facilities for spawning goroutines on remote hardware, and

Treating local and remote goroutines in the same manner, both syntactically and semantically.
As a result, distributed application code becomes orders of magnitude shorter, as compared to using traditional alternatives. For isntance, we have been able to write large real-world cloud applications — e.g. streaming multi-stage MapReduce pipelines — in as many as 200 lines of code from the ground up.
For lifecycle maintenance, the circuit provides a powerful toolkit that can introspect into, control and modify various dynamic aspects of a live circuit application. Robust networking protocols allow for complex runtime maneuvers like, for instance, surgically replacing components of running cloud applications with binaries from different versions of the source tree, without causing service interruption.
The transparent source of the circuit runtime makes it easy to instrument circuit deployments with custom logic that has full visibility of cross-runtime information flow dynamics. Out of the box the circuit comes with a set of tools for debugging and profiling in-production applications with minimal impact on uptime.
Petar Maymounkov developed at Tumblr, Inc. @maymounkov
I obtained my B.A. in Mathematics and Computer Science from Harvard University in 2001, where my adviser was Prof. Michael Mitzenmacher. My interest in Coding Theory and Probabilistic Algorithm Design—which later reached fruition in well-received works on rateless and network coding—was born in Michael's classes. From 2001 to 2003, I attended the Ph.D. program in Computer Science at the Courant Institute of Mathematical Sciences at NYU, which resulted in my M.Sc. in Computer Science. My tastes and interests, in systems and theory, were very influenced by my adviser Prof. David Mazières—now a Stanford University professor—and Prof. Joel Spencer—a true gentleman of Mathematics. With David I completed my currently most impactful work on Kademlia: a distributed algorithm that enables billions of peers to store and discover information collaboratively. In 2012, I obtained my Ph.D. from MIT, Computer Science and Artificial Intelligence Lab, happy to have been advised by Prof. Jonathan Kelner and supported by Prof. Frans Kaashoek. At MIT I concentrated on Spectral Graph Theory and its applications to distributed combinatorial and numerical algorithms. My work with Jon on electric routing is the foundation for a new generation of robust, resilient, secure and anonymous computation systems, built on top of socially-engineered often-faulty communication topologies. Following, during 2012, I worked at Tumblr, Inc. where I developed a new paradigm for efficient development and sustenance of data-processing distributed applications, called the Go Circuit. I am interested in all aspects of distributed applications: the languages and tools we use to build them, the higher-level software metaphores we build to hide away the complex semantics of subsystem failures in distributed systems, design and analysis of numerical algorithms for pivotal problems like solving linear systems and sketching massive structured data, even more general tools for the more challenging peer-to-peer setting. I am an avid Ashtangi and rock climber.
Recorded at Strange Loop conference ( in St. Louis, MO, Oct 2013.