Lisp Hackers: François-René (Faré) Rideau

François-René Rideau works at ITA Software, one of the largest employers of lispers, which was acquired by Google a year ago. While at ITA he stepped up to support and improve ASDF, the system definition facility, that is at the core of Lisp package distribution. He's also the co-author of the recently published Google Common Lisp Style Guide, which as well originated at ITA.


He's also an active writer: both of code and prose. He's thoughts and articles can be found on twitter, Facebook, Google+, Livejournal, and his site.

Tell us something interesting about yourself.

I like introducing myself as a cybernetician: someone interested in the dynamic structure of human activities in general.

Programming languages and their semantics, operating systems and reflection, persistence of data and evolution of code, the relation between how programmers are organized and what code they produce — these are my topics of immediate professional interest. For what that means, see for instance my slides (improved) from ILC'09: "Better Stories, Better Languages" or my essay "From Creationism to Evolutionism in Computer Programming".

However I'm also interested in cybernetics as applies to Civilization in general, past, present and future. See for instance my essay "Identity, Immunity, Law and Aggression on the Rapacious Hardscrapple Frontier" or my writings about Individual Liberty and the basic principles of Economics

Last but not least, I was recently married to my love Rebecca Kellogg, with whom I have since had a daughter Guinevere Lý "Véra" Kellogg Rideau (born last May). This gives me less free time, yet somehow made me more productive.

What's your job? Tell us about your company.

For the last 7 years or so, I have been working at ITA Software, now part of Google Travel. I have been working on two servers written in Lisp, at first briefly on QPX the low (air)fare search engine behind Orbitz and Google Flights then mostly on QRes, a reservation system now launched with Cape Air. These projects nowadays each count about half a million lines of Common Lisp code (though written in very different styles), and each keep growing with tens of active developers.

I suspect that my login "fare" (at itasoftware) was a pun that played in favor of recruiting me at ITA; however, it wasn't available after the Google acquisition, so now I'm "tunes" (at google), to remind myself of my TUNES project.

At ITA, I have been working mostly on infrastructure:

• how to use better compilers (moving from CMUCL to SBCL, CCL),
• how to build, run and test our software,
• how to maintain the free software libraries we use and sometimes write,
• how to connect QRes to QPX and follow the evolution of its service,
• how to persist objects to a robust database,
• how to migrate data from legacy systems,
• how to upgrade our software while it's running, etc.

And debugging all of the above and more, touching many parts of the application itself along the way.

I think of my job at ITA so far as that of a plumber: On good days, I design better piping systems. On bad days, I don gloves and put my hands down the pipes to scrub.

Since you're mentioning me as working at ITA and on ASDF, I suppose it is appropriate for me to tell that story in full.

In building our code at ITA, we had grown weary of ASDF as we had accumulated plenty of overrides and workarounds to its unsatisfactory behavior. Don't get me wrong: ASDF was a massive improvement over what existed before (i.e. mk-defsystem), making it possible to build and share Common Lisp software without massive headaches in configuring each and every library. We have to be grateful to Dan Barlow indeed for creating ASDF. But the Common Lisp ecosystem was dysfunctional in a way that prevented much needed further improvements to ASDF. And so I started working on a replacement, XCVB.

Now, at some point in late 2009, I wrote a rant explaining why ASDF could not be saved: "Software Irresponsibility". The point was that even though newer versions of ASDF were written that slowly addressed some issues, every implementation stuck to its own version with its own compatibility fixes; no vendor was interested in upgrading until their users would demand upgrades, and users wouldn't rely on new features and bug fixes until all vendors upgraded, instead caring a lot about bug-compatibility, in a vicious circle of what I call "Software Irresponsibility", with no one in charge, consensus required for any change, no possible way to reach consensus, and everyone discouraged.

However, I found a small flaw in my condemnation of ASDF as unsalvageable: if, which was not the case then, it were possible to upgrade ASDF from whichever version a vendor had installed to whichever newer version you cared for, then ASDF could be saved. Users would be able to rely on new features and bug fixes even when vendors didn't upgrade, and vendors would have an incentive to upgrade, not to stay behind, even if their users didn't directly demand it. The incentive structure would be reversed. Shortly after I wrote this rant, the current ASDF maintainer stepped down. After what I wrote, I felt like the honest thing to do was to step forward. Thus, I started making ASDF self-upgradable, then massively improved it, notably making it more robust, portable, and easy to configure — yet fully backwards compatible. I published it as ASDF 2 in 2010, with the help of many hackers, most notably Robert Goldman, and it has quickly been adopted by all active Common Lisp vendors.

You can read about ASDF and ASDF 2 in the article I wrote with Robert Goldman for ILC 2010: "Evolving ASDF: More Cooperation, Less Coordination". I'm also preparing a talk at ILC 2012 where I'll discuss recent enhancements. I have to admit I didn't actually understand the fine design of ASDF until I had to explain it in that paper, thanks to the systematic prodding of Robert Goldman. Clearly explaining what you're doing is something I heartily recommend to anyone who's writing software, possibly as a required step before you declare your software complete; it really forces you to get the concepts straight, the API clean, and the tests passing. That also did it for me with my more recent lisp-interface-library, on which I'm presenting a paper at ILC 2012: "LIL: CLOS reaches higher-order, sheds identity, and has a transformative experience".

One double downside of ASDF 2 is that it both took a lot of resources I didn't put in XCVB, and made for a much better system for XCVB to try to disrupt. It isn't as easy anymore to be ten times better than ASDF. I still hope to complete XCVB some day and make it good enough to fully replace ASDF on all Common Lisp platforms; but the goal has been pushed back significantly.

Now one important point that I want to explicitly stress is that the problem with ASDF was not a strictly technical issue (though there were many technical issues to fix), nor was it strictly a social issue; it was an issue at the interface between the social and the technical spheres, one of how our infrastructures and our incentives shape each other, and what kind of change can bring improvement. That's the kind of issues that interest me. That's why I call myself a cybernetician.

Do you use Lisp at work? If yes, how have you made it happen? If not, why?

I've made it happen by selection. I applied at ITA Software precisely because I knew (thanks to the Carl de Marcken article published by Paul Graham), that the company was using Lisp to create real-world software. And that's what I wanted to do: create real-world software with a language I could use without wanting to kill myself every night because it is turning me into a pattern-expanding machine rather than a human involved in thinking and using macros as appropriate.

"I object to doing things that computers can do." — Olin Shivers

Yet, in my tasks as a plumber, I have still spent way too much time writing shell scripts or Makefiles; though these languages possess some reflection including eval, their glaring misdesign only lets you go so far and scale so much until programs become totally unmanageable. That's what pushed me over the years to develop various bits of infrastructure to do as much of these things as possible in Lisp instead: cl-launch, command-line-arguments, philip-jose, xcvb, asdf, inferior-shell.

Interestingly, the first and the last, cl-launch and inferior-shell, are kind of dual: cl-launch abstracts over the many Lisp and shell implementations so you can invoke Lisp code from the Unix shell; it is a polyglot lisp and shell program that can manipulate itself and combine parts of itself with user-specified Lisp code to produce an executable shell script or a dumped binary image; I sometimes think of it as an exercise in "useful quining". inferior-shell abstracts over the many Lisp and shell implementations so you can invoke Unix shell utilities from any Lisp implementation, remotely if needs be (through ssh), and with much nicer string interpolation than any shell can ever provide; it is a classic Lisp library notably available through Quicklisp. With the two of them, I have enough Unix integration that I don't need to write shell scripts anymore. Instead, I interactively develop Lisp code at the SLIME REPL, and have a shell-runnable program in the end. That tremendously improved my quality of life in many situations involving system administration and server maintenance.

What brought you to Lisp? What holds you?

My first introduction to Lisp was in high-school, in using the HP RPL on my trusty old HP 28C (eventually upgraded to a HP28S, with 32KB of free RAM instead of 4KB!). When I became student at Ecole Normale Supérieure, I was taught Caml-light by xleroy himself, I learned to use Emacs, and I met Juliusz Chroboczek who introduced me to Scheme and Common Lisp, continuations and SICP. Finally, during my vain efforts to gather a team to develop an operating system based on a higher-level language as part of the TUNES project, I have been introduced to Lisp machines and plenty of other interesting concepts.

I use Lisp because I couldn't bear to program without higher-order functions, syntactic abstraction and runtime reflection. Of all Lisp dialects, I use Common Lisp mainly because that's what we use at work; of course a large reason why we use it at work is because it's a good language for practical work. However, frankly, If I were to leave ITA (by Google), I'd probably stop using Common Lisp and instead use Racket or Maru, or maybe Factor or Slate, and try to bootstrap something to my taste from there.

What's the most exciting use of Lisp you had?

I remember being quite exhilarated when I first ran the philip-jose farmer: it was a server quickly thrown together by building green-threads on top of arnesi's (delimited) continuation library for CL. With it, I could farm out computations over a hundred servers, bringing our data migration process from "way too slow" (weeks) to "within spec" (a few hours). It's impressive how much you can do in Lisp and with how little code!

While I released the code in philip-jose, it was never well-documented or made user-friendly, and I suspect no one ever used it for real. This unhappily includes ITA, for my code never made it to production: I was moved to another team, our customer went bankrupt, and the new team used simpler tools in the end, as our actual launch customer was 1/50 the size of our first prospect.

What you most dislike about Lisp?

For Lisp in general, I would say the lack of good ways to express restrictions on code and data. Racket has been doing great work with Typed Racket and Contracts; but I'm still hoping for some dialect with good resource management based on Linear Logic, and some user-extensible mechanism to define types and take advantage of them.

For Common Lisp in particular, though I do miss delimited continuations, I would say that its main issue is its lack of modularity. The package system is at the same time low-level and inexpressive; its defsystem facilities are also lacking, ASDF 2 notwithstanding; up until the recent success of Zach Beane's Quicklisp, there wasn't a good story to find and distribute software, and even now it's still behind what other languages have. This is part of a vicious circle where the language attracts and keeps a community of developers who live happily in a context where sharing and reusing code is relatively expensive (compared to other languages). But things are getting better, and I have to congratulate Zach Beane once again for Quicklisp. I believe I'm doing my small part.

Among software projects you've participated in what's your favorite?

I unhappily do not have a great history of success in software projects that I have actively participated in.

However, I have been impressed by many vastly successful projects in which I had but a modest participation. In the Linux kernel, the Caml community, the Racket community, (QPX and QRes at work might also qualify but only to lesser degrees), there were bright people unified by a common language, by which I mean not merely the underlying computer programming language, but a vision of things to come and a common approach to concepts: not just architecture but architectonics. Another important point in these successful projects was Software Responsibility (as contrasted to the previously discussed Software Irresponsibility): there is always someone in charge of accepting or rejecting patches to any part of the system. Patches don't linger forever unapplied yet unrejected, so the software goes forward and the rewarded contributors come back with more and/or better patches. Finally, tests. Lots of them. Automatically run. All the time. Proofs can do, too, though they are usually more expensive (now if you are going to do testing at the impressive scale of sqlite, maybe you should do proofs instead (see CPDT). I discovered, the hard way, that tests (or proofs) are the essential complement to programs, without which your programs WILL break as you modify them.

If you had all the time in the world for a Lisp project, what would it be?

I would resurrect TUNES based on a Linear Lisp, itself bootstrapped from Racket and/or Maru.

Describe your workflow, give some productivity tips to fellow programmers.

First, think hard and build an abstract model of what you're doing. Guided by this understanding of where you're going, code bottom up, write tests as you do, and run them interactively at the SLIME REPL; make sure what you write is working and passing all tests at all times. Update your abstract model as it gets pummeled into shape by experience. Once you've got the code manually written once or twice and detect a pattern, refactor it using macros to automate away the drudge so the third time is a piece of cake. Don't try to write the macro until you've written the code manually and fully debugged it. Never bother with low-level optimization until the very end; but bother about high-level optimization early enough, by making sure you choose proper data structures.

Unhappily, I have to admit I am a serial under-achiever. I enjoy thinking about the big picture, and I like to believe I often see it better and further than most people; but I have the greatest trouble staying on track to bring about solutions: I have so many projects, and only one life to maybe complete a few of them! The only way I can actually get a few things done, is to decompose solutions into small enough steps such that I can keep focused on the next one and get it done before the focus goes away.

A year ago Google bought ITA, which was, probably, the largest Lisp company recently. What were the biggest upsides and drawbacks of using Lisp on the scale of ITA? Does Lisp have a future inside Google?

On the upside, we certainly have been able to write quite advanced software that we might not have otherwise managed. A million lines of Lisp code, including its fair share of macros and DSLs, would be so many more million lines of code without the syntactic abstraction made possible by Lisp. However hard and expensive it was with Lisp, I can only imagine how many times worse it would have been with anything else.

At the top of the tech bubble in 2008, we had over fifty Lisp programmers working just on QRes, gathered at an exponential rate over 3 years. That's a lot. We didn't yet have good common standards (Jeremy Brown started one, later edited by Dan Weinreb; I recently took it over, expanded it, merged it into the existing beginning of a Google Common Lisp Style Guide and published it), and it was sometimes hard to follow what another hacker wrote, particularly if the author was a recently hired three-comma programmer. But with or without standards, our real, major, problem was with lack of appropriate management.

We were too many hackers to run without management, and none of our main programmers were interested in becoming managers; instead managers were parachuted from above, and some of them were pretty bad: the worst amongst them immediately behaved like empire-building bullies. These bad managers were trying to control us with impossibly short, arbitrary deadlines; not only did it cause overall bad quality code and morale burnout, the renewing of such deadlines quarter after quarter was an impediment to any long-term architectural consideration for years. What is even worse, the organization as setup had a lot of inherent bad incentives and created a lot of conflicts, so that even passable managers would create damage, and otherwise good engineers were pitted against each other one two sides of absurd interfaces, each team developing a lot of scar tissue around these interfaces to isolate itself from the other teams. Finally, I could witness how disruptive a single bad apple can be when empowered by bad management rather than promptly fired.

I have had a lot of losing fights with QRes management at a time when, hanging on a H1B visa, I was too much of a coward to quit. Eventually, the bad people left, one by one, leaving behind a dysfunctional organization; and great as the people that manned it may have been, none was able or willing to fix the organization. Then finally, Google acquired us. There's a reason why, of two companies founded at about the same time, one buys the other and not the other way around: one grew faster because it got some essential things right that the other didn't. Google, imperfect as it necessarily is, gets those essential things right. It cares about the long term. It builds things to scale. It has a sensible organization. It has a bottom up culture. So far, things have only improved within QRes. Also, launching was also good in many ways. It makes us and keeps us real.

Lisp can be something of a magic tool to solve the hardest technical issues; unhappily it doesn't even start to address the social issues. We wasted a whole lot of talent due to these social issues, and I believe that in an indirect way, this is related to the lack of modularity in Common Lisp, as it fostered a culture of loners unprepared to take on these social issues.

So I'm not telling you this story just to vent my past frustration. There too I have a cybernetic message to pass on: incentives matter, and technical infrastructure as well as social institutions shape those incentives and are shaped by them.

As for the future of Lisp at Google, that million line of Common Lisp code ain't gonna rewrite itself into C++, Java, Python, Go, or even DART. I don't think the obvious suggestions that we should rewrite it were ever taken seriously. It probably wouldn't help with turnover either. But maybe, if it keeps growing large enough, that pile of code will eventually achieve sentience and rewrite itself indeed. Either that, or it will commit suicide upon realizing the horror.

Anything else I forgot to ask?

Ponies.

submit