Lisp Hackers: John Fremlin

John Fremlin has created a couple of very performant Common Lisp programs beating on some microbenchmarks the fastest similar software written in any other language, including C: the teepeedee2 dynamic webserver, that managed to break the c10k record on a single core machine, and cl-irregex regex library. Working at MSI in Japan he also had written an object persistence DB for CL manardb. Besides, he writes interesting blogs on topics of software optimization, programming languages and technology in general.

Tell us something interesting about yourself.

I've been to more than eighty countries; I want to go everywhere!

What's your job? Tell us about your company.

I work at Facebook on the growth team, on data-driven improvements to the sign-up flows.

Do you use Lisp at work? If yes, how you've made it happen? If not, why?

I used to at msi.co.jp. It is a Japanese consultancy based in Tokyo called originally Mathematical Systems Institute. Mr Kuroda leads the Lisp group there and I think it has hovered around five or six people over many years. He's done a great many very interesting projects for a range of companies over the years: for example a crash-test data inspection tool for a big Japanese car company, text mining, graph visualisation and so on. I worked primarily on building up a visualisation and mapping of a very large set of routers for the world's biggest telecoms, which led to the creation of manardb.

I really enjoyed working for Mr Kuroda and I'm sorry I had to leave for personal reasons. There were always many very fascinating problems around — with great people to discuss them and find solutions. It was a very stimulating workplace!
At Facebook, I use PHP, Python, C++, Java and miscellaneous things. I think we would all be better off if we hadn't balkanised the different systems that we program for — and Lisp is one of the few programming languages with the flexibility to serve in all these roles.

What brought you to Lisp? What holds you?

My initial programming was following Michael Abrash's graphics books and building on his ideas, by doing things like runtime native code generation for drawing dynamically generated bitmaps efficiently. This is not so interesting for modern processors as they have good branch prediction but the idea of code generation stuck with me and Lisp is one of the few programing languages that makes this easy and efficient.

I appreciate the intellectual coherence of Lisp, and its sensible approach to numeric computations. In terms of using it today, I feel that Common Lisp has an advantage over many other programming languages in that it has multiple mature independent implementations. Running on multiple compilers tends to greatly increase the quality of a program in my opinion, as the code is exposed to different static analyses.

What's the most exciting use of Lisp you had?

I helped someone use Lisp for an automated trading project.

What you dislike the most about Lisp?

In trying to make efficient code one ends up fighting against the compiler and the runtime system and most of the time is spent in coming up with clever ways to circumvent and outwit both. This is not a good use of resources, and means that it usually makes more sense to start with C++.

Tell us about your approach(es) to optimizing Common Lisp code (and maybe code optimization in general)?

The most important thing is to try to hold in your head an understanding of where the program is going to spend time. Profilers can be misleading and inaccurate, and it is sometimes difficult to get representative workloads to profile. I think their main utility is in confirming that there is no sloppy mistake (in Lisp, typically, consing accidentally) that prevents you from achieving the natural performance of your approach.

Complexity analysis in terms of computation, network usage, disk accesses and memory accesses is a first step as obviously if you can improve the asymptotic usage of a bottlenecked resource, you will very likely do much better than trying to tweak some little detail. The second step is to try to characterize interactions with caches and, in Lisp, garbage collection, which is pretty tricky.

Among the software projects you've participated in what's your favorite?

I think the one I enjoyed most was an embedded H.264 decoder in 2005. This was for the VideoCore, a really wonderful CPU architecture that could deal with parallelizable problems incredibly efficiently if programmed correctly. It would have been awesome to use Lisp for it!

If you had all the time in the world for a Lisp project, what would it be?

I wish there were Lisp bridges to other runtime systems (Java, Android, Objective C, Perl, Python, C++, R, etc.) so that the libraries and tools for each could be leveraged efficiently in Lisp and vice versa. That would mean being able to call Java code and handle Java objects in Lisp, for example -- perhaps initially by spinning up a Java implementation in a separate process running a CL-SWANK style interface.

I really don't think this would be that difficult and it would make a huge difference to the convenience of building programs in Common Lisp!

Describe your workflow, give some productivity tips to fellow programmers.

I use emacs and I have a bunch of elisp code that I keep meaning to publish!

submit