Lisp Hackers: Marc Battyani

Marc Battyani is one of the people who put Lisp at the foundation of their business, and he doesn't seem to regret that decision. And the business itself is exploring very interesting aspects of high-performance and reconfigurable computing. Besides, he has a notable open-source contribution with cl-pdf/cl-typesetting libraries (which even I use, so thanks Marc!). He elaborates on all these and more in much detail in the interview.

Tell us something interesting about yourself.

I think maybe the most unusual things I do is that I work on very different application domains which are even sometimes completely at opposite extremes both in electronics and software. For instance form ultra low power smart sensors based on $1 microprocessors which will run continuously for 5+ years only powered by a small coin battery up to the world's lowest latency supercomputers based on FPGA costing thousands of $ per chip. On the software side I use programming languages ranging from the lowest level languages like assembly or even below with VHDL up to really high level languages like Common Lisp.

I've always enjoyed mixing electronics design and higher level computer science and all that diversity probably gives me a different and original view on say computing and programming in general.

What's your job? Tell us about your company.

I'm the CTO of NovaSparks a startup I founded in 2008 to make ultra-low latency FPGA based supercomputers for the financial markets. BTW These things are really incredibly fast. For instance on 10Gb/s Ethernet market data packets coming from exchanges like the NASDAQ we process the IP/UDP/multicast network stack, extract the messages from the packets, parse/decode/filter/normalize those messages, maintain the indexed order book data structures, aggregate the price levels per stock, generate output messages and finally send them to a server through PCI-express or 10Gb/s Ethernet network stacks. The nice thing is that we do all that fully pipelined at a rate of one message every 12 nanoseconds! To have an idea of how fast it is, in 12 ns the light will only travel 3.6 meters (11.8 ft). Another way to view this performance is that the system can process 83 Millions of financial messages per second without any queuing.

As an aside it is interesting to note that the Domain Specific Language Compilers and various other tools written in Common Lisp have been key enabling factors for the creation of NovaSparks.

I'm also the CEO and CTO of Fractal Concept which is the company where we were developing that technology before starting NovaSparks but as NovaSparks has been using more than 100% of my time for the last years Fractal Concept has been less active with only the development and maintenance of the smart sensors going on.

Do you use Lisp at work? If yes, how you've made it happen? If not, why?

I've always used Common Lisp for most of my work and even when I need to use other programming languages (like VHDL for NovaSpark's FPGAs or javascript for web stuff) I generally use Common Lisp at least for a lot of related tasks like prototyping, designing and testing algorithms, extracting statistics, performing simulations, generating test data, analysing test runs.

The next step is very often to generate some or all the code in other languages by designing various domain specific languages (DSL) which will take care of the tedious aspects of programming in less powerful languages. I really like it when from a few 100s of lines written in a easy to use high level DSL we generate 10000 to 60000+ lines of very low level VHDL code saving months of development.

About that point I think I would even say that I'm mostly doing Language Oriented Programming by trying to abstract the domain specific knowledge into various domain specific languages with very different syntaxes like s-expressions, C like or other languages or even sometimes adding to the mix some GUI or data-based inputs. Then those DSLs would generate most if not all the application code in whatever language is needed be it VHDL, asm, C, javascript or Common Lisp.

What brought you to Lisp? What holds you?

A friend of mine gave me a version of Le_Lisp a version of Lisp used in the 80's by the French universities and engineering schools to teach high level programming concepts. At that time I was mostly programming in Z80 assembly language on a TRS80 and the straight jump from ASM to Lisp was quite a shock and an eye opener.

Since then I've used Common Lisp for countless projects ranging from a few hours of work to multi-years ones and I still find it awesome. Where else can you find a language providing such powerful and multi-paradigms features like CLOS, generic functions, the MOP, macros, closures, s-expressions, lambdas, an interactive REPL (Read Eval Print Loop), native compilation of applications, on the fly native compilation of generated code, the condition system, interactive live programming, real time live debugging of running software and more!

What's the most exciting use of Lisp you had?

I've got so many of them than picking only one would be difficult so here are a few of them:

• softscan: real time driving and data acquisition of automated non-destructive testing installations with real time 3D display (a first in 1995)
• my web application framework: Automatic generation of really complex web applications (the whole stuff from the HTML/javascript front-end to the server back-end). It has been fully Ajax-based with dynamic modification of the displayed pages without reloading the page. This seems obvious nowadays but it was also a first in 2000 and IIRC the term "Ajax" was only invented 5 years later.
• hpcc: The awesome DSL to VHDL compiler. In many aspects VHDL is the complete opposite of Common Lisp. It's a hardware description language used to program FPGA and it deals with the lowest possible programming level with data types like signals, clocks, and bits. Programming at that level is really tedious, time consuming and verbose so it's really a relief to be able to generate tens of thousands of lines of highly optimized VHDL code from just a few hundred lines of some high level Domain Specific Language.
• cl-pdf/cl-typesetting: Being able to generate the first PDF files from scratch in only 107 lines of Common Lisp was an Haha moment.

What you dislike the most about Lisp?

The language itself is somewhat good enough and anyway Common Lisp makes it really easy to change most of itself to add the new and cool stuff or ideas of the day.

In fact what I dislike about Lisp is outside the language and more related to the (mis)perception that people have about it. Having almost everytime to justify its use and sometimes even to fight to be able to use it is somewhat annoying and tiresome.

Of course, Lisp is not for everybody but this is not a reason to have nobody using it. In fact I view Lisp as some kind of amplifier which will give awesome things when used by brilliant developers but will end up giving an incredible mess when used by people without any clues about what they are doing. That's one aspect that makes Lisp very different from other languages which are especially designed to try to normalize and average what people can do with them.

Describe your workflow, give some productivity tips to fellow programmers.

I obviously use emacs and all the nice stuff that work with it like slime, org-mode, magit and lots of other packages. I switched from subversion (svn) to git for all my projects and I now use git mostly form emacs with magit.

In general I have several instances of Lispworks running as standalone apps on my laptop but connected to Slime through M-x slime-connect rather than being started by slime. I do a lot of exploratory programming and interactively play with the code while I write it and if I have to optimize some code I very often use #'disassemble.

About my Lisp coding style I really do like generic functions, CLOS and the MOP. As mentioned earlier I try to generate as much of the code as possible by using macros in simple cases and full blown DSLs in more complex ones. Sometimes I do not dislike making some premature optimizations when I know that speed will be important for some project. BTW when speed matters I like to generate and compile code on the fly when it's useful (and possible) and try to generate really optimized code. It's always cool when you have Common Lisp applications running much faster than C++ ones.

I also refactor a lot. Generally when I start on some new stuff I try to make a first version that works well enough but then once it's done or generally after some time I have some cool ideas to make something much better. At that point I'm really happy that Common Lisp makes refactoring very easy thanks to optional and key args, macros and generic functions with multiple dispatch because all that enables me to make even very deep modifications very easily. I really find those Common Lisp features make software very resilient to code modifications.

What are the advantages of using Lisp in the HPC field? What are the drawbacks? Are you happy with your technology choice?

The HPC field is huge and I can only talk about the small corner of it I know which is the ultra-low latency processing of vast amounts of data we do in NovaSparks. For that the various DSL compilers generating VHDL code have really been a key enabling factor. Programming FPGAs is notoriously difficult and time consuming and this is the major factor limiting the use of reconfigurable computing in the HPC field. The use of those DSL compilers has made it possible to use those FPGA on a more practical basis for processing financial data.

BTW I'm looking for other HPC domains in which those DSL VHDL compilers could be used so feel free to contact me if you have some ideas.

If you had all the time in the world for a Lisp project, what would it be?

Again that's a difficult choice. Speaking about pure Lisp projects I've been willing for a long time to clean up and modernize my web app framework before releasing it as open-source so maybe that could be a good project to start.

Otherwise as mentioned above, I'm looking for possible applications and opportunities of leveraging the DSL => hardware compilers and the FPGAs around some Big Data processing. I have a few ideas but nothing specific for now.

Anything else I forgot to ask?

A conclusion? (Everybody wants a conclusion.) So here is mine:

Common Lisp is Awesome! It is much easier nowadays to use it thanks to all the projects and libraries that Quicklisp makes available. If you do not know Common Lisp then learn it and this will make you a better programmer anyway even if you do not use it directly after.

submit