Lisp Hackers: Marijn Haverbeke

The next person in our series is Marijn Haverbeke, who's not only the author of several pretty useful Common Lisp libraries (some of which he touches in the interview), but also a succesful JavaScript hacker, winning JS1K contest, and writing a profound book about the language — "Eloquent JavaScript". Besides, he hacks on Mozilla Rust and his own language Hob, not to mention co-authoring a JavaScript-to-Common-Lisp transpiler, writing some games in Scheme, and experimenting with Haskell. In the interview he shares his perspective, based in knowledge and experience with so many different languages.

Tell us something interesting about yourself.

I once thought I wanted to be a Sociologist. Before that, I also once thought that I wanted to become a treasure hunter. And a dinosaur researcher. I ended up a programmer, largely self-taught, and I'm very happy with how things turned out.

What's your job? Tell us about your company.

I'm working free-lance. Currently, I'm doing a lot of work for Mozilla, hacking on their Rust compiler. Rust is a programming language that's supposed to fit the C++ niche, without the insanity of actual C++. If the project works out, and it's going swimmingly so far, Mozilla will try to apply it to writing a next-generation browser, without the constant danger of buffer overrun exploits and threading bugs that comes with C++. True to Mozilla's spirit, all development on Rust is happening in the open, on github.

Another things I'm busy with is my own CodeMirror project, a code editor written in JavaScript. It's not a 'job', really, since it's open-source and no one is paying me to maintain it, but it's grown popular enough to generate a decent amount free-lancing gigs and a small income stream from support contracts.

Do you use Lisp at work? If yes, how you've made it happen? If not, why?

Not currently, beyond that I always have a REPL open in emacs, and switch to it to do quick computations and experiments. I've spent several years doing almost exclusively Lisp (mostly working on AllegroGraph, Franz' RDF database), and it has definitely been somewhat painful to find myself working with non-interactive, macro-less languages again, after being used to the luxuries of Common Lisp and Slime.

What brought you to Lisp? What holds you?

Ten years ago, during my sociology study, I randomly chose a Lisp course in university. I was spending so much time in emacs that I figured I might as well learn more about its scripting language. The course worked through Structure and Interpretation of Computer Programs, using the video lectures from Abelson and Sussman. Having been exposed mostly to Java, C++, and Python before that, the elegance and depth of this material was something of a revelation to me.

My professor for that course, Eduard Hoenkamp, was a real Lisp chauvinist with an MIT background, and properly indoctrinated me with regards to Lisp's glorious superiority. I wrote a text adventure game in Scheme for the end project of the course, my first non-trivial Lisp endeavor. I still have the code. I ended up starting a PhD with this very professor, and though I never finished it, it did provide me with ample opportunity to learn Common Lisp.

What's the most exciting use of Lisp you had?

I think that'd be CL-JavaScript, a JavaScript-to-Common-Lisp transpiler that I wrote together with a friend. It actually handles the whole ECMAScript 3 language and produces programs that run reasonably fast—about on par with the 2009 generation of browser JavaScript engines. This is quite amazing, given that it's less than 4000 lines of code, at least an order of magnitude smaller than those engines. The Lisp compiler handles most of the optimization for us. (I should point out that the browser JS engines have a much higher speed of compilation, which is a necessity for handling web content.)

What you dislike the most about Lisp?

I must say that the lack of static typing really gets in the way on larger projects. Being able to confidently change datatypes and function signatures, knowing that the compiler will point out most inconsistencies that you introduce, is something that I've really come to appreciate after working with Rust and Haskell. Test coverage helps, of course, but I'm not a very diligent test writer, and tend to feel that type signatures are easier to maintain than an exhaustive test suite.

Describe your workflow, give some productivity tips to fellow programmers.

My approach to writing software is embarrassingly unstructured. I'm not sure I can claim to have something that merits the term 'workflow'. One thing that is very important for me, and that, had I figured it out earlier, would probably have saved some past projects from dying, is to never have a system in a half-refactored, non-functional state for longer than an hour or so. A lot of momentum and motivation comes from having a working program in front of me. This sometimes means I have to split up big changes in a number of more indirect, trivial steps, and resist the temptation to start hammering out my big change in one go (which, even when seems straightforward in my head, will always hit some complications during implementation).

Also, really learning git in depth has had a big influence on how I code. Once you really understand that changes (if you bother to commit often and with care) can be reverted and picked at will, and that branches are cheap and easy to create and maintain, you start to see a codebase differently. I often take multiple shots at making a tricky change — if it feels like my first approach is not working out, I simply commit it, back up, and start again, knowing that if the new approach doesn't work out, I can simply flip back to the old one again. Knowing that this is theoretically possible is not enough—you need to have actually done it, often, to start applying it properly.

Among software projects you've participated in what's your favorite?

Postmodern, the PostgreSQL client, was my first non-trivial open-source project, and the first piece of actually useful software that I designed and executed on my own. I was extremely proud when I got it to work the way I wanted (modular, fast), and meticulously wrote out a long set of docs (copying Edi Weitz's style). Seeing people actually use it (and even write good things about it!) really drove home for me the fact that it is worthwhile and rewarding to package up the stuff I write, and share it.

The community involvement in Postmodern, though always low-volume, has also been awesome. Most bug reports came with a patch, or at least an informed view of the problem and useful diagnostic information. This in sharp contrast with the JavaScript community, where of course great contributors also occur, but one has to deal with a lot of noise and lazy bug reporting.

If you had all the time in the world for a Lisp project, what would it be?

I am working on my own programming language, Hob, which tries to combine the virtues of Lisp and ML. This is a huge project, and unfortunately it's moving forward only slowly. If I didn't have financial constraints, I'd spend a year (or two) finishing it. Its 'bootstrap compiler' (the compiler that will compile the first self-hosting compiler) is in the process of being written in Common Lisp. The writing that I currently have online about it is quite old and out of date. I just spent a week rewriting the compiler from scratch, moving the language to a completely regular syntax—think s-expressions, though Hob looks somewhat different. My work on Rust has convinced me that, even with all the good will in the world, you can't bolt macros onto a classical language with a non-regular syntax without making them very painful to write (and use!). They have to be part of the design from the start.

You've been actively using Common Lisp and JavaScript. When you use JavaScript, what do you miss the most from CL? What do you miss from JavaScript, when you're in Lisp land?

The greatest thing about Common Lisp is that the standardization process drew from years and years of experience and organic growth. This makes the language feel singularly solid and practical. Most languages feel immature compared to CL. I suspect that this is not just age, but also the fact that Lisp allows users to extend the language to an unprecedented degree, which allows for much more experimentation, and thus faster growth and invention.

The great thing about JavaScript is that it has so few concepts. It manages to strike a balance between being dumbed-down enough to be easy to grasp, which has allowed it to become as widespread as it currently is, and still being a more-or-less decent lambda language, which makes it scale up to complex programs relatively well. It is on its way to become a lingua franca of scripting, which Common Lisp will never be, because it is much more demanding of its programmers—they need to understand a lot more concepts.

And here is the Hacker News discussion. submit

Lisp Hackers: Christophe Rhodes

Next in our series of interviews with inspiring lispers is a conversation with SBCL's principal maintainer Christophe Rhodes. He was among the first two people to join the project, after it was solely developed for more than a year by William Newman. And he is, probably, the most "stable" contributor, still remaining very active in the community, after more than a decade. What fascinates me personally about him, is the aura of a profoundly creative mind, fiddling with reality in interesting and unexpected ways.

Tell us something interesting about yourself.

I'm a singer, specializing in performance (admittedly somewhat rare at the moment) of unaccompanied music of the European Renaissance. My next concert is Brumel's "Earthquake Mass" with the vocal ensemble De Profundis. I did my doctoral research a few doors down from Stephen Hawking. AndI was a cast member on "Una Stravaganza dei Medici", a reconstruction of the Florentine festivities for the marriage of Ferdinand de Medici and Christine of Lorraine - which means that there's at least some chance I have an Erdős-Bacon number.

What's your job? Tell us about your company.

Oh, well, that's a bit complicated! :-) Most of my time at the moment is spent with Teclo Networks, a Lisp-flavoured startup which is bringing reliability and usability to mobile broadband. I'm on leave of absence to pursue this from Goldsmiths, University of London, where I am a lecturer in Computing, responsible for undergraduate teaching, postgraduate supervision and independent research.

Do you use Lisp at work? If yes, how you've made it happen? If not, why?

I do. I should say that I'm no longer, if I ever was, a programmer by trade; most of my work at Teclo is not writing code, but when I do it is in C, R or Common Lisp. Working in a Lisp-flavoured startup (I'm not the most famous Lisp programmer in the company, and quite some way from being the best...) has made Lisp a natural tool to use when it's appropriate; while it might be a bit of a shame that Lisp is not used exclusively, the balance of idealism and pragmatism is pretty good, and it's fairly satisfying to know that the high-performance network element that we ship has Lisp code controlling its operation.

I wouldn't make an argument that C is a Lisp by any stretch, but I've popped up at a couple of events in the last year to argue that R has some claims to being a Lisp; it has conditions, handlers, restarts, generic functions, macros operating on parse trees... and a SWANK backend.

Another part of what I do is write documents, both for Teclo and for Goldsmiths. I have in the last couple of years fallen in love with org-mode and in particular with "reproducible research", or living documents; the ability to have executable sections of code for generating results, which can then themselves be further analysed by other bits of code — in other languages, if that's appropriate — and the whole exported as a document typeset to what I consider to be high-quality standards. The fact that there is also acceptable-quality web output from the exact same process is gravy.

What brought you to Lisp? What holds you?

Two seminal moments on my road to Lisp: the first was, as an undergraduate, getting a job in the summer of 1997 to work on simulating fluid dynamics in the context of galaxy formation, to assist in trying to understand how the spiral arm structures form. I'd being told "here is K&R, here is USENET, here is XEmacs, here is Tcl; in five weeks let's see what you've come up with" — I distinctly remember reading comp.lang.lisp and a "Lisp is slow" thread, with active participants combating the troll with disassembly.

The second, a few years later, was wandering into the #lisp IRC channel on openprojects, to find an almost-empty channel, about 10-20 participants; one of whom was there twice (dan_b and dan`b), using the channel to paste bits of alpha assembly between his laptop and his desktop, in his efforts to get SBCL on alpha working.

As for what keeps me, well, when what I have to do is solve problems that no language has built-in libraries for (scalable audio similarity detection, say, or musical structure inference) then I want to be working in a language where I can focus on the problem itself, rather than the detail of that language — and for me, Lisp permits that kind of separation of concerns, given its easy support for DSL and embedded language generation, protocols, and the ability to run partial programs.

Among software projects you've participated in what's your favorite?

I enjoyed most working on an editor for lute tablature, I think. Partly because it was a chance to learn about things (parsing, CLIM, user interfaces) that I hadn't really thought about before. Partly because of the feeling of being able to build up incrementally and rapidly from a very simple prototype to deliver something which was actually useful; and partly because it was also interesting to learn about the problem domain. Gsharp (G#), the score editor largely designed by Robert Strandh, is an editor for what you might call "common practice" notation: five lines in a staff; dots and lines to indicate pitch (by height) and duration (by glyph) of notes. In my research group at Goldsmiths, though, we have a lutenist (a player of the lute) who had a research project on cataloguing and transcribing a particular repertoire of 16th-century lute music, which was notated in "tablature": six lines, one per string of the lute; letter or number glyphs on each line to indicate finger position; and glyphs above the staff to indicate duration.

The point about this was that when I first came on the scene the project had a truly awful workflow, because they had a batch renderer from the textual format to encode this music to something visual. This made the process of encoding painful, in particular for the error correction phase: it was fairly easy to find an error in the visual rendering, but to go from there to the text corresponding to it was painful. So I wrote a little CLIM app with an editor buffer and a display; the display was a rendering of the "Tabcode" in the editor buffer, and each glyph in it was associated with a text cursor position, so that you could click on the display and have the editor cursor warp to the Tabcode corresponding to the glyph. This was a productivity win for the project.

What's the most exciting use of Lisp you had?

I don't get easily excited by technology. So for me, what was exciting was seeing a community of users and developers grow around SBCL — in particular, working with extremely focused and motivated individuals on improving the system, subject to all sorts of constraints, not least those imposed by real-world uses (including the QPX poster-child of ITA Software).

SBCL contributors seem to be a very active, closely-knit, and united community. How did that happen? Maybe you could share some insight into the history of the project. Also what were the factors, that determined its success?

I think part of SBCL's success as a community was the finding of like minds — which came about partly because it started off as a project when access to information channels was growing, but also partly because there were some explicit principles laid down: Bill Newman made it very clear that he was aiming for something maintainable, and also taught by example the meaning of maintainability. In the SBCL source distribution, there are over 100 lines of textual detail describing his ideas about what he was aiming for, and although it's nothing like an explicit charter which every SBCL developer (or user!) has to sign up to, it's nevertheless a useful indication of a direction to drive in. Both detailed and motivated enough that even a graduate student with no software development experience could understand that there were tradeoffs involved, and someone had thought hard about which side of the tradeoffs they wanted to be. I'd also point out some of Bill's advodiary entries of the time, including http://www.advogato.org/person/wnewman/diary/6.html and http://www.advogato.org/person/wnewman/diary/9.html, which perhaps illustrate that he was thinking about this kind of software maintenance all the time. ("To foil the maintenance programmer..." :-)

I think this is particularly important when the bulk of the work is done by volunteers; it helps to keep focus (I've written before about how software is only sustainable if it can stay alive in the minds of its developers).

If you had all the time in the world for a Lisp project, what would it be?

There's lots that I'd like to do, both hacking on Lisp itself and things I'd like to explore, for which I'm likely to reach for Lisp first. I'd also like to get more fluent with emacs lisp, enough to be able to quickly write the user interfaces I would like to use to various bits of software that I use every day.

submit

Lisp Hackers: Edi Weitz

The first post of the series got some very good responses, so I'm continuing it with another very obvious candidate — Edi Weitz. His vast contributions to open source Lisp, made during the last decade, collectively known as Ediware, include the indispensable regex library CL-PPCRE, written on a bet in Hamburg café, and a whole stack of web-related libraries with the most widely used Lisp application server Hunchentoot and HTTP client Drakma. Together with Arthur Lemmens he also co-organizes European Common Lisp Meeting. And that's, surely, not all...

Tell us something interesting about yourself.

Well, I'll leave it to someone else to tell you what's interesting about me. I'll rather tell you what I find interesting in addition to Common Lisp: I collect photo books and I'm doing a bit of photography myself. I like to listen to the music of Frank Zappa and to Jazz. I read a lot. I'm interested in mathematics, especially in set theory.

What's your job? Tell us about your company.

I'm a professor for mathematics and computer science at the University of Applied Sciences in Hamburg. I started this job in September 2011.

Before that, I was a freelance hacker for about 13 years.

Do you use Lisp at work? If yes, how you've made it happen? If not, why?

In my new job, I've been using CL in my math lectures a couple of times and will continue to do so.

In my old job, I've been using CL exclusively for the last six or seven years. As I was working freelance, this was kind of easy — I either had projects where the customer didn't care about the programming language that I used as long as I got the job done, or I was hired specifically for my CL skills.

What brought you to Lisp? What holds you?

I came to Lisp via Emacs Lisp in 1999 or so. What got me hooked was the wonderful book "Writing GNU Emacs Extensions" by Bob Glickstein. It opened my mind for the beauty of the Lisp language family — something I had missed the first time I had encountered Lisp (in university, a few years earlier). The two CL books by Paul Graham and Norvig's PAIP then paved the way for Common Lisp.

What holds me is that I haven't found a better programming language so far — and I don't expect to find one very soon.

What's the most exciting use of Lisp you had?

I don't know if "exciting" is the right word, but it makes me happy that so many people use "The Regex Coach" and like it. I stopped keeping track, but there must have been at least half a million downloads since 2003.

I'm also kind of proud that some of my open source libraries are used by various commercial and research projects around the world.

But probably the most awe-inspiring encounters I had with Lisp were the few occasions when I played around with Genera or watched someone else using it. I think this OS really was a work of art.

What you dislike the most about Lisp?

There's nothing I really dislike about Common Lisp. There are a few warts here and there, but so far I've found nothing that was serious enough to prevent me from being productive.

Among software projects you've participated in what's your favorite?

Working on the Windows port of Piano — an extremely impressive application which has been around for almost 20 years and has been used by almost every aircraft manufacturer in the world. Dimitri Simos, Piano's main author, has been the most enjoyable client I've worked with so far.

Describe your workflow, give some productivity tips to fellow programmers.

I usually just start up the LispWorks IDE and hack away. The best productivity tip I can give is to stick with one implementation and IDE and to invest a lot of time to really learn how to use it — including all the implementation-specific goodies like debuggers, inspectors, steppers, browsers, and so on.

Ediware became hugely popular (by Lisp standards), and with this popularity came a lot of work and responsibility. You seem to have mostly handed over supporting it to Hans Hübner. What's up next for you in the land of programming and Lisp in particular?

I'm planning to give a lecture about the use of AI techniques in games in the next year and I might use some Lisp there. I might also — as a sideline — resume my CL consulting work sooner or later. I don't expect to publish new open source code in the near future, though.

If you had all the time in the world for a Lisp project, what would it be?

When I was still working as a hacker, I always dreamt of finding someone to pay me for working on an open-source CLOS object store — written in pure Common Lisp, OS-independent, portable, not relying on third-party software, fast, reliable, thread-safe, well-documented, etc.

submit

Lisp Hackers: Zach Beane

I'm starting a series of interviews with lispers, who inspire me. Lisp community includes a lot of amazing programmers, who's thoughts, I believe, deserve more prominence and recognition, than they usually enjoy. Besides, unlike many other languages, lisp users span several generations of programmers: there are a lot of people, who'd made great contributions, but have since moved on. I hope to have some of them in this series as well.

And the candidate for the first interview was obvious to me. Zach Beane is the uniting link of the whole community. The creator of Quicklisp and a number of useful open-source libraries, like Vecto and ZS3, he is also always active on #lisp irc channel answering questions, as well as keeping a blog, where he shares interesting Lisp news, and supporting Planet Lisp blog aggregator. And that's not all...

Tell us something interesting about yourself.

I live in Maine and enjoy hacking Common Lisp projects for fun and profit.

What's your job? Tell us about your company.

I work at a small telephone and Internet company.

Do you use Lisp at work? If yes, how you've made it happen? If not, why?

I currently use Lisp to analyze some large data files we get from vendors and partners and produce reports. I've recently used Lisp to produce KML files for Google earth visualization of some internal data. I use stuff like cl-ppcre, drakma, postmodern, cl-mssql, cxml, and more to gather data from various systems when preparing reports. The growing library ecosystem has made it really easy to get stuff done.

Lisp is accepted because I can produce useful stuff pretty quickly with it. It's a small company with a small team so there isn't a bureaucracy to fight to use something non-standard. Does the tool work? Is the result useful? Then there's no problem.

In some cases, where it isn't a good fit for the final product, I use Lisp to prototype ideas before writing a final thing in some other language, e.g. C or Perl. But I even use CL to generate data and code in other languages, so it's still in the mix, still a part of my workflow.

What brought you to Lisp? What holds you?

Paul Graham's Beating the Averages describes a really exciting way to use an uncommon tool to great advantage. That got me interested in how I could do the same thing. I started off with some stuff in Scheme because I thought CL was old and crufty and gross, but when I started using SBCL I found it was a great, practical tool, and I never stopped using CL after that.

I continue to use CL because it has a great mix of features (object system, first-class functions, fast compiled code, multiple implementations, a fantastic standard, etc, etc) and it works well in my favorite working environment, Emacs on Linux. I feel like I know it well and I can see quickly how to use it to explore and solve new problems.

What's the most exciting use of Lisp you had?

Making graphics with Lisp for wigflip.com. It's always fun to have an idea about some kind of visual toy and then tinker with it until it's right.

It's also fun to do something fast. A few years ago I got a project to fill out some PDF forms automatically. The project started with a few days scheduled to research third-party solutions, but in those few days I had a working prototype that used CL-PDF and CLSQL.

What you dislike the most about Lisp?

There are a lot of negative perceptions about Common Lisp that are reinforced by current and former Common Lisp users. I can accept that CL is not for everyone, but some of the criticism is just years (sometimes decades) of moaning and nitpicking about decisions made in the distant past that are not really up for review right now. I wish the people who are vocally, chronically dissatisfied with CL would go off and do their own thing and stop bothering people that are happy with Common Lisp.

There are some remarkable trolls that like to pick on CL, but they're not usually taken seriously by anyone, but "insiders" who complain about CL are perceived to be giving some sort of genuine insight. It's very annoying.

Among software projects you've participated in what's your favorite?

Quicklisp has been very rewarding. There's a lot of positive feedback from people who feel it really helped them use CL more easily. More generally, the whole CL ecosystem has been a lot of fun. I enjoy trying new libraries, sending feedback and bug reports, helping people get started, and all that stuff. Common Lisp has a lot of smart, helpful, friendly people who share a lot of knowledge, and I feel lucky to get to learn from them and to try to share what I know, too.

Describe your workflow, give some productivity tips to fellow programmers.

For typical projects I feel like I'm building more tools into my REPL workbench. I don't write scripts that I call from the command line, I write functions that I call from the REPL, and use the slime and Emacs environment to create and interact with data, whether it's data from a file, from a computation, from the web, a database, etc.

I find it really helps to have small, focused functions that are easy to follow and that do one thing and do it well. The fine granularity of testing, tracing, intermediate values, etc. can help wire together a very useful set of behaviors from a small core of simple functions.

Some productivity ideas...

Knock the rough edges off your working environment. Write code to automate stuff. Make it easy to repetitively do complicated but boring stuff. For example, I used to be afraid of the hassle of making new releases of my projects, but recently wrote a CL program that does everything for me, from PGP-signing the tarballs to uploading them along with the documentation to my website. Now I don't care if I make ten project releases in a day, it's just a few function calls.

Customize your environment to make it comfortable. Make it easy to look up info in the hyperspec or in other documentation sources. Make it easy to create new projects. I use quickproject a lot for that, but I also have some Emacs templates that put some boilerplate into my files automatically. Make a program do your work for you.

How did quicklisp change your life? What are current plans for its development?

I can get up and running in a new environment very quickly now. Before Quicklisp, I could usually build up a set of libraries that were comfortable and useful, but it could be a hassle to move them from computer to computer, or to make sure they were up-to-date. Now I just use Quicklisp and don't worry about it.

As Quicklisp maintainer, it's really helped me see where some people are coming from when they want to try Common Lisp. It's a totally different mindset than what I'm used to. I think there's room for some documentation or tutorial on the system-oriented, REPL-workbench style of Common Lisp development that I like to use.

I want to write more documentation for Quicklisp, particularly how users can publish their own software for it or customize it via its internal protocols.

I also want to gather and share documentation for all Quicklisp-loadable projects, so there can be a single place to learn what a given project does, and get access to all its documentation.

I'd also like to make it easy for people to share feedback about their experiences with a project, so you could decide if it's likely to fit your needs. Something like the feedback you see for products on Amazon, but for Lisp libraries.

Anything else?

Common Lisp is a great system that rewards in-depth study.

P.S. Xach gave a talk about Quicklisp at the last ECLM, and I think his motto for Lisp development, he presented there, is really worth sharing and following :)

PPS. And here is the Hacker News discussion. submit

Реальная норма часов работы программиста в день

В 2009-10 годах у меня был некоторый опыт удаленной работы программистом на основе почасовой оплаты. С тех пор я периодически задумывался над тем, сколько же реальных часов работы над кодом в день должен в среднем выдавать программист. На самом деле, этот вопрос, а точнее его последствия, актуален для любого формата работы: как офисной, так и удаленной, так и всего, что между. В "офисном" варианте оплачивается фиксированное время работы и задача в оптимизации реального времени в рамках этого интервала (гибкий график, уменьшение количества встреч и т.п.) Проблемой фриланс-варианта является то, что большинство заказчиков не понимают, что они должны платить не только за время реальной работы над кодом, но и за сопутствующие активности, типа ковыряния в носу, и/или не знают, какое должно быть соотношение этих активностей к собственно программированию.

И вот, на днях на хакерньюз я наткнулся на отличную статью профессиональной писательницы о повышении эффективности ее труда в 5 раз (с 2 до 10 тысяч слов в день). Фактически, ей удалось это сделать за счет выработки метода систематического вхождения в поток. А как все мы творческие работники знаем, поток — это залог эффективной работы*. Но, что в этой статье для меня было действительно важным — это то, что человек, добившийся такой эффективности, констатировал факт: собственно непрерывной работы у нее в среднем 4 часа в день — в самых удачных случаях до 7 — плюс 2 часа с утра, которые расходуются на подготовительную работу. А если вырабатывать больше, то эффективность начинает падать. Не говоря о том, что можно перегореть.

Это наблюдение совпадает и с моим опытом: у меня такое же количество (4-6) часов эффективного писания кода в день. В этом нет ничего удивительного, поскольку труд программиста ничем не выделяется из ряда других творческих профессий: писателя, дизайнера или же композитора. Что же тогда делать с 8-ми часовым рабочим днем? На самом деле, ничего плохого в том, что только 50-75% рабочего времени тратится собственно на работу нет: остальное время тоже не пропадает зря, посколько тратится на комуникацию (без которой любая компания и проект обречены), а также какие-то нерабочие активности, которые формируют ту самую мифическую "корпоративную культуру". И даже если этих активностей нет (фриланс) — это не значит, что: а). программист может это время потратить на работу над кодом б). программисту они не нужны (нужны, возможно, только в другой форме: социализация нужна всем).

Т.е. выходит 2 числа, характеризующие нормального программиста: максимум 6 часов (а в среднем: 4 часа) программирования в день и коэфициент 1.5 соотношения часов программирования к часам работы. И эти числа нужно принять как исходные данные, на основании которых менеджеры могут стоить какие-то предположения, оценки и методики.

Кроме того, такой взгляд разрешает для меня диллему эстимейтов: должен и может ли, вообще, их делать программист?

Функция time профилировщиков программ выдает обычно 2 числа: total time и real/cpu time. Первое значение — это фактическое время, которое прошло от старта программы до завершения. На него могут влиять такие факторы, как кеширование, ожидание ввода-вывода и т.п. А сpu time — это время, которое программа действительно выполнялась на процессоре. По-хорошему, именно это, второе время, может научиться оценивать программист: т.е. время, которое ему понадобиться для написания и отладки кода, когда он будет находится в "разогретом" режиме работы. А вот общее время — которое, на самом деле, интересует бизнес — может научиться оценивать только менеджер, беря во внимание оценки программистов (и их историческое качество), а также 100500 других факторов, которые могут повлиять на режим его работы: кеширование, своевременное заполнение пайплайнов, переключене контекста и т.д.

* Те, кто не знает, может очень быстро узнать из уст человека, который сформулировал эту концепцию (или из его классической книги Flow):

Clojure & Complexity

I gave a rather messy lightning talk at the recent ECLM on this topic (see below). I think, the messiness can be attributed mostly to my undesire to criticize anything, built with good intentions, including Clojure. Yet in software development there's clearly a need for thorough evaluation of different approaches, languages and technologies, because, we must admit, lots and lots of decisions on such things as architecture or language/platform choice are made on purely subjective and even emotional bases (see "hype"). So below is a more detailed account of the (excessive) complexities, I've encountered working with Clojure in a real-world environment: a half-year project to develop a part of a rather big Java-based server-side system. Also I should note, that I was following Clojure almost since its initial public release, have participated in the early flames on c.l.l. and even edited a Clojure introduction article in the Russian functional programming journal fprog.ru. But this was only the first chance to make a reality check...

But, first of all, I'd like to refer you to the talk of Rich Hickey at the Strange Loop conference "Simple made Easy", the principles of which really resonate with me. Yet it's so often the case, that it's hard to follow your abstract principles, when you're faced with reality (also guilty of that). And another point is that it's not really beneficial to push the principles to the extreme, because there's always the other side, and engineering is the art of making trade-offs: if you don't find room for them, the other side will bite you. So the points below basically boil down to these 2 things: examples of "complecting", and "things should be made as simple, as possible, but not simpler" (attributed to Einstein).

Interactive development

Lisp is considered a so called platform-language in a sense that it implies the existence of certain facilities and runtime environment constraints, that form a distinct and whole environment. Unlike some other languages, usually called scripting, which rely on a pre-existing environment, like the OS (e.g. Posix), web server, web browser etc. Other platform languages are, for example, Java, C# or Erlang, while scripting languages are JavaScript, Perl or PHP. Clojure is a language on a pre-existing platform, which is JVM, and so doesn't define its own platform. This is the source of, probably, the biggest complecting in the language, as it tries to be a stand-alone dynamic, functional language, explicitly discouraging imperative object-oriented style. But the JVM-platform is oriented at static imperative object-oriented ones.

From the Lisp dynamic point-of-view a lot of JVM's facilities are inferior:

• mostly static runtime image and object system with only partial opportunities for redefining things on-the-fly instead of a fully dynamic one
  
• static namespaces (tied to a file system) instead of dynamic packages
  
• static exception system instead of a dynamic (restartable) condition system
  
• limited and flawed number representation system instead of a full numeric tower
  
• more limited calling convention (only positional parameters and absense of multiple return values)
  
• more limited naming scheme
  
• XML-based build facilities instead of eDSL-based ones - althouh, here Clojure can provide its own option, but the currently existing one - Leiningen is a weird beast of its own (for example, you can get CLASSPATH out of it, but can't put it in except through project.clj, which has a lot of limitations)
  
• absense of tail call optimization

Surely, there are also some current advantages of the Java platform:

• availability of a concurrent GC (although it's not the default one)
  
• good JIT-optimizing compiler
  
• and what's most important, larger amount of available tools and libraries

Yet, if we return to the top of the list of shortfalls, we can see, why interactive development in Clojure is much less productive, then in Lisp. What adds to it is, that Clojure uses a one-pass compiler (not very modern).

Going into more details on this will be a whole separate post, so I'll just sum up, that interactive development in Clojure is hampered both by the JVM sticking in different places (especially, if you work on projects, combining Clojure and Java code) and Clojure's own misfeatures.

Syntax

From its early days Clojure was positioned as a "modern" Lisp. And what this "modernity" actually implied is:

• more accessible syntax and broader support for vectors and maps as opposed to Common Lisp, in which only lists, allegedly, were first-class citizens
  
• built-in concurrency primitives
  
• being Lisp-1 instead of Lisp-2, which makes heavy functional style programming more concise
  
• cleaning up some minor "annoyances": 4 equality operators, interning reader, etc.

Starting from the bottom, the minor issues make it not easy to start, but they are actually conceptually simple and useful, once you get accustomed. Lisp-1 vs Lisp-2 is a matter of taste: for example, I prefer #', because it's an annotation, while others perceive it as noise. And there's no objective advantage of one over another: yes, Lisp-1 makes something like (<map> </key>) instead of (gethash <map> <key>) possible, yet it makes macros more complicated. And concurrency I'll discuss separately.

What's left is broader support for vectors and maps, including destructuring. I agree, that declarative syntax for common datastructures is a crucial for productive use of any language up to the point of defining the same literal syntax ({}) for hash-tables in CL. Thankfully, that is supported by the language, so this syntax is as first-class in CL, as in Clojure, and, as in many aspects, nothing prevents "modernizing" Lisp in this aspect without creating a whole separate language... This doesn't hold for Clojure, as it doesn't have facilities to control the reader in the same way CL does: actually, in this regard Clojure is very different from Lisp — it hardly provides facilities for controlling any aspect of the language — and this control is a critical part of Lisp's DNA.

And pushing syntax to the extreme has it's shortcomings in Clojure as well. Rich argues, that defining both lists and vectors with parens (in Lisp list is '() and vector is #()) is complecting. But, I'd say, that a much harder case of complecting is this:

)))))))])))))])]) — the joy of debugging ASTs

And it's not even about Clojure: although here it's even worse, because for some reason, that escapes me, let (and defn, and many others) uses vectors for argument lists. Aren't they called lists for a reason? So this once again actualizes the problem of counting closing parens, effectively solved for Lisp long ago with emacs.

At the same time "modern" Clojure poorly supports such things as keyword arguments in functions or multiple return values and many other not so "modern", but very effective facilities, that I personally would expect to see in a modern language...

There's only one true way: functional

In my talk I referred to one of the ugliest pieces of code, I've ever written, which was a very complicated Clojrue loop, i.e. a loop/recur thing (and I've honestly tried to factor it out).

Basically there are two opposite approaches to iteration: imperative looping and functional recurring. Many languages have a strong bias towards one or another, like Python discouraging recursion and Clojure discouraging imperative loops by imposing immutability. But the thing is, that there are problems, for which one of the approaches yields by far more concise and understandable code. If you want to traverse a tree, recursion is a way to go. While if you are accumulating several sequences at once, which may reference results, obtained at the previous computations and also at each iteration there's not one, but several outcomes, recursions often becomes too messy. Yet in Clojure there's not even good support for recursion (which has an advantage of factoring different pieces of code into functions), but a special construct loop/recur, which shares the downsides of both approaches and does hardly provide any of the benefits. That's a pity, as iteration is the basic programming construct and no code file can do without it. And here we see a case of detrimental over-simplification.

And there are also lazy sequences, which complect sequences and streams. In theory, those are the same things, but, as the saying goes, in theory, theory and practice are the same, but in practice... Surely this makes writing a compiler easier at the cost of complicating reasoning about sequences in day-to-day programming.

EDIT: as was correctly pointed by the commentators, you can have sort of mutable local variables with transients, and that allows to express some of the imperative loops in a more concise manner.

Concurrency

In the early days of Clojure in one of the discussions on c.l.l. S.Madhu (if I remember correctly) called Clojure's concurrency primitives "snake oil". I thought, that there may be some personal issues in such attitude, but having tried it for myself and learned about all the alternatives in the concurrency space, I don't think it was too far from reality. First of all, Clojure addresses only shared-state concurrency on a single computer, while most hard concurrency problems appear in the context of distributed systems and are currently solved with approaches, like the Actor model or MapReduce. And on the single machine I've seen very few problems, that can't be solved with a combination of thread pools (with thread-local storage), Lisp-style special variables and databases. In fact Clojure provides its own (often limited) variants of all the above mentioned approaches and nothing more:
- agents instead of actors
- vars (analogue of CL's special variables)
- STM for databases
- map/reduce, pmap

To be able to provide them Clojure imposes some restrictions, most notably the one of immutability. Also in their implementation it doesn't follow the Lisp principle of giving control to the programmer: you can't swap one STM variant/strategy for the other. Heck, you can't even control the number of threads, that pmap uses!

Among all this approaches, STM is the most notable one (as others are just a copy of the established technologies). And it has a tough competition from databases and other transactional datastores. The advantages are no impedance mismatch and, possibly, better efficiency. Yet the database is language-agnostic, which is often useful, it's more accessible and understandable. And, what's most important: there's durability and there's a choice, that you can utilize, depending on your needs. The best use case for STM I've found so far was to hold statistical counters, accessed simultaneously from many threads, yet this problem is easily solvable with Redis, for example. And the same applies to other uses I can think of, So, the paradox of Clojure is that, although it was ushered with the idea of solving concurrency problems, it still has a lot to prove in this space: and not with toy examples of ant colonies, but real-world applications, made possible with its approach (like RabbitMQ or Ejabberd showcasing Erlang's aptitude for building massively parallel systems).

Final thoughts

I've written a long article, but there are many more details left out (just adding code snippets, will make it twice as big). There's also a lot of good things about Clojure, which I didn't mention: its seamless integration with Java, for example, which makes it a good scripting language for Java applications. Macros, basic datastructures, vars, they all work great. Ring was also an enlightenment.

Yet, overall the language doesn't leave up to its promise of being a modern Lisp: actually, it's not a Lisp at all. Well, how do you define Lisp? S-expressions, macros, closures etc? Surely, all (at least most of) those features may be present, although Dylan was a Lisp without s-expressions, for example. But, in my opinion as a Lisp programmer, what makes a true Lisp is dynamicity, simplicity and putting control in developer's hands (flexibility and extensibility). The mentioned features are mostly a derivative of this principles, and also they all should have a backend for extension: reader, compiler, condition system, object system, special variables etc, they all do in Lisp. And Clojure gives up on all these principles if not altogether, than substantially: many of the features are there and the new ones arrived, but the backend is missing.

Common Lisp is well optimized for the common use-cases in the design space of dynamic flexible languages. And it's probably very close to local maximum in it. At least it's good enough. Clojure is more of a hybrid, and hybrids are built by complecting: in Clojure's case complecting Lisp and Java, Lisp and Haskell. The result is interesting, but, in my opinion, it's not optimal. Surely it has its use-cases.

Still, Clojure sees decent adoption (which is good, because it proves, that parens aren't a problem after all :) I think it's a combination of several things. The language still has a lot of good stuff from Lisp (as well as some good things from Haskell, although they don't match very well), and those who start using it are mostly Java and Ruby programmers, for whom it breaks the psychological barriers to using Lisp (we all know the FUD). The other thing is lack of a decent native (not like Jython or JRuby) dynamic language on the JVM. And finally, there's marketing, i.e. Clojure concurrency primitives. And the challenge for Clojure community is to produce a "killer" application, that will utilize them, or the hype will wane, and pretty soon...

Videos from ECLM 2011

Today I've finally finished uploading all the videos to the ECLM channel on blip.tv/eclm. There are a lot of very interesting talks and the topics range from Lisp-based companies' experiences to exciting Lisp projects (like quicklisp or zen X server) to community issues (like the announcement of the Stichting Common Lisp Foundation or good Lisp style). Of 7 lengthy talks, announced at the event's website, only 5 are available now:

• Luke Gorrie's account of his exciting startup Teclo Networks will be published somewhat later not to compromise the business (and it's a great talk, so don't miss it :),
• and Paul Miller's showcase of LinkExplorer, a Windows LispWorks CAPI-based GUI application was not recorded by the request of his company.

In total, there are 12 videos, including the lightning talks (and the lucky 13th will join later). My personal favourite is Jack Harper's story of building Secure Outcomes, with a lot of invaluable engineering insights.

The sound quality is not stellar, because, unfortunetely, there was a laying of the ground 50Hz signal on the mic's signal at the site. But judging from my experiences in organizing TEDxKyiv, sound is the usual point of failure at conferences, and it's pretty hard to get it right. Yet, I hope, that next meeting will have a more professional recording set up (maybe through CLF), because, in my opinion, these videos are extremely valuable to the community. I really miss some talks from ECLM 2009, like Kuroda Hisao's, Dave McClure's or Dan Weinreib's, and most of all, the Piano.aero one...

For me the overall meeting experience was really fantastic, and the talks actually form only a minor half of that. Immersing myself into the Lisp community and participating in so many profound conversations with brilliant Lisp programmers was the really exciting part. The good overview of the conference experience is given by Luís Oliveira. So I'm really looking forward for the ECLM 2012. Maybe, organize it in a different European city this time?..

Book review: Algorithms of the Intelligent Web

TL;DR The book should have been named "Building Machine Learning Apps in Java for Dummies". Worth reading if such name excites you.

This book aims to become a comprehensive guide in commonly used Machine Learning technics. And generally it succeeds in that with one reservation: it more resembles a sophomore synopsis, than a professor's lecture.

I should say, that I couldn't manage to finish the book, and had read only Chapters 1,2 and 5. But the structure of each chapter is roughly the same: a brief high-level description of the common approaches in the described sphere. Then a list of Java libraries, available for the task. Selection of one of them and a description of how solid, robust and mature it is. Then several pages of Java code, showing how to prepare input parameters and actually make use of the libraries. Finally a note, that there are some limitations or pecularities, but discussing them in more detail is out of the scope of the book. Thus, the amount of new material for anyone at least slightly familiar with the subject and programming in general is minimal (I would say, that for me it was around 10%.) No insights, nothing novel, that can't be found on the first page of Google results on the topic, no account of personal experience or "war stories". Such book doesn't require 20 years of industrial experience under the hood to write — this could be done by any working programmer. The recipe seems simple: for each topic take the first two pages of Wikipedia entry on it, list the available libraries for solving the problem, add a couple of examples from their manual, bingo!

There's also another problem: the code is in Java. This is good for the authors, as it increases the book size and makes it seem much more solid, than in reality. But all the unnecessary cruft like half-line type declarations and bookkeeping code for loops (which amounts to more than half of the code, I'd say) makes it really hard to follow along. You need to really focus to be able to translate the code in mind to some high-level conceptual form before grasping it. (Although, there's a shortcut: just assume, that all the code is mostly irrelevant and skip). It's like the authors used the trick of all students: hide your poor understanding of the subject behind all the recondite words you've heard. So the advice for any algorithmical book: if you want to make the code easy to follow write it in Python or Lisp. At least it would make really apparent, if there's not a lot of essence to it... ;)

So the book is worth reading only if you're new to the field, don't want to spend time learning the subject, but need to get some working program fast (and don't object using Java for that).

various Lisp stuff

In the spirit of xach's "policy" for half-baked projects I've also published some of my experiments:

• cl-git-store
• ASDF2 broader version support proposal

If somebody finds that useful, let me know.

Роль CTO в технологической компании

Есть известная максима, давно проповедуемая Полом Гремом и взятая на вооружение большей частью американского стартап-сообщества, о том, что технологические стартапы должны иметь минимум двух основателей. Можно много дискутировать на тему, с чем это связанно: например, есть версия, что "если предприниматель не способен убедить в своей идее хотя бы еще одного человека, то он тем более не сможет убедить в этом рынок". Я же пришел к выводу, что всё упирается в ответ на вопрос, что можно аутсорсить, а что нет? Как известно, нельзя этого делать с ключевыми функциями компании. И первой ключевой функцией является формирование ее идентичности на рынке. Этим занимается человек, называемый американцами CEO, хотя более правильно было бы назвать его просто лидером проекта. По своей базе он может быть как инженером, так и маркетологом, финансистом или продажником. Это не столь важно, поскольку его главная роль, которая заключается в том, чтобы вести компанию в правильном направлении и рассказывать об этом рынку через маркетинг, биздев, продажи, PR и т.д. и т.п., не требует особой специализации. Очевидно, что при этом он должен стоять у истоков идеи, которая реализуется, и сам крепко верить в нее, иначе он не сможет убедить в ней других. И эта роль настолько обширна, что она должна быть единственной сферой ответственности такого человека. Да, он может быть крутым программистом и вносить свой вклад, а, воможно, и наибольшее ноу-хау в технологии своей компании, но он не может быть ответственным за эту сферу в целом. Потому что, если отвечать за два дела одновременно, то ни одно из них не выйдет сделать хорошо. И это показывает опыт, в том числе и мой собственный.

Вот здесь и выходит на арену CTO, роль которого в компании заключается в технологическом лидерстве. Часто считается, что СТО должен написать основную часть кода, если это касается софтверного стартапа. Это не обязательно. Код могут написать хоть 10 фрилансеров или 100 миллионов обезьян, поскольку это довольно стандартная работа, и если правильно поставить задачу, с ней прекрасно справится наемный сотрудник. Но вот, что сложно — так это поставить задачу и сформулировать рамки, в которых должен развиваться проект, а потом следить за тем, чтобы они соблюдались. В технологическом бизнесе, как ни крути, а основные риски все-таки технологические. И все остальные участники процесса (наемные сотрудники, клиенты и т.д.) будут их только увеличивать. Единственный, кто может и должен с ними бороться — это СТО.

СТО — это роль того, кто принимает окончательное решение, выбрать технологию Х, за которую выступает разработчик, или Y, которая сейчас на волне хайпа. Это тот, кто не спит ночами из-за того, что не настроен бекап базы данных. Это тот, кто вводит практику ревью кода и добивается ее реального воплощения. Тот, кто отвечает программистам на вопросы: "зачем?", "почему?" и "когда?" Тот, кто отбирает подходящих инженеров. И принимает еще 100500 разных решений, а потом добивается их исполнения. В общем, это тот, кто создает технологическую культуру компании и не дает появляться технологическуму долгу, который в будущем (и, скорее, близком, чем отдаленном) потянет компанию ко дну или же не даст ей развиваться необходимыми темпами.

Так что правило про двух основателей я переформулировал бы так.

На этапе становления у технологического бизнеса есть 2 ключевые роли, которые должны выполнять 2 разных основателя компании: бизнес-лидер и технолонический лидер.