Steve Vinoski's Blog

Ted Neward finally responds to my comments about his remarks concerning Erlang. I really don’t mean to pick on Ted — I like Ted! — but unfortunately, this time around his response misses the mark in more ways than one.

First, Ted says:

Erlang’s reliability model–that is, the spawn-a-thousand-processes model–is not unique to Erlang. In fact, it’s been the model for Unix programs and servers, most notably the Apache web server, for decades. When building a robust system under Unix, a master-slave model, in which a master process spawns (and monitors) n number of child processes to do the actual work, offers that same kind of reliability and robustness. If one of these processes fail (due to corrupted memory access, operating system fault, or what-have-you), the process can simply die and be replaced by a new child process.

There’s really no comparison between the UNIX process model (which BTW I hold in very high regard) and Erlang’s approach to achieving high reliability. They are simply not at all the same, and there’s no way you can claim that UNIX “offers that same kind of reliability and robustness” as Erlang can. If it could, wouldn’t virtually every UNIX process be consistently yielding reliability of five nines or better?

Obviously, achieving high reliability requires at least two computers. On those systems, what part of the UNIX process model allows a process on one system to seamlessly fork child processes on another and monitor them over there? Yes, there are ways to do it, but would anyone claim they are as reliable and robust as Erlang’s approach? I sure wouldn’t. Also, UNIX pipes provide IPC for processes on the same host, but what about communicating with processes on other hosts? Yes, there are many, many ways to achieve that as well — after all, I’ve spent most of my career working on distributed computing systems, so I’m well aware of the myriad choices here — but that’s actually a problem in this case: too many choices, too many trade-offs, and far too many ways to get it wrong. Erlang can achieve high reliability in part because it solves these issues, and a whole bunch of other related issues such as live code upgrade/downgrade, extremely well.

Ted continues:

There is no reason a VM (JVM, CLR, Parrot, etc) could not do this. In fact, here’s the kicker: it would be easier for a VM environment to do this, because VM’s, by their nature, seek to abstract away the details of the underlying platform that muddy up the picture.

In your original posting, Ted, you criticized Erlang for having its own VM, yet here you say that a VM approach can yield the best solution for this problem. Aren’t you contradicting yourself?

It would be relatively simple to take an Actors-based Java application, such as that currently being built in Scala, and move it away from a threads-based model and over to a process-based model (with the JVM constuction[sic]/teardown being handled entirely by underlying infrastructure) with little to no impact on the programming model.

Would it really be “relatively simple?” Even if what you describe really were relatively simple, which I strongly doubt, there’s still no guarantee that the result would help applications get anywhere near the levels of reliability they can achieve using Erlang.

As to Steve’s comment that the Erlang interpreter isn’t monitorable, I never said that–I said that Erlang was not monitorable using current IT operations monitoring tools. The JVM and CLR both have gone to great lengths to build infrastructure hooks that make it easy to keep an eye not only on what’s going on at the process level (“Is it up? Is it down?”) but also what’s going on inside the system (“How many requests have we processed in the last hour? How many of those were successful? How many database connections have been created?” and so on). Nothing says that Erlang–or any other system–can’t do that, but it requires the Erlang developer build that infrastructure him-or-herself, which usually means it’s either not going to get done, making life harder for the IT support staff, or else it gets done to a minimalist level, making life harder for the IT support staff.

I know what you meant in your original posting, Ted, and my objection still stands. Are you saying here that all Java and .NET applications are by default network-monitoring-friendly, whereas Erlang applications are not? I seem to recall quite a bit of effort spent by various teams at my previous employer to make sure our distributed computing products, including the Java-based products and .NET-based products, played reasonably well with network monitoring systems, and I sure don’t recall any of it being automatic. Yes, it’s nice that the Java and CLR guys have made their infrastructure monitorable, but that doesn’t relieve developers of the need to put actual effort into tying their applications into the monitoring system in a way that provides useful information that makes sense. There is no magic here, and in my experience, even with all this support, it still doesn’t guarantee that monitoring support will be done to the degree that the IT support staff would like to see.

And do you honestly believe Erlang — conceived, designed, implemented, and maintained by a large well-established telecommunications company for use in highly-reliable telecommunications systems — would offer nothing in the way of tying into network monitoring systems? I guess SNMP, for example, doesn’t count anymore?

(Coincidentally, I recently had to tie some of the Erlang stuff I’m currently working on into a monitoring system which isn’t written in Erlang, and it took me maybe a quarter of a workday to integrate them. I’m absolutely certain it would have taken longer in Java.)

But here’s the part of Ted’s response that I really don’t understand:

So given that an execution engine could easily adopt the model that gives Erlang its reliability, and that using Erlang means a lot more work to get the monitorability and manageability (which is a necessary side-effect requirement of accepting that failure happens), hopefully my reasons for saying that Erlang (or Ruby’s or any other native-implemented language) is a non-starter for me becomes more clear.

Ted, first you state that an execution engine could (emphasis mine) “easily adopt the model that gives Erlang its reliability,” and then you say that it’s “a lot more work” for anyone to write an Erlang application that can be monitored and managed? Aren’t you getting those backwards? It should be obvious that in reality, writing a monitorable Erlang app is not hard at all, whereas building Erlang-level reliability into another VM would be a considerably complicated and time-consuming undertaking.

A commenter named Nick left a thoughtful response to my post about my “Multilanguage Programming” column. Rather than respond to it with another comment, I thought I’d turn my response into a full posting, as I think Nick’s feedback is representative of how many people feel about the topic.

Nick said:

I would say that instead of spending a lot of time on a conceptually different language it could be more beneficial to study, say, distributed algorithms or software/system architecture principles or your business domain. There is so much knowledge in this world that learning how to code the same thing in, roughly speaking, one more syntax seems like a waste of time. Even paying real attention to what is going on in the cloud computing can easily consume most of one’s spare time.

I think there are assumptions here that are not necessarily true. Specifically, you’re not necessarily learning how to code the same thing in multiple languages; rather, the idea is that by choosing the best language for the task, coding is “just right” for the problem at hand. For example, I know from significant first-hand experience that if you want to write a set of distributed services that support replication, failover, and fault tolerance, the code you’d write to do that in C++ will be extremely different from the code you’d write in Erlang to achieve the same thing (well, actually, you’d be able to achieve far more in Erlang, in far fewer lines of code).

This is about much more than syntax. It’s about facilities, semantics, and trade-offs. If it were just syntax, then that would imply that all languages are equal in terms of expressiveness and capability, which we already know and accept to be untrue.

The cloud computing topic actually provides a good example of why knowing multiple languages can be useful. To use the Google App Engine, for example, you need to develop your applications in Python. What if you don’t know Python? Too bad for you.

From a real life perspective, it takes years or working on nontrivial software to master a language. For example, some people still manage to have only a vague idea of util.concurent — and this is just a small enough (and well explained in the literature) part of Java. How realistic is it to expect that the majority of developers will be able to master multiple languages concurrently?

I disagree that it takes years to master a language. One of the best OO developers I ever worked with was a mechanical engineer who taught himself programming. One of my current coworkers — a relatively young guy — started programming in Erlang only a few months ago, and he’s already writing some fairly sophisticated production-quality code. In 1988, I started using C++; by 1989, I was starting to help guys like Stan Lippman, Jim Coplien, and others correct coding mistakes in their excellent books. I have a BSEE, no formal computer science training whatsoever, and am completely self-taught as far as programming languages go. (The only class I ever had in any computer language was a BASIC class I had to take in 1981.) Two other coworkers started with Python just a few months ago and they do quite well with it at this point. I can cite numerous such examples from throughout my career. I don’t think any of us are super-programmers or anything like that, so if we can do it, I don’t see why it would be a problem for anyone else.

Perhaps you’re falling trap to the “huge language” problem I mentioned in my column. It certainly can take some people many years to master enormous languages like Java and C++, but most languages are simply nowhere near that big.

And who wants to maintain a code base written in widely different languages? Which most likely means multiple IDEs, unit testing frameworks, build systems (hey, not everyone is using even Maven yet), innumerable frameworks etc. And most of the interpreted languages among those are not even likely to run in the same VM. Not to mention the number of jobs asking for non-C++/Java skills.

I use a number of languages daily and I really have no trouble maintaining the code regardless of which language any particular piece happens to be written in, or whether I wrote the code or one of my teammates did. Once you know a language, you know it; switching to it is no more difficult than using your one and only language if you’re a monolingual developer.

You also mention the “multiple IDE” problem. The first draft of my column contained some fairly direct language describing my dislike of IDEs, or more accurately, my dislike of the IDE culture, but Doug Lea suggested I take it out, so I did. The problem is that some folks let the tool drive their solutions, rather than using the tool as a means to developing solutions. I’ve had numerous people tell me they won’t consider using a language unless their IDE fully supports it with Java- or Smalltalk-like refactoring. To me, that’s completely backwards. I’d rather use an extensible editor that can handle pretty much any language, thus letting me develop optimal solutions using the right languages, rather than having a mere editing tool severely limit my choice of possible solutions.

But there are language mavens and there are tool mavens, and they typically disagree. Follow that link and read, as the posting there is incredibly insightful. I am definitely a language maven; languages are my tools. I suspect, though, that Nick and others who raise similar questions to the ones quoted here lean more toward being tool mavens. I’m not passing judgment on either; I’m only pointing out the different camps to help pinpoint sources of disagreement.

As far as unit test frameworks, build systems, and frameworks go, I haven’t ever found any big issues in those areas when using multiple languages. The reason, not surprisingly, is that knowing multiple languages gives you multiple weapons for testing and integration. Ultimately, when you’re used to using multiple languages, you’re used to these kinds of issues and thus they don’t really present any formidable barriers.

And as far as jobs go, the best developers I’ve known throughout my career have been fluent in a number of programming languages, and each of them could work virtually wherever they wanted to. I don’t believe this correlation is mere coincidence.

Curiously enough, this argumentation is hardly ever mentioned. Authors tend to assume that developers are lazy or have nothing else to learn.

I don’t assume developers are lazy. Rather, I think our industry generally has a bad habit of continually seeking homogeneity in platforms, in languages, in tools, in frameworks, etc., and we really, really ought to know better by now. Once you learn to accept the fact that heterogeneity in computing is inevitable — since nothing can do it all, right? — you find yourself able to use that heterogeneity to your advantage, rather than continually battling against it and losing.

Personally, I am planning to look at Scala and probably Erlang but even judging from the number of books on those it’s clear to me that they represent merely a niche market.

Today’s niche market is tomorrow’s mainstream market. Regardless of whether either of those languages continues to grow, learning one or both will make you a better developer than you are today.

Consider the final question I ask in my column:

After all, do any of us really believe we’ve already learned the last programming language we’ll ever need?

I suspect the vast majority of developers would answer “no” to this question. Assuming that’s the case, then if you don’t regularly practice learning new languages, how do you know when you really need to start learning a new one, and how capable will you be of learning that next language when the need arises? The longer you stay with one language, the more isolated you become, typically without even realizing it. Shifting gears gets harder and harder. Then one day you look up and all the interesting work is elsewhere, out of your reach. Is that a position you want to put yourself in?

I finally got around to watching and listening to Bob Ippolito‘s “Exploring Erlang” video, and I don’t know of any more thorough hour-long introduction to the language than this one. If you’re interested in Erlang but are too busy or too lazy to read Joe Armstrong’s great book or any of the many papers and articles that describe Erlang, then Bob’s talk is for you.

A few interesting points for me:

• In a few places Bob mentions that while he finds Erlang syntax unusual it’s really no big deal, and you quickly get used to it. I can verify that he’s absolutely right. A lot of people seem to get really hung up on this issue; if you’re one of them, all I can say is that your concerns are very greatly exaggerated.
• Along the lines of the previous point, one of the questions Bob takes after his presentation is from a gentleman who says that his technical lead refuses to use Erlang — in a telephony application, no less — and that his tech lead rolled his own stuff for concurrency and interprocess communication. Hmm, I wonder if that technical leader’s initials are “N.I.H.” It would be interesting to see how well his solution could hold up in the environments in which Erlang has already well proven itself. Given all the attempts I’ve seen over my career at solving such things, many of them made by people who mistakenly think it’s not that difficult to get right, my guess is “not that well.”
• Bob mentioned that he finds Erlang to be a very small and thus easily learnable language. He’s absolutely right about that, too; I wish I could count the number of times I’ve said the exact same thing in conversation, in presentations, and in writing. Many people who come from Java or C++ or other general-purpose languages assume that languages need to be huge to be useful, so they assume that small languages must be toys, and they also assume any “real” language requires years to learn. How incredibly wrong they are.
• Bob keeps the whole discussion grounded by explaining how Erlang saves him and his team time and money. In case it’s not clear, he’s talking about actual time and actual money. He explains that he can serve millions of web requests per day from just a single machine (though he naturally uses more than one for redundancy and reliability), for example, and by mentioning the small size of his team — I think he said 8? — he hints at the productivity that Erlang affords. My experiences are, again, quite similar.

Coincidentally I also watched Jim Weirich‘s “Shaving with Occam” presentation from the MountainWest RubyConf 2008 held at the end of March, and was pleasantly surprised to find that he devoted a good portion of his talk to Erlang. He mentioned how it was like Lisp in the sense that it has a small core, simple rules for manipulating that core, and powerful abstractions for building new things from the simple rules and small core. Yet, I shouldn’t be surprised. I don’t know Jim at all, but judging from the dates he mentions in his talk I think his career spans around 30 years. It’s also pretty clear that he’s had broad and deep experience with many technologies during that time. From what I’ve seen, people like Jim who have an incredibly deep first-hand understanding of the utility and importance of simplicity, brevity, and powerful abstractions in software development seem to have no problem whatsoever quickly seeing and comprehending the immense value of Erlang.

You’re still not exploring Erlang? What are you waiting for?

I just returned home from QCon London, and its excellence exceeded my expectations. As usual, the quality of speakers QCon attracts (just like JAOO) is outstanding, and they cover a very wide variety of topics.

Kent Beck‘s keynote was excellent. It was about developer responsibility, developer integrity, and the relationships developers have with those around them (here’s a good summary). Extremely insightful, not unexpectedly of course, and covering important topics that are unfortunately often taboo among technical folk.

It’s been awhile since I spent any quality time with Kent, but we did manage to sneak off to a back room at the speaker party on Thursday evening and get 20 minutes or so of conversation in, which is better than nothing. Given that the first two to drop in were Jim Webber and Joe Armstrong, though, neither of us minded the extra company. Generally, though, I think I could easily spend a couple solid days talking to Kent about development issues.

I spoke in Stefan Tilkov‘s track on Thursday, and I thought it went OK. Stefan has already provided detailed notes on each talk (except his own, which was really good) on his blog, so I won’t bother with repeating any of that. I got to meet Paul Fremantle in person for the first time, which was nice, though I have to say I remain puzzled by his “REST is too hard for the average developer” claim he made in his talk, since it definitely doesn’t match my recent experiences with watching others learn it and apply it. I’ve heard Sanjiva say the same thing before as well, so I wonder if it’s just something they keep telling themselves, hoping it will come true if they repeat it enough. ;-)

On Friday I thoroughly enjoyed Simon Peyton-Jones‘s Haskell talk and Joe Armstrong‘s Erlang talk. Both were excellent speakers, and both talks ventured into areas that I’m sure many of the attendees were not familiar with, so I’m sure most everyone who listened learned a lot. Thankfully there were no “Haskell vs. Erlang” wars, but that’s not surprising given that Simon and Joe are friends, plus they know that that argument serves neither language.

The best part of the week, though, was getting to meet and hang out with Joe and other Erlang folk. Joe’s really an excellent guy. He’s quite energetic, and his brain just doesn’t stop. He’s curious about a lot of technical things beyond Erlang, and I found discussions with him to be full of interesting questions and insights. Given the fact that I work with Erlang quite a lot these days, my hope going in was simply that I’d get a chance to just say hi to him, but I turned out to be lucky enough to spend many hours with him over the course of the conference. I also met Francesco Cesarini, a well-known and long-time Erlang consultant, whom I’m sure has probably forgotten more Erlang than I’ve learned so far. I also met Alexis Richardson, who works on RabbitMQ, the right way to implement AMQP. Francesco graciously invited me to speak at the Erlang Exchange this summer, so I’ll be seeing them all again before too long to talk even more about what’s quickly become one of my favorite programming languages.

Congrats to Floyd, Kresten, and the whole QCon team for putting on yet another excellent conference. I’m looking forward to JAOO Australia (in both Brisbane and Sydney) next!

When I spoke at QCon San Francisco last November, Stefan Tilkov interviewed me, and the video is now available on InfoQ.com.

We covered a range of topics: CORBA, dynamic languages, REST, distribution, concurrency, Erlang. Stefan asked some great questions, and I hope I gave some worthwhile answers. Thanks again, Stefan.