Thinking in Language, But Not Clearly

May 9th, 2008  |  Published in commentary, distributed systems, erlang, languages, reliability  |  8 Comments  |  Bookmark on

Ted Neward finally responds to my comments about his remarks concerning Erlang. I really don’t mean to pick on Ted — I like Ted! — but unfortunately, this time around his response misses the mark in more ways than one.

First, Ted says:

Erlang’s reliability model–that is, the spawn-a-thousand-processes model–is not unique to Erlang. In fact, it’s been the model for Unix programs and servers, most notably the Apache web server, for decades. When building a robust system under Unix, a master-slave model, in which a master process spawns (and monitors) n number of child processes to do the actual work, offers that same kind of reliability and robustness. If one of these processes fail (due to corrupted memory access, operating system fault, or what-have-you), the process can simply die and be replaced by a new child process.

There’s really no comparison between the UNIX process model (which BTW I hold in very high regard) and Erlang’s approach to achieving high reliability. They are simply not at all the same, and there’s no way you can claim that UNIX “offers that same kind of reliability and robustness” as Erlang can. If it could, wouldn’t virtually every UNIX process be consistently yielding reliability of five nines or better?

Obviously, achieving high reliability requires at least two computers. On those systems, what part of the UNIX process model allows a process on one system to seamlessly fork child processes on another and monitor them over there? Yes, there are ways to do it, but would anyone claim they are as reliable and robust as Erlang’s approach? I sure wouldn’t. Also, UNIX pipes provide IPC for processes on the same host, but what about communicating with processes on other hosts? Yes, there are many, many ways to achieve that as well — after all, I’ve spent most of my career working on distributed computing systems, so I’m well aware of the myriad choices here — but that’s actually a problem in this case: too many choices, too many trade-offs, and far too many ways to get it wrong. Erlang can achieve high reliability in part because it solves these issues, and a whole bunch of other related issues such as live code upgrade/downgrade, extremely well.

Ted continues:

There is no reason a VM (JVM, CLR, Parrot, etc) could not do this. In fact, here’s the kicker: it would be easier for a VM environment to do this, because VM’s, by their nature, seek to abstract away the details of the underlying platform that muddy up the picture.

In your original posting, Ted, you criticized Erlang for having its own VM, yet here you say that a VM approach can yield the best solution for this problem. Aren’t you contradicting yourself?

It would be relatively simple to take an Actors-based Java application, such as that currently being built in Scala, and move it away from a threads-based model and over to a process-based model (with the JVM constuction[sic]/teardown being handled entirely by underlying infrastructure) with little to no impact on the programming model.

Would it really be “relatively simple?” Even if what you describe really were relatively simple, which I strongly doubt, there’s still no guarantee that the result would help applications get anywhere near the levels of reliability they can achieve using Erlang.

As to Steve’s comment that the Erlang interpreter isn’t monitorable, I never said that–I said that Erlang was not monitorable using current IT operations monitoring tools. The JVM and CLR both have gone to great lengths to build infrastructure hooks that make it easy to keep an eye not only on what’s going on at the process level (“Is it up? Is it down?”) but also what’s going on inside the system (“How many requests have we processed in the last hour? How many of those were successful? How many database connections have been created?” and so on). Nothing says that Erlang–or any other system–can’t do that, but it requires the Erlang developer build that infrastructure him-or-herself, which usually means it’s either not going to get done, making life harder for the IT support staff, or else it gets done to a minimalist level, making life harder for the IT support staff.

I know what you meant in your original posting, Ted, and my objection still stands. Are you saying here that all Java and .NET applications are by default network-monitoring-friendly, whereas Erlang applications are not? I seem to recall quite a bit of effort spent by various teams at my previous employer to make sure our distributed computing products, including the Java-based products and .NET-based products, played reasonably well with network monitoring systems, and I sure don’t recall any of it being automatic. Yes, it’s nice that the Java and CLR guys have made their infrastructure monitorable, but that doesn’t relieve developers of the need to put actual effort into tying their applications into the monitoring system in a way that provides useful information that makes sense. There is no magic here, and in my experience, even with all this support, it still doesn’t guarantee that monitoring support will be done to the degree that the IT support staff would like to see.

And do you honestly believe Erlang — conceived, designed, implemented, and maintained by a large well-established telecommunications company for use in highly-reliable telecommunications systems — would offer nothing in the way of tying into network monitoring systems? I guess SNMP, for example, doesn’t count anymore?

(Coincidentally, I recently had to tie some of the Erlang stuff I’m currently working on into a monitoring system which isn’t written in Erlang, and it took me maybe a quarter of a workday to integrate them. I’m absolutely certain it would have taken longer in Java.)

But here’s the part of Ted’s response that I really don’t understand:

So given that an execution engine could easily adopt the model that gives Erlang its reliability, and that using Erlang means a lot more work to get the monitorability and manageability (which is a necessary side-effect requirement of accepting that failure happens), hopefully my reasons for saying that Erlang (or Ruby’s or any other native-implemented language) is a non-starter for me becomes more clear.

Ted, first you state that an execution engine could (emphasis mine) “easily adopt the model that gives Erlang its reliability,” and then you say that it’s “a lot more work” for anyone to write an Erlang application that can be monitored and managed? Aren’t you getting those backwards? It should be obvious that in reality, writing a monitorable Erlang app is not hard at all, whereas building Erlang-level reliability into another VM would be a considerably complicated and time-consuming undertaking.


  1. Patrick Logan says:

    May 12th, 2008 at 10:18 pm (#)

    Hmm. Yeah. Seen several attempts to build “erlang in x” where x = common lisp, python, java, dotnet, etc. To the extent I’ve followed them they implement a fraction of the message passing and then don’t go much further. Scala, etc. on the jvm and those on other conventional runtimes have a problem where to the extent they can integrate with those runtimes. e.g. the java classes etc. they can interact with a not shared-nothing, so you have to be aware of what is and what is not following the actor paradigm.

    As for monitoring, certainly a quick glance would have shown Ted there is SNMP, as Steve points out. But also isn’t the entire process-linking and supervisor hierarchy an important kind of “monitoring”, i.e. “monitoring for availability”? This is something all these other platforms lack.

    So the more I think about it, the less these other platforms have for messaging *or* monitoring!

  2. Mohan Radhakrishnan says:

    May 14th, 2008 at 4:03 am (#)

    Not related to Network-monitoring means but JVM monitoring includes facilities like DTrace probes and

  3. Bruce Rennie says:

    May 15th, 2008 at 6:39 am (#)

    Good evening Steve,

    I got the reference to this from reading Patrick Logan (as above). Erlang is just one of a number of languages of the concurrent flavour. It is not the first of the ilk nor will it be the last. Many of the ideas that are espoused by Erlang I came across in the early 80’s (in particular a research language called – I believe – Concurrent) in various research papers published at that time.

    There are few languages that can truly say that they are the first of their kind, most will utilise ideas from all sorts of sources. Few can the COBOL, Fortran, Algol, APL, Smalltalk, LISP, SNOBOL or Forth of their day today. There will be cross-fertilisation of ideas – some will be successful and some will not. That is the nature of the game that people play.

    The real question is – can we learn from what we see and make things easier and more reliable? If we can’t, then what are we doing?


    Bruce Rennie
    (God’s Own Country Downunder)

  4. steve says:

    May 15th, 2008 at 5:43 pm (#)

    Bruce: I don’t disagree, and I don’t think anyone from the Erlang community would disagree either. For example, when I spent time with him in London a few months ago, Joe Armstrong spoke often about the other languages that influenced Erlang.

    You ask whether we can learn from what we see and make things easier and more reliable, and I agree that’s a worthwhile question. What isn’t worthwhile, though, is the activity that Patrick Logan and I both objected to in our blogs: the hand-waving claims of “Oh, my language could easily do what Erlang does.” They’re always made by people who have never actually used Erlang, they’re wildly incorrect, and I believe they can actually inhibit or even prevent the kind of learning you mentioned in your comment above.

  5. Weekly software development and architecture crumbs #4 - Service Endpoint says:

    May 16th, 2008 at 6:13 pm (#)

    […] Steve Vinoski The language debate continues […]

  6. Steve Nicolai says:

    May 16th, 2008 at 10:11 pm (#)

    I’d love to see someone that knows QNX well compare it’s process model (which I believe supports multiple computers) and the Erlang process model.

  7. Barry Kelly says:

    May 18th, 2008 at 5:21 pm (#)

    I’m on your side of this debate, Steve. I think Ted is missing an appreciation of the gestalt of Erlang, and doesn’t quite appreciate some of the more fundamental differences between the process model of Erlang versus the threads of the JVM. I’ve expanded a bit more on my own blog.

  8. The Scala vs Erlang whirlwind at Ted Leung on the Air says:

    May 23rd, 2008 at 2:52 am (#)

    […] Vinoski (1, 2, 3) – Steve’s articles are more about the reliability aspects of Erlang, and he’s […]