Steve Vinoski's Blog

A technical discussion stops being a vehicle for learning when the following start to occur:

• Someone starts making stuff up.
• Instead of answering questions put to them, someone starts pointing out “flaws” in the questions themselves.
• One challenges the other to some sort of programming contest.
• Name calling.

The first two aren’t so bad, but when either of the latter two appears, it’s time to stop. Unfortunately, the third item has now entered my back-and-forth with Ted Neward. Since Ted has given me the last word, I’ll take it, but it’s clearly time to move on.

Given that a number of statements Ted’s made about Erlang in this discussion simply aren’t true, it’s quite clear Ted has never written any production Erlang code. [Update: Patrick Logan has posted a detailed analysis of Ted’s misunderstandings of Erlang.] Being a long-time author, it bothers me when people write authoritatively on topics they have no business writing about, so my only goal with my responses in this conversation has simply been to set the record straight with respect to Erlang. Ted originally said Erlang was a study in concurrency; I merely pointed out that it was more importantly a study in reliability. That’s really not even debatable. Unfortunately, it’s turned into a frustrating one-sided conversation because Ted lacks any detailed knowledge of Erlang, so he keeps unhelpfully trying to shift the focus elsewhere.

In his past two responses, Ted has picked at my questions like a grammar school English teacher, accusing me of conflating things, making bad assumptions, etc. I see that Patrick Logan is trying to clarify things, which might help. Yet Ted still hasn’t adequately explained why he’s taken such a hard stance against reliability being a fundamental feature of Erlang, nor how UNIX processes and Erlang processes are the same, as he keeps asserting, nor has he explained why he thinks it’s much, much harder to make an Erlang application manageable and monitorable than it is to build Erlang’s reliability into other systems like the JVM or Scala.

But now, we see the worst: the “programmer challenge.” Ugh. Thankfully, I’m sure most readers know that a programming contest of the sort Ted proposes would prove absolutely nothing. I guess he proposed it because I mentioned how I recently spent a quarter of a day making an Erlang application monitorable, in response to his continued claims that doing so is really hard, so now he wants to make a competition of it. I’d rather that you just explain, Ted, the experiences you’ve had that have led you to claim that Erlang applications can’t be easily managed or monitored. Better yet, since you’re the one who wants a contest, and given that you’re the one making all the claims, why don’t you go off and see how quickly you can build Erlang’s reliability into Scala and the JVM, since you claim it’s so simple?

If you’re a regular reader of Ted’s blog, you know that Ted generally offers good advice and you can learn useful things from him. He’s a good writer and a wonderful conference presenter, as he can make hard concepts easier to grok and generally does so with humor to keep you awake. But I feel that anyone in Ted’s position has a responsibility to avoid passing off incorrect information to his readers as fact. My advice therefore is simply that you don’t take what Ted says as gospel for this particular topic. Let me assure you that Erlang offers far, far more value than just exceptional concurrency support, which is where Ted’s initial posting in this thread seemed to want to limit it, and which is all I objected to. Unlike Ted, I’ve written quite a bit of Erlang code, and I use it every single day. If you write distributed systems, you owe it to yourself to explore Erlang’s capabilities and features. I’ve been writing and researching middleware and distributed systems for nearly 20 years now, and I’ve seen a lot over the years. Erlang is by far the most innovative and sound approach to distributed systems development I’ve ever seen and experienced — the trade-offs its designers chose are simply excellent. Like I’ve said numerous times over the past year, I really wish I’d found Erlang a decade ago, because I know for certain it would have saved my teams and me countless hours of development time.

Ted Neward finally responds to my comments about his remarks concerning Erlang. I really don’t mean to pick on Ted — I like Ted! — but unfortunately, this time around his response misses the mark in more ways than one.

First, Ted says:

Erlang’s reliability model–that is, the spawn-a-thousand-processes model–is not unique to Erlang. In fact, it’s been the model for Unix programs and servers, most notably the Apache web server, for decades. When building a robust system under Unix, a master-slave model, in which a master process spawns (and monitors) n number of child processes to do the actual work, offers that same kind of reliability and robustness. If one of these processes fail (due to corrupted memory access, operating system fault, or what-have-you), the process can simply die and be replaced by a new child process.

There’s really no comparison between the UNIX process model (which BTW I hold in very high regard) and Erlang’s approach to achieving high reliability. They are simply not at all the same, and there’s no way you can claim that UNIX “offers that same kind of reliability and robustness” as Erlang can. If it could, wouldn’t virtually every UNIX process be consistently yielding reliability of five nines or better?

Obviously, achieving high reliability requires at least two computers. On those systems, what part of the UNIX process model allows a process on one system to seamlessly fork child processes on another and monitor them over there? Yes, there are ways to do it, but would anyone claim they are as reliable and robust as Erlang’s approach? I sure wouldn’t. Also, UNIX pipes provide IPC for processes on the same host, but what about communicating with processes on other hosts? Yes, there are many, many ways to achieve that as well — after all, I’ve spent most of my career working on distributed computing systems, so I’m well aware of the myriad choices here — but that’s actually a problem in this case: too many choices, too many trade-offs, and far too many ways to get it wrong. Erlang can achieve high reliability in part because it solves these issues, and a whole bunch of other related issues such as live code upgrade/downgrade, extremely well.

Ted continues:

There is no reason a VM (JVM, CLR, Parrot, etc) could not do this. In fact, here’s the kicker: it would be easier for a VM environment to do this, because VM’s, by their nature, seek to abstract away the details of the underlying platform that muddy up the picture.

In your original posting, Ted, you criticized Erlang for having its own VM, yet here you say that a VM approach can yield the best solution for this problem. Aren’t you contradicting yourself?

It would be relatively simple to take an Actors-based Java application, such as that currently being built in Scala, and move it away from a threads-based model and over to a process-based model (with the JVM constuction[sic]/teardown being handled entirely by underlying infrastructure) with little to no impact on the programming model.

Would it really be “relatively simple?” Even if what you describe really were relatively simple, which I strongly doubt, there’s still no guarantee that the result would help applications get anywhere near the levels of reliability they can achieve using Erlang.

As to Steve’s comment that the Erlang interpreter isn’t monitorable, I never said that–I said that Erlang was not monitorable using current IT operations monitoring tools. The JVM and CLR both have gone to great lengths to build infrastructure hooks that make it easy to keep an eye not only on what’s going on at the process level (“Is it up? Is it down?”) but also what’s going on inside the system (“How many requests have we processed in the last hour? How many of those were successful? How many database connections have been created?” and so on). Nothing says that Erlang–or any other system–can’t do that, but it requires the Erlang developer build that infrastructure him-or-herself, which usually means it’s either not going to get done, making life harder for the IT support staff, or else it gets done to a minimalist level, making life harder for the IT support staff.

I know what you meant in your original posting, Ted, and my objection still stands. Are you saying here that all Java and .NET applications are by default network-monitoring-friendly, whereas Erlang applications are not? I seem to recall quite a bit of effort spent by various teams at my previous employer to make sure our distributed computing products, including the Java-based products and .NET-based products, played reasonably well with network monitoring systems, and I sure don’t recall any of it being automatic. Yes, it’s nice that the Java and CLR guys have made their infrastructure monitorable, but that doesn’t relieve developers of the need to put actual effort into tying their applications into the monitoring system in a way that provides useful information that makes sense. There is no magic here, and in my experience, even with all this support, it still doesn’t guarantee that monitoring support will be done to the degree that the IT support staff would like to see.

And do you honestly believe Erlang — conceived, designed, implemented, and maintained by a large well-established telecommunications company for use in highly-reliable telecommunications systems — would offer nothing in the way of tying into network monitoring systems? I guess SNMP, for example, doesn’t count anymore?

(Coincidentally, I recently had to tie some of the Erlang stuff I’m currently working on into a monitoring system which isn’t written in Erlang, and it took me maybe a quarter of a workday to integrate them. I’m absolutely certain it would have taken longer in Java.)

But here’s the part of Ted’s response that I really don’t understand:

So given that an execution engine could easily adopt the model that gives Erlang its reliability, and that using Erlang means a lot more work to get the monitorability and manageability (which is a necessary side-effect requirement of accepting that failure happens), hopefully my reasons for saying that Erlang (or Ruby’s or any other native-implemented language) is a non-starter for me becomes more clear.

Ted, first you state that an execution engine could (emphasis mine) “easily adopt the model that gives Erlang its reliability,” and then you say that it’s “a lot more work” for anyone to write an Erlang application that can be monitored and managed? Aren’t you getting those backwards? It should be obvious that in reality, writing a monitorable Erlang app is not hard at all, whereas building Erlang-level reliability into another VM would be a considerably complicated and time-consuming undertaking.

I see that at least one person on the planet believes I’m the “Father of CORBA.” I can certainly understand why people would think that, but I can definitely say that it’s inaccurate.

In 1989 Hewlett-Packard bought Apollo Computer, where I worked as a diagnostics engineer. Five years prior to that I started my career at Texas Instruments as an electrical engineer working on integrated circuits, and ended up having to write a lot of testing software despite having no software training whatsoever, except for a freshman class in BASIC required for all engineering majors. I found I really liked software, though, so I joined Apollo because the job there was half hardware and half software. By late 1990, though, Hewlett-Packard politics had just about killed the group I worked in, and my manager told me I’d be smart to find myself something else to work on before I was forced to do so. I had been developing a hardware debugger in C++ that involved distributed computing, so I looked around the former Apollo site for any groups using C++ in distributed systems. Turned out Jim Waldo was leading such a group — they were building the first Object Request Broker (ORB). I joined them in January 1991, which was 6 months before the first version of the CORBA spec was published. Ken Arnold was also part of that group. I joined just as they wrapped up the first ORB prototype.

One could argue, therefore, that Jim Waldo is the Father of CORBA, since he led the development of the first ORB. Alternatively, one could argue that Joe Sventek is the Father of CORBA, since he was the editor of the first CORBA specification (Joe was also at HP, though he worked in a different group located in California). However, while they both played important roles in initially defining CORBA, there is to the best of my knowledge no single person who can be called the Father of CORBA. Rather, it was definitely a “design by committee” effort, and I certainly don’t count as one of the fathers since at that time I wasn’t even part of the committee.

BTW, here’s a bit of trivia: for those of you who remember Cliff Stoll‘s “Stalking the Wily Hacker” and “Cuckoo’s Egg” publications, documenting his successful effort to identify who was hacking into computer systems at Lawrence Berkeley Laboratory, Joe Sventek’s account was one that the hacker used to gain access. Joe explained to me that he had been away from the laboratory for quite awhile, residing in the U.K. I believe, so red flags went up when there was activity in his account.

Either it’s suddenly become 2004 again, or somebody didn’t get the memo. It’s hard to choose where to begin with this one — hmm, maybe this little tidbit:

Now here’s the absolute coolest part (at least in my opinion). Armed with the WSDL document, you can use freely available, open source tools to automatically generate stub code to send requests to and receive responses from the SOAP web service in just about any modern programming language of your choosing.

For both the client and the server.

The stub code generates and parses all of the XML. As a developer working in the language of your choice, you are completely abstracted from the sending and receiving of data on the wire.

I guess I was wrong; it’s not 2004, it’s 1994.

When I spoke at QCon San Francisco last November, Stefan Tilkov interviewed me, and the video is now available on InfoQ.com.

We covered a range of topics: CORBA, dynamic languages, REST, distribution, concurrency, Erlang. Stefan asked some great questions, and I hope I gave some worthwhile answers. Thanks again, Stefan.