Steve Vinoski's Blog

Since a number of people seem to be experimenting with adding Erlang-style concurrency to other languages, Ulf Wiger has written a nice explanation of what Erlang-style concurrency actually is. Definitely informative.

On a related note, I chuckled when I saw this posting from Robert Virding, who helped create Erlang, in the erlang-questions list about a month ago:

After reading the blogs about how good Erlang’s concurrency model is and how we just just made a super implementation of it in XXX I have been led to formulate Virding’s First Rule of Programming:

Any sufficiently complicated concurrent program in another language contains an ad hoc informally-specified bug-ridden slow implementation of half of Erlang.

This is, of course, a mild travesty of Greenspun but I think it is fundamental enough to be my first rule, not the tenth.

I can understand where he’s coming from. When I see the kind of blog postings he’s referring to, I always wonder why they don’t just use Erlang itself instead of trying to reinvent it in another system whose design trade-offs are unlikely to be able to support it. Well, I guess it would make sense in the case where they’re specifically studying various implementations of concurrency, but other than that, I don’t get it.

On another related note, I found the comment below in another blog. It refers specifically to my own use of Erlang:

it’s [sic] be interesting to see how Steve mix [sic] Erlang and HTTP, since Erlang is asynchronous in nature and does not have any “resource” concept and HTTP’s sweet spots are just the opposite

That’s easy. If you want to experience HTTP as implemented in Erlang, just go to the Yaws website, which of course is powered by Yaws itself. As Ulf’s blog posting explains, Erlang message passing is asynchronous, but that really has nothing to do with Erlang’s ability to support protocols like HTTP. For example, when I was learning Erlang, given my CORBA background I wrote a subset of IIOP, and it was really quite easy. Furthermore, Erlang doesn’t need a “resource” concept to support HTTP. A resource in that context is, after all, ultimately just a chunk of code that processes incoming requests, and of course Erlang can do that.

If you want to know more, go pick up Joe Armstrong’s Erlang book — it’s quite excellent.

I see my old pal Bill is spreading dynamic language FUD again. I’m the “dynamic language evangelist colleague” to whom he sent the email he mentions.

First, I find it funny that he calls me a “dynamic language evangelist.” I’m really more of a multi-language evangelist, or as Ola Bini recently put it, a polyglot programmer (in fact much of Ola’s recent thread on this topic matches up well with what I said on the closing JAOO 2006 panel). This year marks my 20th year of using C++, for example, and I still use it every day, so that probably automatically disqualifies me from membership in any dynamic language clubs. But this year also marks my 20th year of using both Perl and emacs-lisp, so maybe there’s hope for me yet.

Bill gets irrationally upset when I tell him I write in five different languages every day. They are, in no particular order, Erlang, C++, Python, Perl, and emacs-lisp. I really like Ruby but unfortunately don’t get much chance to use it at the moment. I even have to delve into SQL every now and again. Apparently, this notion drives Bill crazy not only because I don’t use Java (though I’ve written my share of it in the past), but also because it means I can’t use an IDE, since no IDE can handle all of the above. Apparently, if you can’t use an IDE to automatically rename your methods for you every 10 minutes or so, you’re not a real programmer.

Bill’s posting, like his original email to me that sparked it, is yet another “we’ve got to put these dynamic language folks in their place” postings (read to the bottom of that link for the context). He basically says so himself. His posting is full of the usual misunderstandings, rumors, poor guesses, and misinformation.

Issue 1: Dynamic Language XXXX didn’t do well against Java in XXXX benchmark

According to Bill, glaring differences between languages in how they perform on benchmarks must be recognized. Yes, that’s true — if all languages performed equally for all tasks, then we wouldn’t need multiple languages. Unfortunately, monolingual developers like Bill don’t view it that way; instead, they view benchmarks as some sort of strange competition where you can try to build up your self-esteem by gloating over the benchmarks for which your one language looks good.

I asked Bill why anyone would choose a language not known for its number crunching capabilities and actually try to use it for real code requiring number-crunching, like calculations involving the Mandelbrot set. He didn’t answer.

In reality, we need multiple languages because there are so many trade-offs to make. One way to address the need to make trade-offs is to choose a general-purpose language like Java and hope that it’s good enough for whatever you have to work on. Another way, and the way I prefer, is to be adept at multiple languages so that you can use the right tool for the job and handle unforeseen requirements. When a monolingual developer runs into a task for which their one language is ill-suited, they typically either bend the problem to fit their language, or they just avoid solving it altogether. Hammer, meet nail. Either way, they’re out of luck, or actually, their customer is out of luck. We polyglots don’t have that problem.

Issue 2: Ruby VM cannot support kernel threads

This issue is already addressed in comments in Bill’s blog, where several people mention all the current work on Ruby internals. But I would add that someone who’s writing authoritatively about Ruby should at least do his homework and already know that this “burning” issue is actually not an issue. Bill mentioned both in his email to me and on his blog that even his beloved Java still needs work, citing the need for “typesafe closures to complement annotations,” “a standard, non-code generating way of adding behavior to annotations,” and “a structural syntax to make initialization easier.” I assume therefore that the fact that Ruby also needs work will not be held against it.

Dynamic languages like Ruby, Python, etc… are not typesafe/statically typed

This is where Bill gets into the IDE religion. When I told him I use five different languages every day and said that I don’t need an IDE, he got mad and called me a name. It wasn’t even a particularly imaginative name, so perhaps he could get his IDE to automatically rename that name for him to turn it into a better insult. I asked if he could recommend a good IDE that could adequately handle all those languages for me, but he didn’t answer.

I use emacs. I’ve used it since 1985. I used Eclipse over the past few years for my Java work, and I really had strong hopes for it for other languages, but it’s still too Java-centric, and my current work doesn’t involve Java anyway. I am highly productive with emacs, and I see no reason whatsoever to change.

Bill asks why we can’t have a dynamically statically typed language. We can; see for example Haskell, and go look at all of Erik Meijer’s work.

Lack of type safety doesn’t scale well to large teams

Prove it. This is pure conjecture, nothing more, nothing less. Scaling development has everything to do with the team members and how they work and work together. Saying that “my buddy Jason was on a team that had to put type names in their Python function names” is certainly not even close to being solid evidence of your claim, Bill. For that particular example, BTW, why not describe the typing expectations for a function in its docstring, thus making it easily accessible to actual human programmers from their interactive Python prompts? That’s way better than applying some weird brittle Hungarian notation convention.

Lack of static typing does not allow for reliable refactoring in modern IDEs

The contrived Ruby example that Bill uses to “prove” this is, well, contrived. Why would anyone write code like that or suddenly get the urge to rename init to init2? I’m no Ruby expert, but I’d probably rename the method and then stick a method_missing in there to catch any call instances I might have missed with my editor. I’m sure a stronger Ruby developer can suggest a better way. Either way, it’s really not a big deal, Bill. I’m sure I could get it done well within the timespan you waste on fantasy football every day. ;-)

In 20 years of writing some pretty large C++ systems, for example, I don’t really recall ever having to go and rename a whole bunch of functions across a codebase. I recently had to move some classes from one C++ namespace to another, but a few lines of elisp automatically fixed 3000+ files for me, all without breaking the build (and yes I’m aware of namespace aliasing, thanks).

I asked him if these magical modern IDEs that raise productivity and eliminate common errors also eliminate defects caused by missing or misunderstood requirements, missed use cases, or just plain ol’ bad programming. Again, no answer. The reason I asked this is that those are the bad bugs; syntactical errors are really the least of your worries. Even if the IDE spits out common idiom/pattern skeletons for you, it’s still quite possible to screw up the code logic, and neither the IDE nor the compiler is going to prevent that.

I asked Bill if he’s ever considered that Java IDEs are the way they are due to issues with the Java language itself that must be compensated for. Consider its verbosity, for example, which is why IDEs spit out those skeletons. I can’t remember if he called me a name for asking that, or if he just didn’t answer. Either way, I got no response.

Bill asks if we can’t just learn how to type, but isn’t it the other way around? Shouldn’t he be asking why he needs the IDE to type for him? Odd.

Considering how old Java is, it’s obvious that it’s taken quite a bit of time to get these IDEs to where they are today. I asked Bill if everyone should have to wait a long time, on the order of 10-15 years, before an IDE truly becomes valuable for a given language. No answer. Personally, I’d rather stick to my emacs and UNIX tools, with their infinite applicability and flexibility that can be used for development in pretty much any language, than wait around for someone else to give me a far less functional IDE that barely addresses only one language. But then again, if one language is all you got, then I guess you have no choice.

The rest of the myths Bill cites just circle back to the same weak IDE arguments again, so I’ll skip those, save for this one:

Software Engineers should use the best language for the job

Yes, this is true, unless you’re an elitist.

Like Bill, I’ve spent much of my career writing middleware. However, unlike Bill, I’m not happy to sit around thinking that C++ or Java is the best we can do. I wish I would have known about Erlang 10 years ago, for example, so I could have saved myself countless development hours and headaches. In hindsight, I wasted way too much time trying to build in C++ and Java what Erlang basically gives you for free. I have only my own ignorance to blame, but now I know better. Thankfully, I now work in a place that lets me apply the right language for the right job, and our productivity is significantly higher than any place else I’ve ever worked.

I told Bill that if he wanted to do benchmarking, he should go benchmark the Yaws web server, written in Erlang, against Apache or some Java web server. Needless to say, that would be far more realistic to a lot of developers than Mandelbrot calculations. His response was that he didn’t need to, because Apache would win. That kind of poor assumption is much like the mindset that led me to poorly reinvent half of Erlang in my years of middleware work. These days, though, if I saw this graph I’d be highly intrigued — and in fact, I was when I first saw it — so much so that I’d want to run my own tests to see if those claims were accurate. My own tests were smaller scale than what that graph shows, but the results were basically the same. I’ve rerun my own tests and some others as well, and I get the same results every time. So Bill, rather than guess and make assumptions, why not try it for yourself? Or are you afraid to find out that dynamic languages actually are quite useful?

The things I write about in this blog are based on significant hands-on experience. Unlike many developers, I’ve heavily used approaches and techniques from both sides of the fence, and based on all my experiences, I’d rather use dynamic languages than static ones. I have more success with them, and the systems I use them for turn out better. That’s not wishful thinking or fabrication, that’s simply the cold, hard, factual truth. I wonder, then, why Bill and others like him insist of trying to “cure” me and others like me of our successes?

If you don’t like dynamic languages, then don’t use them! Leave us alone and go back to your monolingual world, and maybe rename your methods again. You can even have my IDE, since I’m not using it, which should double your productivity yet again, right?

You’ve probably heard a lot by now about Erlang’s concurrency capabilities, and in fact that’s what I covered in my Sept./Oct. Internet Computing “Toward Integration” column, Concurrency with Erlang. But concurrency is only part of the story — Erlang also provides outstanding support for building highly-reliable software. My latest column, Reliability with Erlang, first describes some of the problems that highly-reliable systems face, and then explains some of Erlang’s core primitives that provide a solid foundation for reliable systems. It’s available in HTML and PDF formats.

BTW, I can’t recommend enough that you pick up a copy of Joe Armstrong’s Programming Erlang. It’s a truly excellent book that has deeply and positively affected the way I think about designing, building, and implementing software systems. As I mentioned in my columns, my only disappointment with Erlang is that I didn’t discover it 10 years ago when it was first open-sourced, as it could have saved me a ton of time and trouble in my middleware development efforts over the years.

OK, so you could say that I’m a bit obsessed with Tim Bray’s Wide Finder project. Just a little. I mean, I started this blog just a few weeks ago and so far almost every posting has been about it:

• Tim Bray and Erlang
• More File Processing with Erlang
• One More Erlang Wide Finder
• OK, Just One More WF

There’s also my Sept./Oct. Internet Computing column, Concurrency with Erlang, and sometime in early November, my next column, entitled Reliability with Erlang, will be published. Neither column is connected at all to the Wide Finder, but they just further reveal my current obsession with Erlang.

In my previous post I described my fastest solution up to that point, but here’s an even faster one: tbray16.erl, which is identical in every way to tbray15.erl from my previous post except that it uses wfbm4.erl, which provides all the performance gains. This version of Boyer-Moore searching includes two simple tweaks:

• Uses hard-coded constants for string lengths, since the strings are fixed, rather than constantly recalculating them with length/1.
• Fixes a nagging problem with my Boyer-Moore implementation where it wasn’t handling repeated characters in the fixed pattern very well. What I did in the previous version was choose the lesser of the two shifts if both characters appeared in the pattern, which worked but isn’t technically correct, and it also meant two dict lookups to get the shift values rather than just one. Now, I do the right thing: just keep track of the number of comparisons and subtract that from the shift value, use that if the result is positive and non-zero, otherwise just shift by 1.

This version shaves another whole second off the previous version for about a 25% speedup. The fastest I’ve seen it run on my 8-core Linux box is:

real    0m3.107s
user    0m16.243s
sys     0m2.134s

Meanwhile, Anders Nygren has been exploring eliminating using the dict altogether for the shift value lookup, but nothing I’ve tried there has been an improvement. But thanks to Anders for prompting me to properly fix that Boyer-Moore code and at least eliminate one of the dict lookups.

When writing my previous post I silently hoped I was finished contributing more Erlang solutions to Tim Bray’s Wide Finder project. Tim already told me my code was running really well on his T5120, and yet in my last post, I nearly doubled the speed of that code, so I figured I was in good shape. But then Caoyuan Deng came up with something faster. He asked me to run it on my 8-core Linux box, and sure enough, it was fast, but didn’t seem to be using the CPU that well.

So, I thought some more about the problem. Last time I said I was using Boyer-Moore searching, but only sort of. This is because I was using Erlang function argument pattern matching, which proceeds forward, not backward as Boyer-Moore does. I couldn’t help but think that I could get more speed by doing that right.

I was also concerned about the speed of reading the input file. Reading it in chunks seems like the obvious thing to do for such a large file (Tim’s sample dataset is 236140827 bytes). It turns out that reading in chunks can cumulatively take over a second using klacke’s bfile module, but it takes only about a third of a second to read the whole thing into memory in one shot. By my measurements the bfile module is noticeably faster at doing this than the Erlang file:read_file/1. Even my Macbook Pro can read the whole dataset without significant trouble, so I imagine the T5120 can do it with ease.

So, I changed tactics:

• Read the whole dataset into an Erlang binary in one shot, then break it into chunks based on the number of schedulers that the Erlang system is using.
• Stop breaking chunks into smaller blocks at newline boundaries. This took too much time. Instead, just grab a block, search from the end to find the final newline, and then process it for pattern matches.
• Change the search module to do something much closer to Boyer-Moore regarding backwards searching, streamline the code that matches the variable portion of the search pattern, and be smarter about skipping ahead on failed matches.
• Balance the parallelized collection of match data by multiple independent processes against the creation of many small dictionaries that later require merging.

This new version reads the whole dataset, takes the first chunk, finds the final newline, then kicks off one process to collect matches and a separate process to find the matches. It then moves immediately onto the next block, doing the same thing again. What that means is the main process spends its time finding newlines and launching processes while other processes look for matches and collect them. At the end, the main process collects the collections, merges them, and prints out the top ten.

On my 8-core 2.33GHz Linux box with 8 GB of RAM:

$ time erl -smp -noshell -run tbray15 main o1000k.ap
2959: 2006/09/29/Dynamic-IDE
2059: 2006/07/28/Open-Data
1636: 2006/10/02/Cedric-on-Refactoring
1060: 2006/03/30/Teacup
942: 2006/01/31/Data-Protection
842: 2006/10/04/JIS-Reg-FD
838: 2006/10/06/On-Comments
817: 2006/10/02/Size-Matters
682: 2003/09/18/NXML
630: 2003/06/24/IntelligentSearch

real    0m4.124s
user    0m25.124s
sys     0m1.916s

At 4.124s, this is significantly faster than the 6.663s I saw with my previous version. The user time is 6x the elapsed time, so we’re using the cores well. What’s more, if you change the block size, which indirectly controls the number of Erlang processes that run, you can clearly see a pretty much linear speedup as more cores get used. Below is the output from a loop where the block size starts at the file size and then divides by two on each iteration (I’ve edited the output to make it more compact by flattening the time output into three columns):

$ ((x=236140827)) ; while ((x>32768))
do echo $x
    time erl -smp -noshell -run tbray15 main o1000k.ap $x >/dev/null
    ((x/=2))
done

236140827: 0m38.072s, 0m52.159s, 0m3.984s
118070413: 0m18.294s, 0m37.922s, 0m4.571s
59035206:  0m11.374s, 0m36.694s, 0m9.098s
29517603:  0m4.598s,  0m27.825s, 0m2.180s
14758801:  0m4.225s,  0m26.237s, 0m2.134s
7379400:   0m4.181s,  0m25.779s, 0m1.873s
3689700:   0m4.124s,  0m25.124s, 0m1.916s
1844850:   0m4.149s,  0m24.931s, 0m1.969s
922425:    0m4.132s,  0m24.894s, 0m1.822s
461212:    0m4.170s,  0m24.588s, 0m2.026s
230606:    0m4.185s,  0m24.548s, 0m2.035s
115303:    0m4.215s,  0m24.755s, 0m2.025s
57651:     0m4.317s,  0m25.199s, 0m1.985s

The elapsed time between the top four entries show the multiple cores kicking in, essentially doubling the performance each time. Once we hit the 4 second range, performance gains are small but steady roughly down to a block size of 922425, but then they start to creep up again. My guess is that this is because smaller blocks mean more Erlang dict instances being created to capture matches, and all those dictionaries then have to be merged to collect the final results. In the middle, where performance is best, the user time is roughly 6x the elapsed time as I already mentioned, which means that if Tim runs this on his T5120, he should see excellent performance there as well.

Feel free to grab the files tbray15.erl and wfbm3.erl if you want to try it out for yourself.