Steve Vinoski's Blog

The latest “Functional Web” column is now available. It’s entitled “Build Your Next Web Application with Erlang,” (PDF) and my co-author this time was Dave Bryson of BeepBeep fame. Over the past year or two there’s been a sharp increase in development efforts in the Erlang web framework space, as a number of folks, ourselves included, have found that Erlang works really well for server-side web development. In this column Dave and I provide an introduction to a number of Erlang web frameworks and servers. Future columns will dig into some of these in much more detail.

My thanks to Dave for his efforts on this column; he was a sheer pleasure to work with. If Dave ever invites you to co-author something, whether it’s an article or some code, I recommend you take him up on his offer.

P.S. Sorry for the quiet blog lately — it’s just that I have a whole lot of work on my plate these days.

Today Klacke released version 1.82 of Yaws. This is primarily a maintenance release, so nothing major has changed, except that yaws.conf and other files that used to go into etc now go into etc/yaws. For example, the default install prefix is /usr/local so for the installation on my laptop my yaws.conf file has moved from /usr/local/etc/yaws.conf to /usr/local/etc/yaws/yaws.conf. You’ll want to make sure you move any existing conf files you might have to the new directory.

Various other changes and bug fixes are described in the release notes on the main Yaws website. For the previous release Klacke moved the sources from sourceforge to github, so follow that link if you’re looking for source, and you can follow these instructions to build, install, and run. If you should run into any issues or problems, please send a message to the Yaws mailing list or create a new issue on the Yaws issue tracker.

As far as planning for the next release goes, aside from the usual little bug fixes and enhancements and adding tests, Klacke and I noticed Joe Williams‘s recent Erlang Factory presentation that mentions Yaws CPU usage is sometimes higher than expected, so we’ll be looking into that and doing some general profiling and performance testing work.

QCon is always very, very good, but QCon London 2009 last week was the best one yet. Highlights:

• Ola Bini‘s “Emerging Languages in the Enterprise” track on Wednesday had some great talks, especially Rich Hickey’s Clojure talk. Ola’s deep knowledge and love of programming languages made him the perfect host for this track, and he put together a brilliant lineup.
• Speaking of Rich, I consider myself very lucky to have gotten to meet and spend a fair amount of time with him. He’s very bright, talented, knowledgeable, and experienced, and both of his talks were outstanding.
• I introduced Rich to Joe Armstrong at the conference party Wednesday evening and they spent the next few hours talking at length about functional programming, their respective languages, VM implementation issues, concurrency issues, etc. Ola jumped in as well. They also continued the conversation the next day. I just sat back, listened, and learned.
• Getting to spend time again with Joe was excellent. He always has incredibly useful analyses and opinions to express, and in general is fun to be around and easy to learn from.
• I also finally got to meet Ulf Wiger, Erlang developer extraordinaire, in person. He’s a laid back guy, quite well-informed and a deep thinker who can cover a wide variety of topics in amazingly useful detail. His talk on multicore programming in Erlang covered cutting edge Erlang development and presented some very difficult concurrency issues.
• Ulf’s talk, as well as Rich’s second talk, which was on persistent data structures and managed references, were part of Francesco Cesarini‘s “Functional and Concurrent Programming Languages Applied” track on Thursday. I met Francesco, who like Ulf is one of the world’s top Erlang developers, at QCon London last year. He assembled a great track for this conference, with Rich’s and Ulf’s back-to-back talks being way more than enough to sober up any developer who thinks that multicore is not an issue and that today’s methods for dealing with concurrency will continue to work just fine. Best of luck with that!
• Sir Tony Hoare‘s talk about the null reference being his “billion dollar mistake” was great because of all the detail he recounted from some of the early days of computing. He was both informative and entertaining. I was also impressed with Ulf during this talk, whom Professor Sir Hoare invited to come up to the front and present what turned out to be a pretty convincing argument in favor of the null reference.
• Paul Downey‘s talk on the downsides of standardization was by far the most humorous talk I heard, perfect to close out the track, but it also presented a number of hard-won useful lessons about the perils of standardization efforts.

As with all QCon conferences, there were a bunch of interesting tracks running in parallel, and unfortunately I still haven’t figured out how to be in multiple places at once. I had to miss Michael Nygard‘s talk, for example, because my own talk got moved to the same time slot as his.

My talk (PDF) covered the history of RPC, why it got to be the way it was, and why the forces that created it really aren’t all that viable anymore.

The final conference panel was by far the most inventive panel I’ve ever been on. Modeled after the British game show “It’s a Bullseye!” and hosted by none other than Jim Webber, it had contestants from the audience throwing darts to become eligible for a prize. Once a contestant became eligible, Jim would ask the panel — Michael Nygard, Ian Robinson, Martin Fowler, and me — to answer a question submitted by conference attendees either earlier during the conference or live via Twitter. Based on our answers, audience members held up either a green card if they liked the answers or a red one if they didn’t, and if the majority was green, the contestant would win a book. The questions were hard! We had only two minutes each to answer, which for some questions seemed like an eternity but for most was way too short. Anyway, it was great fun, and given how many there were in the audience after three grueling conference days and how much they seemed to be enjoying themselves, it worked very, very well.

If you have any interest at all in leading edge software and computing topics being presented by the world’s most knowledgeable speakers in a fun atmosphere, go to QCon. I guarantee you won’t be disappointed.

Erlang’s heart feature provides a heartbeat-based monitoring capability for Erlang runtime systems, with the ability to restart a runtime system if it fails. It works reasonably well, but one issue with it is that if an error occurs such that it causes repeated immediate runtime crashes, heart will happily keep restarting the runtime over and over again, ad infinitum.

For yaws 1.80, released a few days ago on Feb. 12, I added a check to the heart setup in the yaws startup script to prevent endless restarts. I thought I’d share it here because it’s useful for Erlang systems in general and is in no way specific to yaws. It works by passing startup information from one incarnation to the next, checking that information to detect multiple restarts within a given time period. We track both the startup time and the restart count, and if we detect 5 restarts within a 60 second period, we stop completely. This is not to say that yaws is in dire need of this capability — it’s extremely stable in general and 1.80 in particular is a very good release — but I added it mainly because other Erlang apps sharing the same runtime instance as yaws may not enjoy that same high level of stability, especially while they’re still under development.

The command heart runs to start a new instance is set in the HEART_COMMAND environment variable. For yaws, it’s set like this (I’ve split this over multiple lines for clarity, but it’s just one line in the actual script):

HEART_COMMAND="${ENV_PGM} \
  HEART=true \
  YAWS_HEART_RESTARTS=$restarts \
  YAWS_HEART_START=$starttime \
  $program "${1+"$@"}

where

• ${ENV_PGM} is /usr/bin/env, which allows us to set environment variables for the execution of a given command.
• HEART is an environment variable that we use to indicate the command was launched by heart.
• YAWS_HEART_RESTARTS is an environment variable that we use to track the number of restarts already seen. The yaws script initially sets this to 1 and increments it for each heart restart.
• YAWS_HEART_START is an environment variable that we use to track the time of the current round of restarts. This is tracked as UNIX time, obtained by the script via the “date -u +%s” command.
• $program is the yaws script itself, i.e., $0.
• ${1+"$@"} is a specific shell construct that passes all the original arguments of the script unchanged along to $program.

The yaws script looks for HEART set to true, indicating that it was launched by heart. For that case, it then checks YAWS_HEART_RESTARTS and YAWS_HEART_START to see how many restarts we’ve seen since the start time. We get the current UNIX time and subtract the YAWS_HEART_START time; if it’s less than or equal to 60 seconds and the restart count is 5, we exit completely without restarting the Erlang runtime. Otherwise we restart, first adjusting these environment variables. If the restart count is less than 5 within the 60 second window, we increment the restart count and set the new value into YAWS_HEART_RESTARTS but keep the same YAWS_HEART_START time. But if the current time is more than 60 seconds past the start time, we reset YAWS_HEART_RESTARTS to 1 and set a new start time for YAWS_HEART_START. Look at the yaws script to see the details of this logic — scroll down to the part starting with if [ "$HEART" = true ].

Note that this approach is much like the way Erlang receive loops generally track state, by recursively passing state information to themselves.

A few months back Klacke gave me committer rights for Yaws. I’ve made a few fixes here and there, including adding support for passing “*” as the request URI for OPTIONS, enabling OPTIONS requests to be dispatched to appmods and yapps, and completing a previously-submitted patch for configuring the listen backlog. Klacke has just started putting a test framework into the codebase and build system so that contributors can include tests with any patches or new code they submit, and I’ve contributed to that as well.

The biggest feature I’ve added to date, though, is a new linked-in driver that allows Yaws to use the sendfile system call on Linux, OS X, and FreeBSD. I never wrote a linked-in driver before, so I was happy and fortunate to have an Erlang expert like Klacke providing hints and reviewing my code.

I did some preliminary testing that showed that sendfile definitely improves CPU usage across the board but depending on file size, sometimes does so at the cost of increasing request times. I used my otherwise idle 2-core 2.4GHz Ubuntu 8.04.1 Dell box with 2 GB of RAM, and ran Apache Bench (ab) from another Linux host to simulate 50 concurrent clients downloading a 64k data file a total of 100000 times. I saw that user/system CPU on the web host tended to run around 33%/28% without sendfile, while with sendfile it dropped to 22%/17%. The trade-off was request time, though, where each request for the 64k file averaged 0.928ms with sendfile but 0.567ms without. With larger files, however, sendfile is slightly faster and still has better CPU usage. For example, with a 256k file, sendfile averaged 2.251ms per request with user/system CPU at 8%/16% whereas it was 2.255ms and 16%/27% CPU without sendfile, which makes me wonder if the figures for the 64k file are outliers for some reason. I performed these measurements fairly quickly, so while I believe they’re reasonably accurate, don’t take them as formal results.

On my MacBook Pro laptop running OS X 10.5.6, CPU usage didn’t seem to differ much whether I used sendfile or not, but requests across the board tended to be slightly faster with sendfile.

I ran FreeBSD 7.0.1 in a Parallels VM on my laptop, and there I saw significantly better system CPU usage with sendfile than without, sometimes as much as a 30% improvement. Requests were also noticeably faster with sendfile than without, sometimes by as much as 17%, and again depending on file size, with higher improvements for larger files. User CPU was not a factor. All in all, though, I don’t know how much the fact that I ran all this within a VM affected these numbers.

Given that Yaws is often used for delivering mainly dynamic content, sendfile won’t affect those cases. Still, I think it’s nice to have it available for the times when you do have to deliver file-based content, especially if the files are of the larger variety. Anyway, I committed this support to the Yaws svn repository back around December 21 or so. If you’d like to do your own testing, please feel free — I’d be interested in learning your results. Also, if you have ideas for further tests I might try, please leave a comment to let me know.