Steve Vinoski's Blog

A few months back Klacke gave me committer rights for Yaws. I’ve made a few fixes here and there, including adding support for passing “*” as the request URI for OPTIONS, enabling OPTIONS requests to be dispatched to appmods and yapps, and completing a previously-submitted patch for configuring the listen backlog. Klacke has just started putting a test framework into the codebase and build system so that contributors can include tests with any patches or new code they submit, and I’ve contributed to that as well.

The biggest feature I’ve added to date, though, is a new linked-in driver that allows Yaws to use the sendfile system call on Linux, OS X, and FreeBSD. I never wrote a linked-in driver before, so I was happy and fortunate to have an Erlang expert like Klacke providing hints and reviewing my code.

I did some preliminary testing that showed that sendfile definitely improves CPU usage across the board but depending on file size, sometimes does so at the cost of increasing request times. I used my otherwise idle 2-core 2.4GHz Ubuntu 8.04.1 Dell box with 2 GB of RAM, and ran Apache Bench (ab) from another Linux host to simulate 50 concurrent clients downloading a 64k data file a total of 100000 times. I saw that user/system CPU on the web host tended to run around 33%/28% without sendfile, while with sendfile it dropped to 22%/17%. The trade-off was request time, though, where each request for the 64k file averaged 0.928ms with sendfile but 0.567ms without. With larger files, however, sendfile is slightly faster and still has better CPU usage. For example, with a 256k file, sendfile averaged 2.251ms per request with user/system CPU at 8%/16% whereas it was 2.255ms and 16%/27% CPU without sendfile, which makes me wonder if the figures for the 64k file are outliers for some reason. I performed these measurements fairly quickly, so while I believe they’re reasonably accurate, don’t take them as formal results.

On my MacBook Pro laptop running OS X 10.5.6, CPU usage didn’t seem to differ much whether I used sendfile or not, but requests across the board tended to be slightly faster with sendfile.

I ran FreeBSD 7.0.1 in a Parallels VM on my laptop, and there I saw significantly better system CPU usage with sendfile than without, sometimes as much as a 30% improvement. Requests were also noticeably faster with sendfile than without, sometimes by as much as 17%, and again depending on file size, with higher improvements for larger files. User CPU was not a factor. All in all, though, I don’t know how much the fact that I ran all this within a VM affected these numbers.

Given that Yaws is often used for delivering mainly dynamic content, sendfile won’t affect those cases. Still, I think it’s nice to have it available for the times when you do have to deliver file-based content, especially if the files are of the larger variety. Anyway, I committed this support to the Yaws svn repository back around December 21 or so. If you’d like to do your own testing, please feel free — I’d be interested in learning your results. Also, if you have ideas for further tests I might try, please leave a comment to let me know.

Nick Gerakines has posted a detailed example of a RESTful web service implemented using Erlang and Yaws. It provides a lot of implementation information that I didn’t have room for in my InfoQ article on this topic.

Sam’s issue isn’t addressed here either, but no matter, the details of Nick’s example are worth examining nonetheless.

Regarding my “RESTful Services with Erlang and Yaws” article on InfoQ.com, Sam Ruby said:

This otherwise excellent article fails my ETag test.

When Sam speaks, I listen, so I’ve given his feedback a lot of thought.

As I wrote in a comment on Sam’s blog, the Erlang/Yaws RESTful services I work on do indeed support conditional GETs, so at least my day-to-day work passes his ETags test. As for the article, there are two ways to think about it:

1. If you focus on the “RESTful Design” portion of the article, then yes, I could have added a “think about where you need to support conditional GETs” item to the “key areas to pay attention to” list.

2. If you focus on the Yaws/Erlang aspect of the article, then keep in mind that dealing with ETags requires dealing with HTTP headers such as If-none-match and the ETag header itself. The article already shows you how to read request headers and write reply headers, though, and how you actually create specific ETag values for use in the headers depends on the particulars of your resources — Leonard Richardson’s and Sam’s excellent book already covers this pretty well.

I intended the focus of the article to be more about item 2 than item 1, so I think not specifically addressing ETags is OK.

One thing I should have included, though, is how to parse POST data. You use the yaws_api:parse_post/1 function for that, passing in an arg record. For typical form data, it’ll give you back a list of key/value pairs over which you can iterate, or from which you can extract expected key/value pairs using yaws_api:postvar/2 (or even proplists:lookup/2 or lists:keysearch/3, if you like). See the documentation at the Yaws website for more details, but all in all, handling POST data in Yaws is fairly trivial.

Today InfoQ.com published a new article I’ve written entitled “RESTful Services with Erlang and Yaws.” Stefan Tilkov recently asked me if I had anything to contribute to InfoQ, and I thought an article on that topic might be interesting, as I hadn’t before seen anything covering REST and Erlang together.

I think it’s one of those articles that could be much, much longer and far more detailed if space (and time) permitted, but hopefully there’s enough there to whet your appetite if you’re considering developing RESTful web services in Erlang. I really can’t say enough good things about using Erlang and Yaws for this purpose — it’s quite a solid platform.

I’ll be giving a talk on the same topic at both JAOO Brisbane and JAOO Sydney at the end of May and beginning of June, respectively.