Steve Vinoski's Blog

IEEE Internet Computing will be assembling its editorial calendar for the 2010-2011 timeframe at its upcoming editorial board meeting in March. We solicit proposals to guest edit theme issues. See the IC website for detailed instructions.

The deadline for proposals for this cycle is March 2, 2009.

My first Internet Computing column of 2009 is now available. It’s entitled “New Year’s Integration Resolutions” and it offers advice pertaining to areas of integration and distributed systems where over the years I’ve seen various companies and individuals, including myself at times, repeat the same common mistakes. I give the advice in the form of some suggested new year’s resolutions, hence the title.

Something to note about this column is that it completes 7 years of “Toward Integration.” That’s a long time, trust me. :-) Two years ago I stopped working on middleware and integration as a full-time role, so it doesn’t really feel right anymore for me to still be doling out advice on those topics, and I’ve definitely used up all the residual topics I had left. What this means is that this is the final “Toward Integration” column you’ll ever see.

Whether that’s good news or bad news depends, of course, on whether you liked or disliked “Toward Integration.” :-) I generally received positive feedback over the years on all the columns, but I know there are some out there who disagreed with some of them. That’s perfectly normal, of course. Regardless of which camp you fall in, if you ever sent me considered feedback on any of the columns, I really appreciate it.

But then again, I should be clear that it’s not truly the end; I’ll be starting a new column on a new topic in the magazine’s next issue. I don’t yet have a new title for the column, and in fact it may not be as focused as “Toward Integration” generally was. But I’ll need to come up with a new title soon, since I have only about two weeks left before I have to submit the first issue of the new column to my editor.

A few months back Klacke gave me committer rights for Yaws. I’ve made a few fixes here and there, including adding support for passing “*” as the request URI for OPTIONS, enabling OPTIONS requests to be dispatched to appmods and yapps, and completing a previously-submitted patch for configuring the listen backlog. Klacke has just started putting a test framework into the codebase and build system so that contributors can include tests with any patches or new code they submit, and I’ve contributed to that as well.

The biggest feature I’ve added to date, though, is a new linked-in driver that allows Yaws to use the sendfile system call on Linux, OS X, and FreeBSD. I never wrote a linked-in driver before, so I was happy and fortunate to have an Erlang expert like Klacke providing hints and reviewing my code.

I did some preliminary testing that showed that sendfile definitely improves CPU usage across the board but depending on file size, sometimes does so at the cost of increasing request times. I used my otherwise idle 2-core 2.4GHz Ubuntu 8.04.1 Dell box with 2 GB of RAM, and ran Apache Bench (ab) from another Linux host to simulate 50 concurrent clients downloading a 64k data file a total of 100000 times. I saw that user/system CPU on the web host tended to run around 33%/28% without sendfile, while with sendfile it dropped to 22%/17%. The trade-off was request time, though, where each request for the 64k file averaged 0.928ms with sendfile but 0.567ms without. With larger files, however, sendfile is slightly faster and still has better CPU usage. For example, with a 256k file, sendfile averaged 2.251ms per request with user/system CPU at 8%/16% whereas it was 2.255ms and 16%/27% CPU without sendfile, which makes me wonder if the figures for the 64k file are outliers for some reason. I performed these measurements fairly quickly, so while I believe they’re reasonably accurate, don’t take them as formal results.

On my MacBook Pro laptop running OS X 10.5.6, CPU usage didn’t seem to differ much whether I used sendfile or not, but requests across the board tended to be slightly faster with sendfile.

I ran FreeBSD 7.0.1 in a Parallels VM on my laptop, and there I saw significantly better system CPU usage with sendfile than without, sometimes as much as a 30% improvement. Requests were also noticeably faster with sendfile than without, sometimes by as much as 17%, and again depending on file size, with higher improvements for larger files. User CPU was not a factor. All in all, though, I don’t know how much the fact that I ran all this within a VM affected these numbers.

Given that Yaws is often used for delivering mainly dynamic content, sendfile won’t affect those cases. Still, I think it’s nice to have it available for the times when you do have to deliver file-based content, especially if the files are of the larger variety. Anyway, I committed this support to the Yaws svn repository back around December 21 or so. If you’d like to do your own testing, please feel free — I’d be interested in learning your results. Also, if you have ideas for further tests I might try, please leave a comment to let me know.

Yesterday I posted a SHA-256 Erlang module, but I figured since other SHA algorithms are similar, I might as well finish the job. I grabbed the Secure Hash Standard and went to work.

The resulting new module is named sha2 and it implements SHA-224, SHA-256, SHA-384, and SHA-512.

• Code
• Light documentation
• Tests are included in the module but you get the test vectors here
• It’s open source, BSD license

I hope you find it useful.

Sriram Krishnan was recently lamenting the lack of SHA-256 support in Erlang’s crypto module, so just for fun I wrote a simple sha256 module based on the pseudocode in Wikipedia. The tests use the test data from this C implementation.

[Update: I’ve posted a new module that implements more SHA digest variants and so renders this module obsolete.]