New Column on Webmachine

March 15th, 2010  |  Published in column, erlang, functional programming, HTTP, REST, web  |  Bookmark on Pinboard.in

“The Functional Web” column is finally back, this time with a column about Webmachine co-authored with Justin Sheehy. The column title is Developing RESTful Web Services with Webmachine, and you can follow that link to retrieve the PDF.

Webmachine is a highly innovative web application framework, and it can teach you a great deal about the specifics of HTTP and the details of REST. It’s also written in Erlang, which continues to be my favorite programming language of all time because of its incredible practicality, utility, and elegance.

My column hiatus was due to extreme startup workload, which for better or worse is showing no sign of letting up anytime soon. But it’s nice to get the column back on track for the March/April Internet Computing issue, and one of my goals is to avoid missing any more issues this year. Many thanks to Justin for his contribution to this issue of the column.

Yaws 1.85 Released

October 19th, 2009  |  Published in HTTP, web, yaws  |  Bookmark on Pinboard.in

Today Klacke announced Yaws 1.85, mainly a bugfix release. You can find the list of changes and fixes at that link, but one addition in this release I wanted to point out was our new streamcontent_from_pid feature, which allows your server application code to temporarily take over the client connection socket from Yaws, thus allowing you to feed data directly to the socket without first passing it back through Yaws. Could be just the ticket for long-polling (Comet) applications, for example.

Prior to this release, the closest feature Yaws provided for this sort of operation was the streamcontent capability, which is still very useful in that it allows you to have an Erlang process deliver data back into Yaws, which in turn sends it in HTTP chunked transfer mode back to the client. For large file resources like video files or install tarballs, or for data sources where content arrives at your server in batches from a separate back-end source, or generally for resources whose sizes are not known up front, streamcontent is perfect because it lets you transfer the data back into Yaws in chunks, at your leisure and without having to copy all the data at once. Still, though, the data has to be sent back through Yaws, which converts it to HTTP chunks and writes it to the socket, plus in this case your only choice is chunked transfer.

With streamcontent_from_pid, you reply to the out/1 upcall from Yaws with the HTTP reply headers and with the following special return tuple:

{streamcontent_from_pid, MimeType, StreamPid}

This tells Yaws that you wish to have process StreamPid take over the client socket in order to send data of media type MimeType directly back to the client. Yaws uses MimeType to set the HTTP Content-Type header, and then after sending that and all the other HTTP headers back to the client, it turns control of the client socket over to StreamPid. The code running within StreamPid must handle the following messages from Yaws:

  • {ok, YawsPid} tells StreamPid that it can proceed with using the socket. The socket is present in the original Arg variable passed to your out/1 function and can be retrieved via Arg#arg.clisock.
  • {discard, YawsPid} tells StreamPid that it shouldn’t send any data on the socket, for example because the client request was an HTTP HEAD request and so there is no response body.

To send data, your code can choose to send chunked data or non-chunked data. To send the latter, first make sure you set the HTTP Content-Length header in your initial reply to Yaws, and then once Yaws calls back to your StreamPid process, just call:

yaws_api:stream_process_deliver(Socket, IoList)

or for chunked data:

yaws_api:stream_process_deliver_chunk(Socket, IoList)

where for both cases Socket is the client socket from the Arg and IoList is an iolist containing the data to be sent. The first case just calls gen_tcp:send and is there primarily so we can maybe someday add SSL socket support for this feature. For the second case Yaws will format the data for you for chunked transfer. Unless a Content-Length header is set, Yaws will assume you want chunked transfer and will set the Transfer-Encoding header appropriately. If you’re sending chunked data, make sure you send your final chunk using the following function:

yaws_api:stream_process_deliver_final_chunk(Socket, IoList)

so that Yaws knows to send the termination chunk to inform the client of the end of the transfer.

You can continue to call these functions from your StreamPid as frequently as you need to in order to deliver your data to the client. Meanwhile, the Yaws process that handed you the socket will just sit back and wait for you (non-blocking, of course). When you’re completely finished sending, just call:

yaws_api:stream_process_end(Socket, YawsPid)

to end the transmission and give control of the socket back to Yaws. At that point, your StreamPid can exit if it wishes.

If you try out this feature, be sure to send feedback either to me or to the Yaws mailing list.

New Column: Build Your Next Web App with Erlang

July 25th, 2009  |  Published in column, erlang, functional programming, web  |  Bookmark on Pinboard.in

The latest “Functional Web” column is now available. It’s entitled “Build Your Next Web Application with Erlang,” (PDF) and my co-author this time was Dave Bryson of BeepBeep fame. Over the past year or two there’s been a sharp increase in development efforts in the Erlang web framework space, as a number of folks, ourselves included, have found that Erlang works really well for server-side web development. In this column Dave and I provide an introduction to a number of Erlang web frameworks and servers. Future columns will dig into some of these in much more detail.

My thanks to Dave for his efforts on this column; he was a sheer pleasure to work with. If Dave ever invites you to co-author something, whether it’s an article or some code, I recommend you take him up on his offer.

P.S. Sorry for the quiet blog lately — it’s just that I have a whole lot of work on my plate these days.

Scala and Lift Podcast

June 18th, 2009  |  Published in column, functional programming, podcast, web  |  Bookmark on Pinboard.in

Scala and Lift — Functional Recipes for the Web” (PDF) co-authored with Debasish Ghosh, is now also available as a podcast.

Yaws 1.82

May 29th, 2009  |  Published in erlang, web, yaws  |  Bookmark on Pinboard.in

Today Klacke released version 1.82 of Yaws. This is primarily a maintenance release, so nothing major has changed, except that yaws.conf and other files that used to go into etc now go into etc/yaws. For example, the default install prefix is /usr/local so for the installation on my laptop my yaws.conf file has moved from /usr/local/etc/yaws.conf to /usr/local/etc/yaws/yaws.conf. You’ll want to make sure you move any existing conf files you might have to the new directory.

Various other changes and bug fixes are described in the release notes on the main Yaws website. For the previous release Klacke moved the sources from sourceforge to github, so follow that link if you’re looking for source, and you can follow these instructions to build, install, and run. If you should run into any issues or problems, please send a message to the Yaws mailing list or create a new issue on the Yaws issue tracker.

As far as planning for the next release goes, aside from the usual little bug fixes and enhancements and adding tests, Klacke and I noticed Joe Williams‘s recent Erlang Factory presentation that mentions Yaws CPU usage is sometimes higher than expected, so we’ll be looking into that and doing some general profiling and performance testing work.