Comments on: Sendfile for Yaws

By: steve

steve — Tue, 06 Jan 2009 06:04:41 +0000

@Andy: keep in mind that the response time differences are all in all pretty small, and that the win in CPU that sendfile affords is potentially very worthwhile if you’re serving a mixture of dynamic and static content.

By: Andy

Andy — Tue, 06 Jan 2009 05:47:43 +0000

I’m very surprised by the increase in response time when Sendfile is enabled. I was always under the impression that Sendfile greatly improves the performance of web servers.

What about other web servers like Apache and lighttp? Do their performances improve when Sendfile is enabled?? Maybe comparing Yaws with those servers will shed light on this.

By: steve

steve — Mon, 05 Jan 2009 16:15:48 +0000

@Nicholas: sendfile itself is a blocking call where the caller waits for completion, so there’s not much we can do about that. But since Erlang uses async I/O everywhere, the linked-in driver is set up to be called back by the Erlang VM when its socket is ready for writing. When the socket is writeable, the driver calls sendfile and tries to send as much as it can, but it eventually gets EAGAIN or EWOULDBLOCK when writing to the socket would block. At that time it stores away how much data it successfully sent and returns control to the VM, waiting to be called back next time its socket is writeable. This ensures we don’t block the VM, which is very important.

Performing the ab test serially, i.e. without simulated concurrent clients, for fetching the 64k file still shows sendfile to be slightly slower, averaging 1.506ms per request with sendfile vs. 1.445ms without. User/system CPU for this case is roughly the same, about 11%/11% with sendfile and 12%/12% without. For the 256k file, I see sendfile taking 3.283ms per request with CPU of 8%/12%, and non-sendfile taking 3.227ms per request with CPU at 11%/17%. For a 1MB file, sendfile takes 10.154ms at 4%/13% while non-sendfile takes 10.103ms at 10%/18%. There’s a 60 us difference in all those cases. And finally, if we go up to a 64MB file, I see sendfile at 593.2ms with CPU 1%/5% whereas non-sendfile takes 596.1ms with CPU 7%/12%; notice that sendfile’s request time is about 3ms faster for this case. So in this serialized test, overall sendfile always wins in CPU but is just slightly slower except for very large files.

By: Nicholas Dronen

Nicholas Dronen — Mon, 05 Jan 2009 13:47:06 +0000

My hunch is that the larger request times for smaller files is caused by sendfile blocking the calling Erlang process (and the operating system thread in which it is executing), preventing other processes from running. This is just a hunch though. A way to find out whether this is happening is to run Apache Bench serially both with and without sendfile on the server; if my hunch is correct, I think you’ll get lower average response times for the 64k files with serially-submitted requests.