Wide Finder in Python

October 7th, 2007  |  Published in performance, python  |  3 Comments  |  Bookmark on Pinboard.in

Fredrick Lundh has posted some really nice code, along with detailed explanations, showing Tim’s Wide Finder implemented in various ways in Python, using both threads and processes. I ran his wf-6.py on an 8-core 2.33 GHz Intel Xeon Linux box with 8GB of RAM, and it ran best at 5 processes, clocking in at 0.336 sec. Another process-based approach, wf-5.py, executed best with 8 processes, presumably one per core, in 0.358 sec. The multithreaded approach, wf-4.py, ran best with 5 threads, at 1.402 sec (but also got the same result with 19 threads, go figure). Using the same dataset, I get 11.8 sec from my best Erlang effort, which is obviously considerably slower. I used the bash shell time built-in for all measurements, for consistency.

I’ve been coding in Python for years, but lately I’ve been using it a lot, and can’t get enough of it. It’s so clean, and as Fredrick’s code shows, it’s got some extremely powerful capabilities that are also extremely easy to use. It’s obviously smoking fast as well. Some folks who commented on some of my previous blog entries seem to equate programs written using dynamic languages like Python with “unmaintainable one-off solutions.” If that’s your experience with such languages, blame the programmer, not the language.

Responses

Feed
  1. mmao says:

    October 8th, 2007 at 4:59 am (#)

    Have you tested it with the Ruby version, I heard that Ruby has performance problem, not sure if it’s true.

  2. steve says:

    October 8th, 2007 at 7:50 am (#)

    Yes. On the same 8-core system, the best time I can get with Tim’s original Ruby version is 2.22 seconds. Not surprising, since Ruby is generally known to be slower than Python and Perl, but it makes up for it with useful language features that allow for highly functional yet very compact code. A lot of work is going into Ruby at the moment, so I would guess the performance difference will be reduced or eliminated within the next year or so.

  3. Warren Henning says:

    October 10th, 2007 at 9:21 pm (#)

    Off-topic: what’s the best way to put together an 8-core machine for tinkering with embarassingly parallel algorithms (optimization heuristics, simulations, etc)?