Benchmarks everywhere

It seems that my benchmarking "initiative" caused some effects, and many vendors/frameworks are doing the test with the same tool.

AIDA/Web vs Seaside:
http://www.aidaweb.si/benchmarks/wapt-swazoo-20.html

Seaside on VisualWorks, Squeak, vs Ruby on Rails :
http://www.cincomsmalltalk.com/blog/blogView?showComments=true&printTitle=More_Seaside_Testing&entry=3372921925

My own tests (which will be replayed):
http://dolphinseaside.blogspot.com/2007/11/some-benchmarks.html

And the initial, and more exhaustive, Seaside on Gemstone benchmark:
http://gemstonesoup.wordpress.com/2007/10/19/scaling-seaside-with-gemstones/

Considering all of us are using the same tool I've choosen (WAPT), we should define a clear and unique test suite, upload it to a public place, and run it against the different implementations. The results published by AIDA and Cincom have a significant difference between them.

Anyway is important that we put "benchmarking" over the table.

See you in Smalltalks 2007.

Some benchmarks

Yesterday I ran some benchmark tests against Seaside on Dolphin, Squeak and VW.

The test is basically a brute force conection initiation, against the basic Seaside Counter.

I won't do any analysis or conclusion yet, just some observations, however I'm publishing the results for you our own review.

All the test were ran by separate in the same machine, an Athlon64 X2 3600+, 1GB RAM, Windows XP Pro.
Simulating 20 concurrent users starting connections against the server, one after the other, to see how much load it can handle.

Dolphin (Swazoo 1.1.4):
http://emaringolo.users.dolphinmap.net/seaside/reports/20071114/SeasideDolphin.html

Dolphin stopped answering connections when they reached the 8000, however, during the test, the image (a workspace) was still responsive (slow, but usable). Once the test ended, the image was responsive again, having more than 8000 instances of WASession, invoking a garbage collect with all those sessions in memory (with all its continuations, i.e. processes), caused a failure, and Windows showed a message saying "Dolphin Smalltalk stop responding". Why it happens at that moment, is unknown to me.

Squeak (Kom 6.2):
http://emaringolo.users.dolphinmap.net/seaside/reports/20071114/SeasideSqueak.html

The result for Squeak are similar as Dolphin. Same number of instances (average), with the difference of no having "processes" for its continuations, during the test the workspace wasn't responsive as Dolphin's one, but you can evaluate some expressions, and with patience get the print-it or do-it. But passed the test without dying. Invoking a GC forced me to interrupt the VM, because the GC blocked it.

VW (WebToolkit):
http://emaringolo.users.dolphinmap.net/seaside/reports/20071114/SeasideVW.html

VW started answer a large number of hit per second, which was as expected considering its fast VM, but started to halve its performance, and in less than 2 minutes started to show "Process Emergency" (or something similar I don't remember exactly). During the test all the windows were a dark grey rectangle, so no interaction can be done. After two minutes of test, the VM showed a out of memory message, and died abruptly.

Benchmarking

I've been running some benchmarks against Seaside on Dolphin.

I must say it runs very well, with an good page/sec rate for business apps. Anyway it can't handle being "slashdotted".

However, there is a severe problem with something leaving dangling processes in suspended or dead state, which collapses the image after 4 minutes of brute requests (above 8000 requests one after the other), generating a similar number of processes (think of it, 8000 processes!!!!).

Something smells near the Continuation implementation. Once solved the request rate should rise.

I'll publish the graphs and results when I learn how to do it. :-)

Even with that "little" problem, the performance is a little better than the Squeak implementation (15% faster).

More coming soon...