Some benchmarks
Yesterday I ran some benchmark tests against Seaside on Dolphin, Squeak and VW.
The test is basically a brute force conection initiation, against the basic Seaside Counter.
I won't do any analysis or conclusion yet, just some observations, however I'm publishing the results for you our own review.
All the test were ran by separate in the same machine, an Athlon64 X2 3600+, 1GB RAM, Windows XP Pro.
Simulating 20 concurrent users starting connections against the server, one after the other, to see how much load it can handle.
Dolphin (Swazoo 1.1.4):
http://emaringolo.users.dolphinmap.net/seaside/reports/20071114/SeasideDolphin.html
Dolphin stopped answering connections when they reached the 8000, however, during the test, the image (a workspace) was still responsive (slow, but usable). Once the test ended, the image was responsive again, having more than 8000 instances of WASession, invoking a garbage collect with all those sessions in memory (with all its continuations, i.e. processes), caused a failure, and Windows showed a message saying "Dolphin Smalltalk stop responding". Why it happens at that moment, is unknown to me.
Squeak (Kom 6.2):
http://emaringolo.users.dolphinmap.net/seaside/reports/20071114/SeasideSqueak.html
The result for Squeak are similar as Dolphin. Same number of instances (average), with the difference of no having "processes" for its continuations, during the test the workspace wasn't responsive as Dolphin's one, but you can evaluate some expressions, and with patience get the print-it or do-it. But passed the test without dying. Invoking a GC forced me to interrupt the VM, because the GC blocked it.
VW (WebToolkit):
http://emaringolo.users.dolphinmap.net/seaside/reports/20071114/SeasideVW.html
VW started answer a large number of hit per second, which was as expected considering its fast VM, but started to halve its performance, and in less than 2 minutes started to show "Process Emergency" (or something similar I don't remember exactly). During the test all the windows were a dark grey rectangle, so no interaction can be done. After two minutes of test, the VM showed a out of memory message, and died abruptly.
Labels: benchmarks, seaside
posted by Esteban @ 3:58 am
8 comments
8 Comments:
Hey Esteban! very interesting tests. Can you elaborate a little on how you performed it? I mean soft you used to make the load simulation, hosts, etc?
cheers
How would forcing GC (following some criteria, like time) affect the results?
Victor
Do you think you could post the script you used to test the performance?
Also, can you do the test again with VW7.6's "to be supported" Seaside 2.8 to see how it compares with the unsupported version?
I just installed WAPT on Parallels under MacOSX to do the same tests you just did using the latest Seaside in VW7.6. I got 3 errors and 9000 odd responds.. and this was while doing Skype, playing with Expose etc on a virtual machine in Parallels... so I'd like to see your results on your setup with the most recent Cincom Seaside supported version.
Ssastre:
The software is WAPT from Softlogica, I ran the trial version, which allows you to simulate a maximum of 20 concurrent users.
Victor:
Expiring sessions or clearing its handlers every 30", or 1', keeps the VM with room to do its housekeeping, and clear the garbage. However that's a different test, not a "brute force" one.
Michael:
I'll replay the test in some time, also including GemStone's Seaside. Is 7.6 public for download?
What does this return
ObjectMemory class>>sizesAtStartup
and
ObjectMemory currentMemoryPolicy growthRegimeUpperBound
and
ObjectMemory currentMemoryPolicy memoryUpperBound
I've been running a VW based blog server for years, and have not seen problems like this.
James:
I'll get those numbers the next week of work.
However... as said before, I'm talking of performance "out of the box", I don't know how to tune VW, Squeak nor Dolphin, so I don't tune them at all. :-D
As as side note:
1) What was the peak rates handled by the server? (hits/sec).
2) You're not serving the blogs with Seaside, so the comparison isn't significant. I'm not talking about VW vs Squeak vs Dolphin, I'm comparing the different Seaside implementations, again, "out of the box".
Esteban:
I ran some tests using Seaside on Squeak, Seaside 2.7 on VW, and Seaside 2.8 on VW:
http://www.cincomsmalltalk.com/blog/blogView?showComments=true&printTitle=Scaling_Seaside_in_Cincom_Smalltalk&entry=3372841275
Post a Comment
<< Home