Thursday, November 15, 2007

Some benchmarks

Yesterday I ran some benchmark tests against Seaside on Dolphin, Squeak and VW.

The test is basically a brute force conection initiation, against the basic Seaside Counter.

I won't do any analysis or conclusion yet, just some observations, however I'm publishing the results for you our own review.

All the test were ran by separate in the same machine, an Athlon64 X2 3600+, 1GB RAM, Windows XP Pro.
Simulating 20 concurrent users starting connections against the server, one after the other, to see how much load it can handle.

Dolphin (Swazoo 1.1.4):
http://emaringolo.users.dolphinmap.net/seaside/reports/20071114/SeasideDolphin.html

Dolphin stopped answering connections when they reached the 8000, however, during the test, the image (a workspace) was still responsive (slow, but usable). Once the test ended, the image was responsive again, having more than 8000 instances of WASession, invoking a garbage collect with all those sessions in memory (with all its continuations, i.e. processes), caused a failure, and Windows showed a message saying "Dolphin Smalltalk stop responding". Why it happens at that moment, is unknown to me.

Squeak (Kom 6.2):
http://emaringolo.users.dolphinmap.net/seaside/reports/20071114/SeasideSqueak.html

The result for Squeak are similar as Dolphin. Same number of instances (average), with the difference of no having "processes" for its continuations, during the test the workspace wasn't responsive as Dolphin's one, but you can evaluate some expressions, and with patience get the print-it or do-it. But passed the test without dying. Invoking a GC forced me to interrupt the VM, because the GC blocked it.

VW (WebToolkit):
http://emaringolo.users.dolphinmap.net/seaside/reports/20071114/SeasideVW.html

VW started answer a large number of hit per second, which was as expected considering its fast VM, but started to halve its performance, and in less than 2 minutes started to show "Process Emergency" (or something similar I don't remember exactly). During the test all the windows were a dark grey rectangle, so no interaction can be done. After two minutes of test, the VM showed a out of memory message, and died abruptly.

Labels: ,

8 Comments:

At 7:24 am, Blogger ssastre said...

Hey Esteban! very interesting tests. Can you elaborate a little on how you performed it? I mean soft you used to make the load simulation, hosts, etc?
cheers

 
At 8:08 am, Blogger Victor said...

How would forcing GC (following some criteria, like time) affect the results?

Victor

 
At 9:26 am, Blogger Michael said...

Do you think you could post the script you used to test the performance?
Also, can you do the test again with VW7.6's "to be supported" Seaside 2.8 to see how it compares with the unsupported version?

 
At 10:18 am, Blogger Michael said...

I just installed WAPT on Parallels under MacOSX to do the same tests you just did using the latest Seaside in VW7.6. I got 3 errors and 9000 odd responds.. and this was while doing Skype, playing with Expose etc on a virtual machine in Parallels... so I'd like to see your results on your setup with the most recent Cincom Seaside supported version.

 
At 6:17 pm, Blogger Esteban said...

Ssastre:
The software is WAPT from Softlogica, I ran the trial version, which allows you to simulate a maximum of 20 concurrent users.

Victor:
Expiring sessions or clearing its handlers every 30", or 1', keeps the VM with room to do its housekeeping, and clear the garbage. However that's a different test, not a "brute force" one.

Michael:
I'll replay the test in some time, also including GemStone's Seaside. Is 7.6 public for download?

 
At 2:17 pm, Blogger jarober said...

What does this return

ObjectMemory class>>sizesAtStartup

and

ObjectMemory currentMemoryPolicy growthRegimeUpperBound

and

ObjectMemory currentMemoryPolicy memoryUpperBound


I've been running a VW based blog server for years, and have not seen problems like this.

 
At 2:34 pm, Blogger Esteban said...

James:
I'll get those numbers the next week of work.
However... as said before, I'm talking of performance "out of the box", I don't know how to tune VW, Squeak nor Dolphin, so I don't tune them at all. :-D

As as side note:
1) What was the peak rates handled by the server? (hits/sec).
2) You're not serving the blogs with Seaside, so the comparison isn't significant. I'm not talking about VW vs Squeak vs Dolphin, I'm comparing the different Seaside implementations, again, "out of the box".

 
At 9:31 am, Blogger jarober said...

Esteban:

I ran some tests using Seaside on Squeak, Seaside 2.7 on VW, and Seaside 2.8 on VW:

http://www.cincomsmalltalk.com/blog/blogView?showComments=true&printTitle=Scaling_Seaside_in_Cincom_Smalltalk&entry=3372841275

 

Post a Comment

<< Home

web site analysis