Performance: Nginx, RestExpress, CppCMS, Playframework, Gwan

Quick and not-scientific performance test comparison. Goal of this test is to find out how fast is the engine for my car. No car is faster than it's engine capacity. Ofcourse engine is not everything and we can't ride on the engine but it's good to know.

Concurrency Level : 200
Total Request: 100,000
OS: Linux ubuntu 2.6.38-13-generic #57-Ubuntu SMP Mon Mar 5 18:29:54 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

NOTE: I have updated PlayFramework results by using "play start" command as some one mentioned in comment which has improved performance drastically.

Summary:

Nginx CppCMS RestExpress Play2.0 Gwan
Time taken (sec) 14.17 14.724 19.653 20.896 25.348
Mean request time ms 28.34 29.448 39.448 41.793 50.697
Total requests/sec 7057.28 6791.69 5088.19 4785.52 3945.03
Transfer rate (kb/s) 2494.86 1883.63 1157.76 1065.53 1587.46
Failed Requests 0 0 0 0 2843
With Nginx Frontend CppCMS RestExpress Play2.0
Time taken (sec) 14.672 23.64 26.810
Req per  sec (ms) 29.345 47.281 53.621
Reqs per sec 6815.00 4230.05 3729.90
Transfer rate (kb/s) 2076.62 1206.23 1114.60
Failed-Requests 18 0 0

Nginx CppCMS RestExpress Play2.0 Gwan
CPU% 90 237 254 210 360
Memory (RES) 10088 34m 404m 577m 30m
Memory (SHR) 3424 4444 9996 14m 7704

CppCMS + Nginx front end:
CPU%  : 247% ( CppCMS + Nginx)
CPU%:  149% (CppCMS)
Memory (RES) : 17252 (CppCMS + Nginx)
Memory (SHR):  8260 (CppCMS + Nginx)
Memory (RES) : 6700 (CppCMS)
Memory (SHR): 4444 (CppCMS)

1.  Nginx (0.8.54)

$ ab -n 100000 -c 200 http://localhost:80/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:        nginx/0.8.54
Server Hostname:        localhost
Server Port:            80

Document Path:          /
Document Length:        151 bytes

Concurrency Level:      200
Time taken for tests:   14.170 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      36200000 bytes
HTML transferred:       15100000 bytes
Requests per second:    7057.28 [#/sec] (mean)
Time per request:       28.340 [ms] (mean)
Time per request:       0.142 [ms] (mean, across all concurrent requests)
Transfer rate:          2494.86 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        5   13   6.5     13     141
Processing:     3   15   7.5     14     145
Waiting:        2   11   6.5     10     142
Total:         16   28   9.8     26     157

Percentage of the requests served within a certain time (ms)
  50%     26
  66%     27
  75%     28
  80%     28
  90%     30
  95%     31
  98%     56
  99%     75
 100%    157 (longest request)


top - 05:18:20 up 12:14,  4 users,  load average: 1.96, 1.65, 1.38
Tasks:   5 total,   0 running,   5 sleeping,   0 stopped,   0 zombie
Cpu(s):  2.0%us, 16.7%sy,  0.0%ni, 77.2%id,  0.3%wa,  0.0%hi,  3.9%si,  0.0%st
Mem:   4056556k total,  3267276k used,   789280k free,   295200k buffers
Swap:   916476k total,     2732k used,   913744k free,  1360740k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                  
20974 www-data  20   0 71608 2212  796 S   46  0.1   0:37.47 nginx                                                                                                                    
20971 www-data  20   0 71608 2212  796 S   17  0.1   0:33.82 nginx                                                                                                                    
20973 www-data  20   0 71608 2212  796 S   14  0.1   0:31.63 nginx                                                                                                                    
20972 www-data  20   0 71608 2212  796 S   13  0.1   0:28.03 nginx                                                                                                                    
20970 root      20   0 71288 1240  240 S    0  0.0   0:00.00 nginx

2. CppCMS (1.0.1)


$ ab -n 100000 -c 200 http://localhost:10000/hello
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:        CppCMS-Embedded/1.0.1
Server Hostname:        localhost
Server Port:            10000

Document Path:          /hello
Document Length:        147 bytes

Concurrency Level:      200
Time taken for tests:   14.724 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      28400000 bytes
HTML transferred:       14700000 bytes
Requests per second:    6791.69 [#/sec] (mean)
Time per request:       29.448 [ms] (mean)
Time per request:       0.147 [ms] (mean, across all concurrent requests)
Transfer rate:          1883.63 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        8   13   2.0     13      55
Processing:     5   16   2.5     15      57
Waiting:        3   12   2.4     11      51
Total:         17   29   2.9     29     108

Percentage of the requests served within a certain time (ms)
  50%     29
  66%     29
  75%     30
  80%     31
  90%     32
  95%     33
  98%     34
  99%     35
 100%    108 (longest request)


top - 05:14:50 up 12:11,  4 users,  load average: 1.84, 1.51, 1.28
Tasks:   2 total,   2 running,   0 sleeping,   0 stopped,   0 zombie
Cpu(s):  4.5%us, 26.5%sy,  0.0%ni, 63.6%id,  0.6%wa,  0.0%hi,  4.8%si,  0.0%st
Mem:   4056556k total,  3186752k used,   869804k free,   295088k buffers
Swap:   916476k total,     2732k used,   913744k free,  1284760k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                  
21182 rjoshi    20   0  701m  34m 4444 R  237  0.9  12:13.20 hello                                                                                                                    


3. RestExpress (0.7.1)

Java settings:
/usr/lib/jvm/jdk1.7.0/jre/bin/java -classpath /Downloads/kickstart/build/classes/main:/Downloads/kickstart/lib/DateAdapterJ-1.0.0.jar:/Downloads/kickstart/lib/RestExpress-0.7.1-build-40.jar:/home/rjoshi/Downloads/kickstart/lib/gson-1.6.jar:/Downloads/kickstart/lib/junit-4.4.jar:/home/rjoshi/Downloads/kickstart/lib/netty-3.2.5.Final.jar:/Downloads/kickstart/lib/xpp3_min-1.1.4c.jar:/Downloads/kickstart/lib/xstream-1.3.1.jar com.kickstart.Main

ab -n 100000 -c 200 http://localhost:8081/hello
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:      
Server Hostname:        localhost
Server Port:            8081

Document Path:          /hello
Document Length:        148 bytes

Concurrency Level:      200
Time taken for tests:   19.653 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      23300000 bytes
HTML transferred:       14800000 bytes
Requests per second:    5088.19 [#/sec] (mean)
Time per request:       39.307 [ms] (mean)
Time per request:       0.197 [ms] (mean, across all concurrent requests)
Transfer rate:          1157.76 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   24 178.9     15    3027
Processing:     1   15   9.4     16     390
Waiting:        1   12   9.2     12     389
Total:          1   39 180.5     31    3326

Percentage of the requests served within a certain time (ms)
  50%     31
  66%     33
  75%     34
  80%     35
  90%     39
  95%     43
  98%     45
  99%     48
 100%   3326 (longest request)



top - 05:31:32 up 12:27,  4 users,  load average: 1.92, 1.54, 1.39
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
Cpu(s):  5.5%us, 24.0%sy,  0.0%ni, 65.4%id,  0.2%wa,  0.0%hi,  5.0%si,  0.0%st
Mem:   4056556k total,  3748332k used,   308224k free,   297168k buffers
Swap:   916476k total,     2732k used,   913744k free,  1395984k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                  
10207 rjoshi    20   0 1355m 404m 9996 S  254 10.2   8:51.39 java

4. Playframework (2.0)

Config:
%production.application.mode=prod
logger.root=ERROR
logger.play=ERROR
logger.application=ERROR

Started using: play start

Java settings:

java -Xms512M -Xmx1536M -Xss1M -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=384M -Dfile.encoding=UTF8 -Dplay.version=2.0 -Dsbt.ivy.home=/home/rjoshi/Downloads/play-2.0/framework/../repository -Dplay.home=/home/rjoshi/Downloads/play-2.0/framework -Dsbt.boot.properties=/home/rjoshi/Downloads/play-2.0/framework/sbt/sbt.boot.properties -jar /home/rjoshi/Downloads/play-2.0/framework/sbt/sbt-launch.jar start
rjoshi    3051  2983 99 13:35 pts/0    00:19:59 java -Dsbt.ivy.home=/home/rjoshi/Downloads/play-2.0/framework/../repository -Djava.runtime.name=Java(TM) SE Runtime Environment -Dsun.boot.library.path=/usr/lib/jvm/jdk1.7.0/jre/lib/amd64 -Djava.vm.version=21.0-b17 -Djava.vm.vendor=Oracle Corporation -Djava.vendor.url=http://java.oracle.com/ -Dpath.separator=: -Djava.vm.name=Java HotSpot(TM) 64-Bit Server VM -Dfile.encoding.pkg=sun.io -Duser.country=US -Dsun.java.launcher=SUN_STANDARD -Dsun.os.patch.level=unknown -Djava.vm.specification.name=Java Virtual Machine Specification -Duser.dir=/home/rjoshi/Downloads/play-2.0/samples/java/helloworld -Djava.runtime.version=1.7.0-b147 -Dsbt.boot.properties=/home/rjoshi/Downloads/play-2.0/framework/sbt/sbt.boot.properties -Djava.awt.graphicsenv=sun.awt.X11GraphicsEnvironment -Djava.endorsed.dirs=/usr/lib/jvm/jdk1.7.0/jre/lib/endorsed -Dos.arch=amd64 -Djava.io.tmpdir=/tmp -Dline.separator=? -Djava.vm.specification.vendor=Oracle Corporation -Dos.name=Linux -Dsun.jnu.encoding=UTF-8 -Djava.library.path=:/usr/local/lib:/usr/lib:/usr/share/lib:.:/lib://opt/arawat/lib:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib -Djava.specification.name=Java Platform API Specification -Djava.class.version=51.0 -Dplay.home=/home/rjoshi/Downloads/play-2.0/framework -Dsun.management.compiler=HotSpot 64-Bit Tiered Compilers -Dos.version=2.6.38-13-generic -Duser.home=/home/rjoshi -Duser.timezone=America/New_York -Djava.awt.printerjob=sun.print.PSPrinterJob -Dfile.encoding=UTF8 -Djava.specification.version=1.7 -Djava.class.path=/home/rjoshi/Downloads/play-2.0/framework/sbt/sbt-launch.jar -Duser.name=rjoshi -Dplay.version=2.0 -Djava.vm.specification.version=1.7 -Dsun.java.command=/home/rjoshi/Downloads/play-2.0/framework/sbt/sbt-launch.jar start -Djava.home=/usr/lib/jvm/jdk1.7.0/jre -Dsun.arch.data.model=64 -Duser.language=en -Djava.specification.vendor=Oracle Corporation -Dawt.toolkit=sun.awt.X11.XToolkit -Djava.vm.info=mixed mode -Djava.version=1.7.0 -Djava.ext.dirs=/usr/lib/jvm/jdk1.7.0/jre/lib/ext:/usr/java/packages/lib/ext -Dsun.boot.class.path=/usr/lib/jvm/jdk1.7.0/jre/lib/resources.jar:/usr/lib/jvm/jdk1.7.0/jre/lib/rt.jar:/usr/lib/jvm/jdk1.7.0/jre/lib/sunrsasign.jar:/usr/lib/jvm/jdk1.7.0/jre/lib/jsse.jar:/usr/lib/jvm/jdk1.7.0/jre/lib/jce.jar:/usr/lib/jvm/jdk1.7.0/jre/lib/charsets.jar:/usr/lib/jvm/jdk1.7.0/jre/classes -Djava.vendor=Oracle Corporation -Dfile.separator=/ -Djava.vendor.url.bug=http://bugreport.sun.com/bugreport/ -Dsun.io.unicode.encoding=UnicodeLittle -Dsun.cpu.endian=little -Dsun.desktop=gnome -Dsun.cpu.isalist= -Dhttp.port=9000 -cp /home/rjoshi/Downloads/play-2.0/samples/java/helloworld/target/scala-2.9.1/classes:/home/rjoshi/Downloads/play-2.0/framework/sbt/boot/scala-2.9.1/lib/scala-library.jar:/home/rjoshi/Downloads/play-2.0/repository/local/play/play_2.9.1/2.0/jars/play_2.9.1.jar:/home/rjoshi/Downloads/play-2.0/repository/local/play/templates_2.9.1/2.0/jars/templates_2.9.1.jar:/home/rjoshi/Downloads/play-2.0/repository/local/com.github.scala-incubator.io/scala-io-file_2.9.1/0.2.0/jars/scala-io-file_2.9.1.jar:/home/rjoshi/Downloads/play-2.0/repository/local/com.github.scala-incubator.io/scala-io-core_2.9.1/0.2.0/jars/scala-io-core_2.9.1.jar:/home/rjoshi/Downloads/play-2.0/repository/local/com.github.jsuereth.scala-arm/scala-arm_2.9.1/0.3/jars/scala-arm_2.9.1.jar:/home/rjoshi/Downloads/play-2.0/repository/local/play/anorm_2.9.1/2.0/jars/anorm_2.9.1.jar:/home/rjoshi/Downloads/play-2.0/repository/local/io.netty/netty/3.3.0.Final/bundles/netty.jar:/home/rjoshi/Downloads/play-2.0/repository/local/org.slf4j/slf4j-api/1.6.4/jars/slf4j-api.jar:/home/rjoshi/Downloads/play-2.0/repository/local/org.slf4j/jul-to-slf4j/1.6.4/jars/jul-to-slf4j.jar:/home/rjoshi/Downloads/play-2.0/repository/local/org.slf4j/jcl-over-slf4j/1.6.4/jars/jcl-over-slf4j.jar:/home/rjoshi/Downloads/play-2.0/repository/local/ch.qos.logback/logback-core/1.0.0/jars/logback-core.jar:/home/rjoshi/Downloads/play-2.0/repository/local/ch.qos.logback/logback-classic/1.0.0/jars/logback-classic.jar:/home/rjoshi/Downloads/play-2.0/repository/local/com.typesafe.akka/akka-ac



rjoshi@ubuntu:~$ ab -n 100000 -c 200 http://localhost:9000/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:      
Server Hostname:        localhost
Server Port:            9000

Document Path:          /
Document Length:        147 bytes

Concurrency Level:      200
Time taken for tests:   20.896 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      22800000 bytes
HTML transferred:       14700000 bytes
Requests per second:    4785.52 [#/sec] (mean)
Time per request:       41.793 [ms] (mean)
Time per request:       0.209 [ms] (mean, across all concurrent requests)
Transfer rate:          1065.53 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   25 197.9     11    3032
Processing:     3   16   5.5     15     267
Waiting:        2   12   4.8     11     263
Total:          3   40 198.2     26    3060

Percentage of the requests served within a certain time (ms)
  50%     26
  66%     30
  75%     32
  80%     33
  90%     37
  95%     39
  98%     44
  99%     46
 100%   3060 (longest request)



top - 13:44:18 up 22 min,  4 users,  load average: 0.41, 0.67, 0.51
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
Cpu(s): 20.0%us, 38.3%sy,  0.0%ni, 37.8%id,  0.2%wa,  0.0%hi,  3.7%si,  0.0%st
Mem:   4056556k total,  2775780k used,  1280776k free,   276848k buffers
Swap:   916476k total,        0k used,   916476k free,   933780k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                    
 3051 rjoshi    20   0 1527m 408m  12m S  430 10.3  22:16.51 java



5. CppCMS with Nginx frontend

top - 05:52:23 up 12:48,  4 users,  load average: 1.60, 1.32, 1.36
Tasks:   6 total,   1 running,   5 sleeping,   0 stopped,   0 zombie
Cpu(s):  5.0%us, 25.8%sy,  0.0%ni, 65.6%id,  0.1%wa,  0.0%hi,  3.5%si,  0.0%st
Mem:   4056556k total,  3281736k used,   774820k free,   298500k buffers
Swap:   916476k total,     2732k used,   913744k free,  1429692k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                     
10425 rjoshi    20   0  568m 6700 4444 S  149  0.2   1:16.31 hello                                                                                                                       
10366 www-data  20   0 71608 2308  892 S   37  0.1   0:14.01 nginx                                                                                                                       
10367 www-data  20   0 71696 2380  892 R   31  0.1   0:15.32 nginx                                                                                                                       
10368 www-data  20   0 71608 2308  892 S   29  0.1   0:09.18 nginx                                                                                                                       
10365 www-data  20   0 71608 2312  896 S    1  0.1   0:11.01 nginx                                                                                                                       
10364 root      20   0 71288 1244  244 S    0  0.0   0:00.00 nginx
$ ab -n 100000 -c 200 http://localhost:9999/hello
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:        nginx/0.8.54
Server Hostname:        localhost
Server Port:            9999

Document Path:          /hello
Document Length:        147 bytes

Concurrency Level:      200
Time taken for tests:   14.672 seconds
Complete requests:      100000
Failed requests:        18
   (Connect: 0, Receive: 0, Length: 18, Exceptions: 0)
Write errors:           0
Non-2xx responses:      18
Total transferred:      31200234 bytes
HTML transferred:       14700468 bytes
Requests per second:    6815.54 [#/sec] (mean)
Time per request:       29.345 [ms] (mean)
Time per request:       0.147 [ms] (mean, across all concurrent requests)
Transfer rate:          2076.62 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        6   12   3.0     12     107
Processing:     3   17   2.8     17      55
Waiting:        3   13   2.7     14      54
Total:         17   29   3.8     29     131

Percentage of the requests served within a certain time (ms)
  50%     29
  66%     30
  75%     31
  80%     31
  90%     32
  95%     33
  98%     34
  99%     35
 100%    131 (longest request)

6. Playframework with Nginx frontend

Configuration:
%production.application.mode=prod
logger.root=ERROR
logger.play=ERROR
logger.application=ERROR


rjoshi@ubuntu:~$ ab -n 100000 -c 200 http://localhost:9999/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:        nginx/0.8.54
Server Hostname:        localhost
Server Port:            9999

Document Path:          /
Document Length:        147 bytes

Concurrency Level:      200
Time taken for tests:   26.810 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      30600000 bytes
HTML transferred:       14700000 bytes
Requests per second:    3729.90 [#/sec] (mean)
Time per request:       53.621 [ms] (mean)
Time per request:       0.268 [ms] (mean, across all concurrent requests)
Transfer rate:          1114.60 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    5  53.1      4    3025
Processing:     3   44 297.0     14    3369
Waiting:        3   43 297.0     12    3368
Total:          3   50 301.6     18    3379

Percentage of the requests served within a certain time (ms)
  50%     18
  66%     22
  75%     24
  80%     25
  90%     29
  95%     32
  98%     38
  99%    284
 100%   3379 (longest request)



top - 13:50:05 up 28 min,  5 users,  load average: 0.32, 0.32, 0.39
Tasks:   6 total,   2 running,   4 sleeping,   0 stopped,   0 zombie
Cpu(s): 18.7%us, 46.5%sy,  0.0%ni, 27.9%id,  0.0%wa,  0.0%hi,  6.9%si,  0.0%st
Mem:   4056556k total,  2817740k used,  1238816k free,   277404k buffers
Swap:   916476k total,        0k used,   916476k free,   944256k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                    
 3051 rjoshi    20   0 1527m 422m  12m S  381 10.7  24:22.37 java                                                                                                                                                                      
 5561 www-data  20   0 72152 2760  796 R   45  0.1   0:06.70 nginx                                                                                                                                                                      
 5559 www-data  20   0 72664 3004  796 S   39  0.1   0:04.60 nginx                                                                                                                                                                      
 5560 www-data  20   0 72636 3244  796 S   28  0.1   0:09.14 nginx                                                                                                                                                                      
 5558 www-data  20   0 71608 2284  864 R   26  0.1   0:07.15 nginx                                                                                                                                                                      
 5557 root      20   0 71288 1248  244 S    0  0.0   0:00.00 nginx

7. RestExpress with Nginx frontend:

$ ab -n 100000 -c 200 http://localhost:9999/hello
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:        nginx/0.8.54
Server Hostname:        localhost
Server Port:            9999

Document Path:          /hello
Document Length:        148 bytes

Concurrency Level:      200
Time taken for tests:   23.640 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      29200000 bytes
HTML transferred:       14800000 bytes
Requests per second:    4230.05 [#/sec] (mean)
Time per request:       47.281 [ms] (mean)
Time per request:       0.236 [ms] (mean, across all concurrent requests)
Transfer rate:          1206.23 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    6 103.7      1    3014
Processing:     3   36 310.7     13    9320
Waiting:        2   35 310.7     13    9320
Total:          3   42 328.2     16    9320

Percentage of the requests served within a certain time (ms)
  50%     16
  66%     20
  75%     22
  80%     24
  90%     28
  95%     29
  98%     31
  99%     40
 100%   9320 (longest request)

8. GWAN (3.3.28 64-bit (Mar 28 2012 11:24:16))

$ ab -n 100000 -c 200 http://localhost:9090/?hello
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:        G-WAN
Server Hostname:        localhost
Server Port:            9090

Document Path:          /?hello
Document Length:        146 bytes

Concurrency Level:      200
Time taken for tests:   25.348 seconds
Complete requests:      100000
Failed requests:        2843
   (Connect: 0, Receive: 0, Length: 1423, Exceptions: 1420)
Write errors:           0
Total transferred:      41205186 bytes
HTML transferred:       14392242 bytes
Requests per second:    3945.03 [#/sec] (mean)
Time per request:       50.697 [ms] (mean)
Time per request:       0.253 [ms] (mean, across all concurrent requests)
Transfer rate:          1587.46 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       10   25   4.5     25      69
Processing:     6   26   4.8     26      71
Waiting:        0   18   4.6     18      71
Total:         25   50   3.7     50      96

Percentage of the requests served within a certain time (ms)
  50%     50
  66%     52
  75%     52
  80%     53
  90%     54
  95%     56
  98%     58
  99%     61
 100%     96 (longest request)

top - 10:52:11 up 17:48,  4 users,  load average: 2.17, 1.53, 1.30
Tasks:   2 total,   0 running,   2 sleeping,   0 stopped,   0 zombie
Cpu(s):  3.0%us, 24.5%sy,  0.0%ni, 64.9%id,  0.2%wa,  0.0%hi,  7.4%si,  0.0%st
Mem:   4056556k total,  3577880k used,   478676k free,   405480k buffers
Swap:   916476k total,     2640k used,   913836k free,  1642436k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                  
 8161 root      RT   0 1662m  30m 7704 S  360  0.8   2:06.54 gwan
at

17 comments:

  1. AnonymousApril 7, 2012 at 6:29 AM

    What exactly did it test? It is not described.

    What application? What kind of page?

    Hello World? (according to the link) is least interesting thing. Also it is not really correct to compare static page and dynamic application.

    Basically this benchmark says that CppCMS is as fast as high performance web server.

    But for real tests you need real applications, not something trivial.

    ReplyDelete
  2. AnonymousApril 7, 2012 at 6:40 AM

    CppCMS, RestExpress and Play framework all are dynamic applications. There is no doubt CppCMS is higher performing because it's in C++. Even if I make a complex application including connecting to DB, I bet still CppCMS will out perform.

    My performance exercise is to find out how much RestExpress is better performing than Play Framework and worst than CppCMS/Nginx.

    ReplyDelete
  3. AnonymousApril 7, 2012 at 6:58 PM

    %production.application.mode=prod setting is for Play 1.x. You should use start command instead of run (https://github.com/playframework/Play20/wiki/Production).

    ReplyDelete
  4. AnonymousApril 8, 2012 at 2:00 PM

    Thanks for information on "play start" command. Now it make sense why playframework is so popular. Infact Now I have to make decision between playframework versus node.js. CppCMS and RestExpress are no longer in criteria.
    RestExpress is good but not mature and no community support.

    CppCMS is great and with my 12+ years of C++ experience, it is perfect choice for me but new members of my project will have too much learning curve in C++ and CppCMS. Thanks Again.

    ReplyDelete
  5. AnonymousJune 11, 2012 at 12:15 PM

    It's strange that Gwan is so slow.
    Are you sure you used it correctly?

    ReplyDelete
  6. AnonymousJune 13, 2012 at 9:25 AM

    I did not use any special configuration for this test on GWAN.

    ReplyDelete
  7. AnonymousJuly 12, 2012 at 3:24 PM

    of course, gwan is slow on dynamic page. you seem Pierre tested mostly are static page and hello world.

    ReplyDelete
  8. UnknownJuly 13, 2012 at 2:42 AM

    For Play, better than play start, make a play dist, unzip the target, und run it directly. (sbt will not run in that case)

    ReplyDelete
  9. AnonymousAugust 8, 2012 at 9:52 AM

    nginx is a single-threaded server.

    GWan is a multi-threaded server.

    So using ApacheBench (single-threaded) to test GWan is idiot: AB will never saturate all the GWan threads.

    Use weightt client written by Lightty, it has the same options but is multi-threaded.

    Several instances of nginx can be used in parallel to act like a multi-threaded server, but there is no way for a native multi-threaded server like GWan to act like a single-threaded server.

    Single-threaded servers are for single-core CPUs. The last one was make in year 2004.

    ReplyDelete
  10. AnonymousAugust 16, 2012 at 4:30 AM

    Without HTTP keep-alives you are testing the TCP/IP stack rather than the web servers...

    ReplyDelete
  11. DeepwaterHorizonAugust 17, 2012 at 8:44 AM

    Sorry that's tldr; and badly formatted :(
    Please invest more time and heartblood when writing articles :)

    It's weird that gwan failed so many requests. I've absolutely ZERO affiliation with them, but when I try that myself I don't get any failed requests. And you've only 3k requests?? I'm really sorry, but you must be kidding. It's been some time since I tried the gwan server, but I got many many times more requests/sec on static AND dynamic c code or "scripts".

    ReplyDelete
  12. Rohit JoshiAugust 17, 2012 at 11:18 AM

    Even I am surprised too as I heard good things about gwan. Infact that is the reason, it is included in the test.
    I didn't see any custom configuration for hello word.

    It is the same hardware/software used for performance test.
    What do you think could be wrong?

    ReplyDelete
  13. Rohit JoshiAugust 17, 2012 at 11:23 AM

    Response to Single threaded/MultiThreaded comment:

    Apache AB is not single threaded. If you look at for test, the concurrency level is set to 200.

    ReplyDelete
  14. AnonymousSeptember 9, 2012 at 3:15 PM

    Rohit Joshi, you're wrong. Look up wikipedia on AB using Single Thread regardless of concurrency level.

    Even if weighttp is benchmark on various web servers. GWan only good with small files, only useful for small site or hobbyist role.

    Play Framework is the most interesting web server and productivity tools that is most valued by Java developers to leverage their existing skills and projects.

    Most companies can't afford to hire C/C++ developers to work with Java side, it can create the confusion in planning.

    http://lonewolfer.wordpress.com/2011/10/08/benchmarking-the-benchmarks-part-2/

    ReplyDelete
  15. AnonymousSeptember 17, 2012 at 5:15 PM

    Apache bench is single threaded but can still generate multiple requests at the same time from that single thread. Just because GWAN is multithreaded and ab is single doesn't make a bit of difference. AB is still generating simultaneous requests.

    Weighttp is multithreaded and slightly better at saturating a gigabit link though.

    My guess is he's not turning on keepalives which explains the poor performance. There is too much time spent establishing and tearing down TCP connections.

    With static content on a core i7 you should be getting close to 500,000 req/s. But it really depends what's being served.
    No point doing a hello world. Do something a little more and compare.

    ReplyDelete
  16. AnonymousSeptember 17, 2012 at 6:44 PM

    Why it only affects GWAN? I didn't see any keepalive config for GWAN?

    ReplyDelete
  17. Rohit JoshiSeptember 17, 2012 at 6:53 PM

    On GWAN site (FAQ):
    Even with one single AB worker G-WAN served 2.5x more requests in 2x less time than Nginx –the singled-thread server architecture before 2004).

    ReplyDelete