Orite Blog

PHP Frameworks Comparison Test

We have done enough reading and digging on the net just to see how others test and compare PHP frameworks, however there are hardly anything recent and convincing. Most of the test are on throughput of a simple 'Hello World' page, which we don't think reflect the real performance of a framework, simply because no Database interaction was involved, hence Model as in MVC is not tested at all. So we decide to conduct our own test with a more real world case, which would require MVC work together intensively to get up the result.

The Test

We have each programmer picked a popular PHP framework, and the task is to setup the framework and create MVC for the benchmark, which involves,

• 1000 database insertion
• 1000 database update
• query 1000 records (all fields), and display in a list
• each record size is about 2kb

We then put all the coded project onto one work station which has the following configuration,

• Pentium 4, 4.3GHz
• 2GB RAM
• Windows XP, SP3
• XAMPP 1.7.0 (patched to work with PDO)
• APC enabled (Zend Optimizer disabled)
• All programs turned off, including Anti-Virus and Firewall

We use a PHP batch script under CMD to make queries to each framework 10 times, and reports on the average response time and memory usage.

The Result

Framework	Database Engine	Avg. Response Time	Avg. Memory Usage
Pure PHP	mysqli	5.28s	0.14MB
oModel*	adodb mysql	7.13s	6.88MB
Yii Framework (yiilite)	pdo_mysql	7.41s	8.38MB
Yii Framework	pdo_mysql	7.7s	9.44MB
Kohana	mysqli	7.68s	11.22MB#
Zend Framework	pdo_mysql	8.37s	7.99MB
Zend Framework	mysqli	11.28s	7.88MB
Akelos	mysqli	12.98s	10.93MB

* The Orite in-house lightweight MVC framework
# With <benchmark> parameter turned off in the database config file, it only uses 1.5MB memory, and takes about 7.8s. Thanks Jeremy Bush for the advice. (updated 24/07/2009)

The Verdicts

Framework	Pro.	Con.
oModel	Fastest framework, light file structure, highly flexible, real short learning curve	Not as comprehensive, not well documented, no community support, need high skill set to work on for large-scale project
Yii Framework	Fast, comprehensive, simple and secure file structure, strictly php5 OO, well documented, code generation	Farely new, still building community awareness
Kohana	Flexibility, Easy start	Big memory footprint, DB feature incomplete
Zend Framework	Great library set, flexible, best community support	Long learning curve, code generation problem*
Akelos	Great RoR port, database migration	Slow, php4

* We have try to run Zend-Tool on two workstations, all failed to run

The Conclusion

We understand this benchmark comparison is still bit off the real world, as it has too much emphasis on the database operation, where as in real web scenario the result can vary on different server deployment. Also, we didn't test cache mechanisms under each framework, which would play a real important role in the production environment, when server gets lot of hits on dynamic content, the result can be really different depend on project nature. Hopefully Orite will further test  real web project taken most the core features of frameworks in consideration. If we were after speed, we would stick with oModel which had been the backbone of lots of recent Orite projects. After testing the frameworks, our team had picked both Zend Framework which has the most supportive community and feature set, and Yii Framework the good combination of comprehensive feature, ease of use and performance. We will start few project based on Yii Framework and continuously keep eyes and hands on Zend. Interesting findings regarding this topic will be posted after we have dived into these frameworks a bit more.

Be Sociable, Share!

• 
• 

• Tweet
• 

10 Comments

• Wei says:

  February 5, 2009 at 10:33 am

  Just to let you know, one of the developers in Yii only lives like < 2km from your office.

  • Frank Liu says:

    February 5, 2009 at 10:56 am

    Thanks Wei, it would be interesting to know the local developer and possibly have a chat with.

• Gerd says:

  March 13, 2009 at 7:22 am

  It would be interesting to also test the Symfony framework http://www.symfony-project.org. I have been using it for a while now but am curious about Yii, which claims much higher performance.

  • Frank Liu says:

    March 13, 2009 at 1:52 pm

    Hi Gerd,
    Thanks for your comment, to my knowledge of Symfony (pre 1.0) it would be lot slower than Yii for sure in terms of output performance. However to evaluate a framework you may want to consider other aspects as well, ie. available tool sets, community activity/support, speed of delivery, deployment, code maintenance, etc. I’d recommend you to try out Yii framework, to see the differences yourself. I believe Yii would certainly please you in quite few things.

• darwin says:

  June 23, 2009 at 9:59 pm

  thx for your review about PHP framework, before read this, i’m confuse which framework that i must use.

• Kee Chien says:

  July 21, 2009 at 8:54 am

  Does anyone ever use doophp? Just curious about its benchmark http://doophp.com/benchmark . Would anyone verify this? thanks

  • Frank Liu says:

    July 21, 2009 at 9:58 am

    Hi Kee,
    Thanks for your comment, we haven’t tried doophp before, however we strongly believe the real world performance of doophp would not out perform the others by that much. As you may notice in our test, pure php mysql connection is only 40% faster than the Yii Framework. Unless cache mechanism is involved, we would be amazed to find any database abstraction would actually be faster than pure php mysql connection.
    It may be 5 or more times faster in producing the ‘Hello World’ page, which only hit the frameworks view/action controller. We believe you need to take other components into consideration while doing a real test-case. In most the real world projects caching would be the final determinate factor of ‘performance’ (at a users point of view), that really matters most. In other words, if you know how to tweak caching on all database, page, assets, media levels, any framework should really be the same, in terms of serving pages (cached).
    Therefore, what really matters to a framework would not so much in performance, but really lies on the feature set and support.
    Cheers!

• Jeremy Bush says:

  July 23, 2009 at 1:35 am

  It would be interesting to see the source code of these benchmarks, and how you did the database code. That memory number seems awfully high.

  • Frank Liu says:

    July 23, 2009 at 11:43 am

    Hi Jeremy, just emailed you the benchmark code, please review and post back your valuable thoughts.

• uG says:

  August 21, 2009 at 11:37 pm

  Where is CakePHP and CodeIgniter ?

Trackbacks/Pingbacks

• PHP Frameworks Comparison Summary « Allan on Web (June 17, 2009 3:18 pm)

RSS feed for comments on this post.