We have done enough reading and digging on the net just to see how others test and compare PHP frameworks, however there are hardly anything recent and convincing. Most of the test are on throughput of a simple 'Hello World' page, which we don't think reflect the real performance of a framework, simply because no Database interaction was involved, hence Model as in MVC is not tested at all. So we decide to conduct our own test with a more real world case, which would require MVC work together intensively to get up the result.
The Test
We have each programmer picked a popular PHP framework, and the task is to setup the framework and create MVC for the benchmark, which involves,
- 1000 database insertion
- 1000 database update
- query 1000 records (all fields), and display in a list
- each record size is about 2kb
We then put all the coded project onto one work station which has the following configuration,
- Pentium 4, 4.3GHz
- 2GB RAM
- Windows XP, SP3
- XAMPP 1.7.0 (patched to work with PDO)
- APC enabled (Zend Optimizer disabled)
- All programs turned off, including Anti-Virus and Firewall
We use a PHP batch script under CMD to make queries to each framework 10 times, and reports on the average response time and memory usage.
The Result
Framework |
Database Engine |
Avg. Response Time |
Avg. Memory Usage |
Pure PHP |
mysqli |
5.28s
|
0.14MB
|
oModel* |
adodb mysql |
7.13s
|
6.88MB
|
Yii Framework
(yiilite) |
pdo_mysql |
7.41s
|
8.38MB
|
Yii Framework |
pdo_mysql |
7.7s
|
9.44MB
|
Kohana |
mysqli |
7.68s
|
11.22MB#
|
Zend Framework |
pdo_mysql |
8.37s
|
7.99MB
|
Zend Framework |
mysqli |
11.28s
|
7.88MB
|
Akelos |
mysqli |
12.98s
|
10.93MB
|
* The Orite in-house lightweight MVC framework
# With <benchmark> parameter turned off in the database config file, it only uses 1.5MB memory, and takes about 7.8s. Thanks Jeremy Bush for the advice.
(updated 24/07/2009)
The Verdicts
Framework |
Pro. |
Con. |
oModel |
Fastest framework, light file structure, highly flexible, real short learning curve |
Not as comprehensive, not well documented, no community support, need high skill set to work on for large-scale project |
Yii Framework |
Fast, comprehensive, simple and secure file structure, strictly php5 OO, well documented, code generation |
Farely new, still building community awareness |
Kohana |
Flexibility, Easy start |
Big memory footprint, DB feature incomplete |
Zend Framework |
Great library set, flexible, best community support |
Long learning curve, code generation problem* |
Akelos |
Great RoR port, database migration |
Slow, php4 |
* We have try to run Zend-Tool on two workstations, all failed to run
The Conclusion
We understand this benchmark comparison is still bit off the real world, as it has too much emphasis on the database operation, where as in real web scenario the result can vary on different server deployment. Also, we didn't test cache mechanisms under each framework, which would play a real important role in the production environment, when server gets lot of hits on dynamic content, the result can be really different depend on project nature. Hopefully Orite will further test real web project taken most the core features of frameworks in consideration.
If we were after speed, we would stick with oModel which had been the backbone of lots of recent Orite projects. After testing the frameworks, our team had picked both Zend Framework which has the most supportive community and feature set, and Yii Framework the good combination of comprehensive feature, ease of use and performance.
We will start few project based on Yii Framework and continuously keep eyes and hands on Zend.
Interesting findings regarding this topic will be posted after we have dived into these frameworks a bit more.