Never fear! A solution is at hand!

What went wrong

The cause of these problems has been the architecture running behind ZuFeed. Although I designed much of the backend to scale greatly, history now shows that one major decision was incorrect… well at least is now incorrect for a site holding more than 3.3 million articles.

As ZuFeed uses MySQL as a backend, and fulltext searching is required, the original decision was to hold all the data in MyISAM tables and search using the built-in fulltext search indexes. As ZuFeed has grown searches have become a lot slower and new articles are added only seconds apart. This is due to two issues with using MyISAM and one coding error on my part.

1. Poor scaling of MyISAM fulltext searching (for a database of 7GB+ in size)
2. Constant MyISAM table level locking for each new article entry
3. A missing index for the top searches

The Solution

The third problem was easily solved by identifying the needed index and adding it to the database. Result? The search summary page now shows up without delays.

Solving the second problem required changing the storage engine to one that used row-level locking instead of table-level locking. The easiest options available were InnoDB and PBXT. Unlike InnoDB, PBXT is not included by default in MySQL installations and is instead available as a plugin. MariaDB (a fork of MySQL) however does include PBXT.

Both InnoDB and PBXT are robust, tranactional and ACID compliant storage engines. However PBXT is also log based, which allows for fast speed in a large variety of scenarios. As I have been using PBXT for a while now in another project, I decided to switch the tables to PBXT.

Since PBXT (and InnoDB) for that matter do not include fulltext searching, I needed a replacement search engine. I decided to use another open source project with a simple setup and very fast search speeds and easy integration with MySQL and php – Sphinx.

The Implementation

I attempted to have as little down time as possible with the testing and then implementation of the new systems. Overall it went something like this:

Part 1 – Sphinx Search Engine

• Add a new auto incremented unique id of type integer to the databases (as required for Sphinx)
• Set up Sphinx to use two indexes – one main and one delta for updates to allow shorter re-indexing times (although not quite realtime)
• Schedule delta index re-indexing every 60 seconds via a cron job.
• Run initial indexing of article database – Warning! This can take long time!
• Test Sphinx vs MySQL fulltext results via php.

The results showed significantly faster searches using Sphinx over the MySQL in-built fulltext search. Sphinx also allowed more information to be searched. I was now able to include summary, author and permalink url fields to the search without penalty.

Part 2 – Integrate Sphinx into ZuFeed

• Change php classes to use Sphinx instead of MySQL fulltext searching
• Add unique id conversion functions between new and old id’s
• Maintain the same API to avoid changing other code
• Test to make sure searching still works in ZuFeed

Part 3 – MyISAM to PBXT Conversion

• Remove fulltext indexes from each table
• Convert each table to use PBXT

When converting a table from one storage engine to another, keep in mind that ALL data has to be copied to a new temporary table first, then renamed to remove the original. As such this takes a long time for multi-gigabytes tables.

Conclusion

And there you have it.

ZuFeed is now quicker at searching, responds to new requests faster, no longer times out on loading, and also produces more relevant results.

It will be running a variety of applications including:

• Mandriva Linux 2010.0
• Web Serving: Nginx, Apache 2.2, PHP 5.3
• Mail: Postfix, Dovecot, RoundCube
• DNS: PowerDNS, Bind
• Database: MySQL
• Virtual Hosting: VirtualBox

Server Specifications:

• Intel SR1530HSH with S3200SHVL Motherboard. 320w PSU, 3x Hot Swap SATA HDD, 2x Gb Nic
• Intel Q9550 Processor – 4x 2.83ghz, 12mb cache, e0 stepping
• Intel Q9400 Processor – 4x 2.66ghz, 6mb cache, r0 stepping
• 4x 2gb (8gb total) Kingston DDR2 800Mhz Ram
• 2x 1Tb Seagate Hard Drives (in RAID 1)

Update: Unfortunately, the E0 stepping Q9550 is not supported for the bios of the s3200 motherboard, so I’ve had to switch down to the Q9400 with only 6mb cache .

Photos:

Intel SR1530HSH 1U Server - Front

Intel SR1530HSH 1U Server - Rear

Intel SR1530HSH 1U Server - Top

Intel SR1530HSH 1U Server - Inside

First thoughts? Nginx is super fast, reasonably easy to configure (once you figure out the php issues) and saves a heap on memory. Downside? I’ve had the odd niggle with moving old sites across – especially since I’m moving to php 5.3 and MySQL 5.1 at the same time, otherwise all good.

The other major change is the look of this blog. The old K2 theme was becoming an issue to deal with (mainly because I didn’t update it enough) so I’ve now moved to this lovely theme: inove. I’ve also done a fresh wordpress install of 2.8.4 and added pretty links – so anyone linking to the old structure might have the odd issue, but otherwise should be painless. Oh yeah – any images on the older posts (pre Aug 2009) are gone.

Anyway here is a list of upgrades currently underway:

• php 5.2.x to 5.3.0
• mysql 5.0 to mysql-max 5.1
• apache 2.2/mod_php to nginx 0.8.10/php-fcgi
• athlon64 2ghz 1gb ram to athlon64 X2 2ghz 2gb ram
• mandriva 2008.x/2009.x to 2010.0 (Cooker)
• wordpress 2.x.x to 2.8.4
• mediawiki 1.x to 1.15.1
• bind 9 to powerdns 2.9.22
• and a few more that i’ve forgotten about