Never fear! A solution is at hand!

What went wrong

The cause of these problems has been the architecture running behind ZuFeed. Although I designed much of the backend to scale greatly, history now shows that one major decision was incorrect… well at least is now incorrect for a site holding more than 3.3 million articles.

As ZuFeed uses MySQL as a backend, and fulltext searching is required, the original decision was to hold all the data in MyISAM tables and search using the built-in fulltext search indexes. As ZuFeed has grown searches have become a lot slower and new articles are added only seconds apart. This is due to two issues with using MyISAM and one coding error on my part.

1. Poor scaling of MyISAM fulltext searching (for a database of 7GB+ in size)
2. Constant MyISAM table level locking for each new article entry
3. A missing index for the top searches

The Solution

The third problem was easily solved by identifying the needed index and adding it to the database. Result? The search summary page now shows up without delays.

Solving the second problem required changing the storage engine to one that used row-level locking instead of table-level locking. The easiest options available were InnoDB and PBXT. Unlike InnoDB, PBXT is not included by default in MySQL installations and is instead available as a plugin. MariaDB (a fork of MySQL) however does include PBXT.

Both InnoDB and PBXT are robust, tranactional and ACID compliant storage engines. However PBXT is also log based, which allows for fast speed in a large variety of scenarios. As I have been using PBXT for a while now in another project, I decided to switch the tables to PBXT.

Since PBXT (and InnoDB) for that matter do not include fulltext searching, I needed a replacement search engine. I decided to use another open source project with a simple setup and very fast search speeds and easy integration with MySQL and php – Sphinx.

The Implementation

I attempted to have as little down time as possible with the testing and then implementation of the new systems. Overall it went something like this:

Part 1 – Sphinx Search Engine

• Add a new auto incremented unique id of type integer to the databases (as required for Sphinx)
• Set up Sphinx to use two indexes – one main and one delta for updates to allow shorter re-indexing times (although not quite realtime)
• Schedule delta index re-indexing every 60 seconds via a cron job.
• Run initial indexing of article database – Warning! This can take long time!
• Test Sphinx vs MySQL fulltext results via php.

The results showed significantly faster searches using Sphinx over the MySQL in-built fulltext search. Sphinx also allowed more information to be searched. I was now able to include summary, author and permalink url fields to the search without penalty.

Part 2 – Integrate Sphinx into ZuFeed

• Change php classes to use Sphinx instead of MySQL fulltext searching
• Add unique id conversion functions between new and old id’s
• Maintain the same API to avoid changing other code
• Test to make sure searching still works in ZuFeed

Part 3 – MyISAM to PBXT Conversion

• Remove fulltext indexes from each table
• Convert each table to use PBXT

When converting a table from one storage engine to another, keep in mind that ALL data has to be copied to a new temporary table first, then renamed to remove the original. As such this takes a long time for multi-gigabytes tables.

Conclusion

And there you have it.

ZuFeed is now quicker at searching, responds to new requests faster, no longer times out on loading, and also produces more relevant results.

What follows from here is my look at the Mandriva vs. Mageia battle for supremacy on my machines. I’ve been using Mandriva for a long time (since the 8.2 days) and many of the reasons I chose it initially such as ease of use, active community, constant innovation, stability, common server/desktop and amazing configuration tools (DrakX), have remained the same and stopped tempting migrations to Fedora, Ubuntu or Suse. However some things change. Mandriva’s commercial stability has been lacking for a long time, which produced the laying off of a large group of developers – the same developers who decided they’d had enough and created Mageia.

Mageia being a fork of Mandriva has allowed it to keep many of the features as above, and the time has come for a decision: stay with Mandriva, move to Mageia or consider something else entirely.

Please note that I prefer to keep all installations that I manage the same, including servers and desktops, which does remove many other excellent distributions from consideration and also is why Mageia is my first port of call.

Mandriva vs. Mageia – Summary

I have followed the development of both distributions since the announcement with eagerness and interest. Here is my summary of the current status and paths of each:

Mandriva

• Heavy development of new UI tools for easier use (some describe as ‘Mac’ like)
• New investment in the company with Russian money (ROSA Labs)
• Attempting to make a huge jump in lots of new software at once – including RPM5 and systemd
• Delays due to reduced personnel and the huge amount of changes.
• Changeable focus and release schedules
• Upgradeable from previous versions but with some difficulty (due major software changes)

Mageia

• Started with Mandriva 2010.1 but with all packages cleaned and upgraded to the latest versions
• Very, very stable pre-releases (as well as the first release Mageia 1)
• No major changes to the implementation
• Very active new community and good work being done with other distributions
• Allows the community to define the direction, development and core ideas in theory and especially in practice
• Upgradeable from Mandriva 2010.1 officially – see the upgrade guide

Mageia Review

The team at Mageia have taken their time to set up their systems and process for long term success and survivability. These considerations molded what the first release Mageia 1 was going to be like.

The stated focus was:

• Building the Build System
• Start from Mandriva 2010.1, clean, review and rebuild everything
• No major changes – just software updates
• Only include packages as requested

All of this resulted in Mageia 1′s pre-releases (Alpha, Betas, RC) and final release all being exceptionally stable, matching all the features from the best Mandriva release 2010.1 and creating a very clean base to build from. There were no huge failures post release, no obvious beta software included and no controversial or courageous decisions. Just a very solid a capable release.

I won’t rehash other full reviews of Mageia 1. It truly is just a very updated Mandriva 2010.1.

New packages, missing packages

All package in Mageia are cleaned and updated. As such it includes: Linux Kernel 2.6.38, KDE 4.6.3, Gnome 2.32, XFCE 4.8.1, Firefox 4, LibreOffice 3.3.2 and VirtualBox 4.0.6. See the Release Notes for further information.

However if no-one was found to maintain a package, clean and rebuild it, it wasn’t included. As such Mageia 1 ships with much less available software than Mandriva 2010.1, but everything works, most packages people want are there and many more a being added post release.

Repositories and naming

During development of Mageia 1 there was a large (somewhat epic) discussion on how to manage the repositories, names for each and what they should hold. The rolling development release is called Cauldron and works much like Mandriva’s Cooker.

The repositories are:

• Core – free open source packages. The default repository for most packages.
• Non-free – free of charge but not open source packages. E.g. Nvidia and AMD graphics drivers
• Tainted – packages that might infringe on patents in some countries. A la PLF for Mandriva or RPM Fusion for Fedora.

Desktop Look and Feel

Mageia stripped out the icons, wallpaper and theme from Mandriva and started again. The dotted wallpaper is nice and basic in my opinion, but is easily changed to many other nicer ones.

Mageia Control Center (MCC)

I much prefer the new icons in the KDE system tray and Mageia Control Center (MCC) as shown.

Mageia KDE System Tray

However the main point of difference on the desktop, and the only point of controversy is the look of the fonts. That’s right, not the fonts themselves but the way they look. Mageia has decided to enable autohinting which produces cleaner, slimmer fonts. Some say it looks ugly (Bug 175), others like it. Now, I used to be in the “ugly” camp and made sure that my Mandriva installations disabled the PLF version of the fonts which had this effect. However I’ve changed my mind after trying them out for a while and quite like them.

One other change is that KDE seems a lot more stable than previously – including that using the nepomuk file search from within dolphin actually works, in real time and doesn’t drastically slow the machine in the background. Though this is due not so much the Mageia team but to the new KDE version and the great work of the KDE team – so all distributions should also be enjoying the benefits.

Release schedule

Both Mandriva and Mageia are departing from the 6 month release schedule of the Mandriva 2010 series. Mandriva is moving to yearly release whilst after much discussion Mageia is moving to a 9 monthly release cycle. Mageia’s next release is scheduled for 4th April 2012.

The Decision

Mageia continues to do everything I wanted from Mandriva and adds excellent stability, transparency and has shown a great willingness to take greater input and direction from the community. The 9 month release schedule allows for up to date software, but allows enough time to make sure all is stable before release without manic last minute bugs being left.

I have decided to migrate all my Mandriva installations to Mageia and to hopefully participate more as part of the Mageia community.

The migration story so far…

For each machine I’ve moved so far to Mageia I’ve listed the machine specs, previous Mandriva version if any, install type, Mageia edition and a few comments from the install.

Machine 1: Laptop

• Specs: Toshiba Satellite A100 – Intel Core Duo T2300, 1GB RAM, GeForce Go 7300
• Previously: Mandriva 2010.1 32bit Powerpack
• Currently: Mageia 1 32bit KDE via fresh install
  My first install of Mageia was to my own laptop to get a feel of things. I tested the Beta 1 (fresh install), Beta 2 (upgrade),  RC (upgrade) and final (fresh install) and found amazing stability – no crashes or major hiccups at all. It has also certainly also felt quite a bit faster than the previous Mandriva 2010.2. As the Skype packages for Mandriva don’t work anymore, I installed Skype using the notes as shown here.

Machine 2: Desktop

• Specs: AMD Phenom II X4 965BE, 8GB RAM, ATI Radeon 4770
• Previously: Mandriva 2010.2 64bit Powerpack (with many backports)
• Currently: Mageia 1 64bit KDE via upgrade
  Upgraded via the command line with urpmi as per the Mageia upgrade guide. No settings went awry and all worked well. Left over packages from Mandriva that had no Mageia equivalent continue to work correctly. Feels slightly faster.

Machine 3: MythTV HDTV Media Center

• Specs: AMD Athlon X2 6000+, 4G Ram, ATI Radeon X1250 (690G), Dvico FusionHDTV card
• Previously: Mandriva 2010.2 64bit Powerpack
• Currently: Mageia 1 64bit LXDE via upgrade
  Upgraded via the command line using urpmi cleanly but ssh remote connection. TV card continued to work without issues. Still running Mandriva PLF MythTV packages for the moment – I didn’t want to risk that upgrade not working yet. Stopped mythtv services to ensure mysql clean upgrade before restarting for the new kernel.

Machine 4: Laptop

• Specs: Toshiba Portege M800 in Pink – Intel Core 2 Duo P7350, 2GB RAM, Intel Mobile 4 Graphics
• Previously: Mandriva 2010.1 64bit Powerpack
• Currently: Mageia 1 64bit KDE via upgrade
  Most risky upgrade – my wife’s laptop. Upgraded safely with no lost settings or files. Main difference was the font look change as mentioned above. Microphone continues the same habit of not working as seen under Mandriva. Overall a great success!

Machine 5: Production Web and Database Server

• Specs: Intel Xeon 3.0Ghz (P4 based with HT), 2GB Ram, 2x80GB HDD in Raid 1 (mdadm)
• Previously: Mandriva 2009.0/2009.1 Hybrid 64bit
• Currently: Mageia 1 64bit via online upgrade
  This server has been constantly upgraded from Mandriva 2007.1 onwards. As only upgrades from 2010.1 are officially supported I felt this was a bit risky, but worth a try. I allocated enough time to rebuild from scratch if needed, however it was not necessary. To reduce the risk I ran urpmi with the extra –test option to make sure all packages were available for install first. All services came up successfully to my pleasant surprise. I did spend a couple of hours slowly removing leftover packages that had slowly built up over time – including 14 obsolete kernels.

So over the past week we’ve moved from 0.3.x to 0.4.2 and now to the newly released RoundCube version 0.5.

I hope everyone enjoys the update (and that no data was lost *crosses fingers*).

I checked around for a more up-to-date rpm for Mandriva 2010.0 but none currently exists. There is 0.8.41 and 0.8.53 avaliable for 2010.1 and cooker. So…. one of three decisions to make.

1. Upgrade the server to 2010.1
2. Use either the 2010.1 or cooker nginx package on my existing install
3. Request a backport for 2010.0
4. Backport the package myself

Since this is a stable server an entire upgrade is a big risk and 2010.0 has a longer support timeframe, option 1 can be ignored. The same can be said about running packages not designed for the system. There goes option 2.

I decided to do both option 3 (Mandriva Bug Report) and 4.

I’ve never built an rpm before and thought that it was about time to learn, especially as I’d like to build up to being a maintainer/packager for the recently forked Mageia. So I set up a 2010.0 64bit build server and a 64bit test server and started using the Mandriva RPM Howto.

I downloaded the cooker source rpm for nginx 0.8.53 and rebuilt it for 2010.0.

Success! nginx-0.8.53-2mdv2010.0.x86_64.rpm

Since I now have the rpms (and along with my desire to do more packaging) I decided to use genhdlist2 and my rsync server to create my own repository – CyprixRPMS.

Using my own repo I upgraded my main server to 0.8.53 after basic testing on my 64bit test virtual machine.

You can add my repo with the command below as root (or sudo) for Mandriva 2010.0 64bit.

urpmi.addmedia cyprixrpms rsync://proteus.cyprix.com.au::cyprixrpms/mandriva/2010.0/x86_64

If you have any feedback on my rpms please email me.

It will be running a variety of applications including:

• Mandriva Linux 2010.0
• Web Serving: Nginx, Apache 2.2, PHP 5.3
• Mail: Postfix, Dovecot, RoundCube
• DNS: PowerDNS, Bind
• Database: MySQL
• Virtual Hosting: VirtualBox

Server Specifications:

• Intel SR1530HSH with S3200SHVL Motherboard. 320w PSU, 3x Hot Swap SATA HDD, 2x Gb Nic
• Intel Q9550 Processor – 4x 2.83ghz, 12mb cache, e0 stepping
• Intel Q9400 Processor – 4x 2.66ghz, 6mb cache, r0 stepping
• 4x 2gb (8gb total) Kingston DDR2 800Mhz Ram
• 2x 1Tb Seagate Hard Drives (in RAID 1)

Update: Unfortunately, the E0 stepping Q9550 is not supported for the bios of the s3200 motherboard, so I’ve had to switch down to the Q9400 with only 6mb cache .

Photos:

Intel SR1530HSH 1U Server - Front

Intel SR1530HSH 1U Server - Rear

Intel SR1530HSH 1U Server - Top

Intel SR1530HSH 1U Server - Inside

On apache, this can occur when your wordpress directory does not have write permissions for the apache user. But I’d already checked that. So I got thinking…

Since nginx and php fastcgi are separate processes I checked the users they were running under – lo and behold the php-fcgi was running under root!!! Big oops! Solution: change php-fcgi to run as user nginx & group nginx (same as the nginx server under Mandriva). As I currently use the spawn-fcgi program I changed by init.d file to use the user and group options as shown below.

/usr/local/bin/spawn-fcgi -u nginx -g nginx

All fixed

At some stage I’ll post my init.d file for php-fcgi and my nginx configuration files once they become somewhat stable.

First thoughts? Nginx is super fast, reasonably easy to configure (once you figure out the php issues) and saves a heap on memory. Downside? I’ve had the odd niggle with moving old sites across – especially since I’m moving to php 5.3 and MySQL 5.1 at the same time, otherwise all good.

The other major change is the look of this blog. The old K2 theme was becoming an issue to deal with (mainly because I didn’t update it enough) so I’ve now moved to this lovely theme: inove. I’ve also done a fresh wordpress install of 2.8.4 and added pretty links – so anyone linking to the old structure might have the odd issue, but otherwise should be painless. Oh yeah – any images on the older posts (pre Aug 2009) are gone.

Anyway here is a list of upgrades currently underway:

• php 5.2.x to 5.3.0
• mysql 5.0 to mysql-max 5.1
• apache 2.2/mod_php to nginx 0.8.10/php-fcgi
• athlon64 2ghz 1gb ram to athlon64 X2 2ghz 2gb ram
• mandriva 2008.x/2009.x to 2010.0 (Cooker)
• wordpress 2.x.x to 2.8.4
• mediawiki 1.x to 1.15.1
• bind 9 to powerdns 2.9.22
• and a few more that i’ve forgotten about

Mandriva Linux 2008.0 is out and going strong. Keeping that in mind, I have now updated this server to run 2008.0 with very few problems (via urpmi –auto-select). This is a first for me with any Mandriva server. Well done guys!

I’ve also installed the Powerpack on my Toshiba Satellite 2410 laptop with no real issues, but with great new fonts and features (like automount ntfs-3g) all around.

I’ve finally finished stage one of my Cyprix Mail Server Howto!

The howto is a very much step-by-step method of setting up a virtual domain, virtual user mail server using:

• Postfix
• MySQL
• Courier-IMAP (and POP3)
• RoundCube Webmail (PHP/AJAX)

Currently the howto is based on Mandriva Linux, however with minor changes it can be applied to most Linux/Unix based distributions.

As I mentioned before this system uses virtual domains and users (and aliases) in a MySQL database. Hence it is highly expandable, and can host as many users and different domain names as you wish (well as much as your hardware will allow).

For those who peruse the Mandriva Club Forums, you will have seen this howto appear before, however it has now been updated & tested to work with Mandriva Linux 2007. (see my release candidate review here)

I’m looking for others to increase the scope and functionality of the howto so feel free to register on the wiki and translate the document or add other modules such as spamassasin, clamav etc.

The whole basis of this system is to be highly flexible, and as such I am developing an AJAX/PHP web application for managing this setup. Simply enough it’s called CyprixMail.

If anyone is interested in helping with development of CyprixMail, comment here or send an email to cyprixmail[at]cyprix[dot]com[dot]au. Hopefully the first alpha release will be open to the public soonish. License will be GPLv2.