The following is a brief timeline as to some unfortunate events which transpired over the last several days.

Initially covered by the Netcraft Team one of our respected competitors fell victim to a zero day attack across their server fleet.

The one item in common across these servers? Each was running the cPanel control panel. Netcraft then followed up with coverage linking the massive redirection based attack to an undiscovered security vulnerability in the cpanel.net software.

Throughout a series of conversations across the hosting community, details gradually came to bear regarding the compromise. How it occured, and how it could have been prevented.

After a few days of my personal reading and discussions with various owners of web hosting operations across the industry, including those affected by this specific vulnerability, questions remain as to the underlying security of the cpanel control panel.

Following a great deal of public pressure, some of which was issued by Network Redux, a formal advisory was issued earlier in the evening from cPanel, Inc. The disclosure briefly covers the focus of the attack, and the specific files which were used as a gateway to perform this exploit.

For those unfamiliar with the internal cpanel operations, a majority of tasks performed by the cPanel (WHM) management software occur via perl scripts which pass requests through a program called a setuid wrapper. The purpose of this wrapper is to allow system calls to take place which require root privileges. The wrapper is owned by the root user, and holds a setuid bit so that as it runs, it has the ability to make privileged calls that only the root user would be allowed to make.

There were two underlying issues to this entire disaster which I will discuss briefly.

First and foremost, the Internet Explorer VML vulnerability which was not addressed by Microsoft in a timely fashion. This is the vulnerability that was ultimately being exploited during the attack. The hosted domains were being infected with iframe congtent that would redirect IE users to URLs containing infectious code, in turn infecting their machines. All that can be said is that it is not surprising to have Microsoft sit on a security vulnerability, leaving remote attackers plenty of time to infect thousands of vulnerable systems.

Secondly, the undiscovered cpanel vulnerability appears to have been a larger set of issues previously forwarded to cPanel by Tim Greer, a system administrator with Hostgator. Tim has been outspoken in his disdain for cpanel’s approach to security, understandably in these regards.

I commend Tim Greer, Hostgator, and Bluehost for their time and effort in handling this matter, it was their joint effort, including outside consultants, which led to the findings, which were in turn forwarded to cPanel’s development team. It does seem rather clear (given this level of compromise) that there are underlying security concerns with cPanel’s usage of setuid c binaries to conduct system operations, several of which have been confirmed to provide root level escalation as discussed by Matt Heaton of BlueHost.

It is disgusting to see a reputable company with high levels of customer satisfication and a good security track record such as hostgator, fall victim to such an incident.

Had Microsoft been more proactive on yet another security issue related to Internet Explorer, would this issue have occured? Probably not, however, all indication shows that the Hostgator systems were compromised a month prior, and the attackers were simply looking for the right window of time to take advantage of millions of Microsoft PCs running vulnerable versions of the popular IE browser.

The facts will continue to flood the discussion forums over the next several weeks. And we will be one of the many providers questioning cpanel on their ability to provide a secure platform, as well as requesting independent, third party security audits of their closed codebase.

My best regards goes out to all those affected by this situation, particularly the hostgator team, which will remain on our list as one of our most respected competitors.

Just over a year ago I took a close look at the CentOS distribution when I was weighing the $349/year price tag per server we were paying to the Red Hat group for basic installation support.

As a system administrator, I’ve taken a strong liking to the CentOS community, and will attempt to explain the differences between the two distributions, and clear up some common misconceptions.

First and foremost, CentOS is Red Hat. It is a fully functioning clone of the Red Hat AS package, which is Red Hat’s most advanced distribution in terms of CPU, Memory and platform support.

Despite public display by the CentOS community: http://www.centos.org/modules/smartfaq/faq.php?faqid=13 — it is a fully operational clone of the Red Hat distribution based on SRPMs (source RPMs) which red hat is required to make publicly available.

There are a couple of immediate drawbacks to a clone distribution:

1. It isn’t backed by a commercial entity such as Red Hat, Inc.
2. Updates commence after Red Hat releases updates, which gives you a period of lag on any updates Red Hat may push via the Red Hat up2date Network.

On the first issue, a community project with the size and underlying support of CentOS has just about the same survival rate (in my opinion) as a publicly traded company such as Red Hat.

On the second issue, the CentOS group has been on top of critical updates released by Red Hat, and has not warranted any scrutiny from our organization on this topic to date.

—

When you are managing dozens of linux servers, I have found the Red Hat Network to be more burdensome than useful. Granted, there are more sophisticated packages and price points to engage, however, at the $349/year price tag you are not even entitled to automated pushed updates from their services, rather you must manually (or via cron) engage the up2date system. The technical support provided is only “installation assistance” which is relatively useless when you are on your 10+ license.

There is also the burden of managing licensing, which takes me back to my days of managing Microsoft Windows Servers. So much time can be wasted on licensing, provisioning, and verifying system license integrity, that any system which avoids licensing tripwires is one to favor.

If you are interested in additional reading material on CentOS I strongly recommend the centos.org website, as well as downloading the distribution and giving it a go. For those of you used to Red Hat systems, other than obvious differences within a few configuration files such as /etc/redhat-release, and package nomenclature, you will feel right at home.

Seldom do web hosting providers offer the nitty gritty on the buying process for a new build. Lets attempt to rectify this.

The following areas are of concern when buying a production server for a hosting environment:

- Vendor
- Processors
- Memory
- Disk Drives
- Power
- Support

This article will attempt to address all aspects, from a Network Redux purchasing perspective.

Dell is our vendor of choice, largely based on consistency with price and support with the 2U PowerEdge 2850 model. Two other serious players to consider are IBM and HP, both of which offer comparable level x86 infrastructure at a moderately similar price level. Sun gets the boot, their VAR model leaves little to be desired and their inability to support 146GB SCSI drives in mid range servers makes them impossible to work with from a single node model (one server handling data, sql, mail for a handful of customers).

There are two methods of purchasing a server through Dell as a small business customer. If you have an established track record of purchases you will be entitled to a premier account, which allows you an online portal where you can build/spec servers and peripheral equipment. Builds are typically priced with a 10-15% markdown, and only usually comes in handy with peripheral retail purchases (APC, Belkin, etc.).

Your best bet in purchasing Dell gear is through their small business center on their dell.com website. Significant discounts are noted during end of quarter periods, usually in the 34% range.

This being said, lets look at a typical Network Redux PowerEdge 2850 build and discuss the outlined choices.

Our first concern is processing power. Though not typically the bottleneck in a shared environment, dual processing capabilities are very important. Our normal choices are Dual Xeon 2.8Ghz/2MB or 3.0Ghz/2MB processors, staying away from the Pentium D (dual core) processor which yields less than adequate performance results in comparison to the two processor Intel model. Unfortunately we can only talk intel at this time, as Dell does not offer an AMD option. With the birth of the 9th generate PowerEdge server (1950, 1955, and 2950) new processing technologies from Intel (microarchitecture) have replaced older technologies (netburst), and AMD options are expected to be released later this year.

Our next consideration is physical memory. The more the better. Our recent builds are equipped with 4GB of physical memory, which we’ve determined to be an appropriate fit based on experience with shared/reseller environments. This is DDR2 400Mhz memory, future builds of the PowerEdge 2950 will contain DDR2 533Mhz Fully Buffered Dimms which we will discuss at a later date.

Perhaps the largest bottleneck of a shared system is disk I/O. The fatal mistake a majority of providers make is the investment in SATA disk technology. The alternative?… SCSI, which holds a substantial performance and reliability standard. This discussion can become complex, a great link for discussion can be found at: Tech Target

With disk comes the need for RAID, which, unfortunately, is rarely seen in shared hosting environments. Our RAID of choice is RAID 5, which is the best balance for 4 disks in terms of read/write performance and data integrity. One physical drive may fail and operations can sustain themselves (though in a degraded state). Hot swappable disks are of utmost importance, allowing an offline disk to be ejected and replaced during operations, and the new disk to be built into the array.

In terms of power, we build our servers with dual, redundant hot swappable power supplies. This is typically 2 550 watt power supplies, one acting as primary and the other as standby. A single power management unit handles dispering power during failure of the primary, so that the server stays online during a failure of the primary power unit.

Lastly, support contracts are imperative. Keeping spares and on-call technicians is a must, however, but when it comes to replacing a series of components at 3AM, you must rely on 24/7/365 4 hour onsite support contracts. This being said, our servers maintain Silver/Gold contracts with Dell support for 4 hour onsite assistance around the clock, which includes both parts and replacement assistance.

This concludes are discussion on a standard Network Redux server build. I will continue to post articles on various technologies which power our internal operations.

An update to this ongoing debacle with the fine folks operating the UCEProtect-Network-Blacklist-Based-On-Spite real time blacklist.

Time Warner Telecom’s Internet Security group provided us with some useful information on this group. From these conversations the following has been determined:

1. UCEProtect-Network is not known to be a widely used blacklist in the U.S.

2. UCEProtect-Network claims a 7 day waiting period for removal, which we will be waiting to expire. The problem with this situation is that there is no indication as to when the listing began (we know it began after our post on their extortion practices on August 31st).

3. This is the unfortunate side affect of black lists which are operated independently, and actions such as these simply diminish their validity.

I personally plan to track this matter for the next several weeks. Legal options will also continue to be entertained as this matter progresses, or regresses.
I am also still waiting on a response from the UCEProtect-Network as to justification for this “Level 3″ blacklisting.

Stay tuned for additional updates.

This will be short for the time being, as more information is gathered.

The following was brought to my attention earlier today:  http://www.mxtoolbox.com/blacklists.aspx?IP=64.128.80.5

Of ~150 blacklists, there is simply no coincidence that the single blacklist I spoke openly about here on August 31st has performed a “Level 3 ISP” outright ban on an entire Class C (256 Addresses) network of ours in the last few days.

I’ll be providing more updates on this unjustifiable retaliatory effort by the UCEProtect Network as we gather additional information.