freeversing

A colleague/client and a new team member recently inspired the following.

2009 was an incredible year for Redux. Continued growth, despite economic conditions and existing enterprise customers tightening their belts on Managed Hosting Services.

Several firsts worth mentioning during 2009 and heading into 2010:

Large scale single customer storage deployment (60TB)
Client MySQL configuration with a 64GB Innodb buffer
Production deployment of a Force10 Switching Fabric
Non-human team members
Creative Director
Super Network Redux Man Episodes 1 and 2
OpenSourceBridge PDX – Sponsors, speaker and hosting
Wet Servers
Discovered Otis Taylor
Turned our Datacenters into a private art gallery, gorgeous cabling by Cameron Smith, System Engineer and Operations lead.

Already pushing strong into 2010, we’ve made numerous changes to our public site. Most notably enterprise/complex hosting finally being on the table for discussion. We’ve been providing it in a variety of forms for close to 6 years, and now are talking about it publicly.

I couldn’t be happier with the Team and Infrastructure we’ve put together. Looking forward to a new round of firsts in the months to come.

Sometimes a little bit of public humiliation is required.

Most, if not all of organizations purchasing Dell Servers in the past 3 months have been stunned by incredible delays in production. We have seen on average 4-5 weeks from purchase time to delivery.

Within the realm of economic uncertainty delays in production are understandable. Dell has long maintained a successful model of keeping the right amount of inventory on hand. Particularly for agile organizations such as ours, who buy a range of server and storage products depending on client and contract demand. What is not acceptable is a failure to resolve their own vendor relationships.

Here comes Ceva Logistics. Seemingly Dell’s preferred (perhaps only) freight carrier of choice. When we have orders exceeding a certain weight they will be transported via Ceva (formerly Eagle).

We generally need our equipment within a few days time frame so we always tack on Next Business Day shipping. Here are our last two experiences:

In November we placed an order for 10 PowerEdge R610 servers. 21 days later, a result of production delays, our servers were out for delivery. Based on our correspondence with Dell, these servers were ready for pickup from Ceva on the Friday morning in question. Next Business Day had been paid for, in turn we expected these servers the following Monday. Ceva’s tracking logistics indicated the same.

Monday arrived, the servers were not delivered. Explanation being that Ceva had picked the servers up from Dell past their 6:00PM cutoff time (6:30PM), and therefore did not consider Monday to be Next Business Day. Fine, we asked if we could go to their warehouse and pick them up ourselves. Not possible they said, the servers were not in Portland, they were still in Austin.

Ok fine. We expected our servers the following day (Tuesday). 3:00PM passed and no servers had arrived. We called Ceva to find that they had no intention of delivering the servers that day. Their notes indicated they only had one pallet of the shipment and were waiting on the second. “If it is OK we will deliver them tomorrow (Wednesday),” the Ceva representative told me. Absolutely impatient beyond belief I told them no and asked them to deliver what they had. After an hour of investigation, the Ceva representative helping us determined that even though one pallet was present and their systems indicated half of the order ready, ALL of the servers were on that pallet and sitting in Portland. It was 4:30PM, they were willing to ship them but it would be later that evening. 6:30PM arrived and so did our servers, in an unmarked van to the side of our Datacenter. It was pouring down rain. We unloaded the servers with the help of the driver, who then (we can only assume) wasn’t fond of the rain and considered it too wet to help us get them to the door and up the elevator — we handled it ourselves.

Dell ended up refunding our large Next Business Day shipping payment with extended apologies for the moronic display of customer service by Ceva.

Production delays continued (and continue) to plague us throughout December and January. On January 8th we placed an order for 60TB of storage units, we were given a shorter lead time of 17 days. Storage units not being on a significant delay cycle, they were out for delivery on the 19th, to be delivered on the 20th (today).

When we arrived to our datacenter, operation managers informed us that they were concerned with the shipment Ceva delivered. The side of the boxes were WET and the Ceva delivery person acknowledged this.

We unpacked two of the units and found water condensation on the chassis itself:

Water had seeped through the first two boxes and managed to get onto one of the units.

If any Sun or HP Vars are reading this, feel free to give us a call. We’re still waiting for Dell customer service to respond to our reps expedited request for assistance.

For a service provider, delay costs are tremendous. Each day we cannot deploy is a day of lost revenue. Now we add servers that are arriving with water on them, and we have a recipe for vendor change.

Perhaps this is specific to Ceva logistics in Portland, however we’ve been down this road before with their prior usage of DHL. I never wrote about that experience, but the end result was Dell dropping DHL as a carrier.

Sometimes it takes a little bit of public humiliation and customer venting to get a problem solved.

Hopefully someone will pick this up and give us a hand.

-Thomas

I’m writing this post because Google Docs is in read only mode:

I was in need of writing a job description on a saturday afternoon, and google docs has failed me. Cloud service providers are going through some growing pains. Our Network Operations were under significant pressure over the last two weeks as we dealt with our own slew of issues.

Google’s email outages this week, followed by a weekend Google Docs outage will surely enrage a large percentage of google users. I’ll say it is a bit annoying, however as the owner of a cloud service focused entity I can understand the frustrations of managing complicated computing environments. I can’t begin to comprehend the level of complexity associated with their operations, which is why I will turn to Office for the mac while I am waiting this one out.

A pair of talented google researchers discovered a flaw in the linux kernel affecting 2.4 and 2.6 trees for all architectures.

This was of significant consequence this week for our organization as we prepared and then proceeded to patch a large amount of systems.

Mass defacers are quickly taking advantage of unpatched systems at a rate quicker than we’ve ever witnessed as a web hosting provider — quicker than distribution vendors such as red hat were able to audit, test and release their packaged kernels.

These experiences can show both the strengths and weaknesses of an open source foundation, which can in some cases feel like the wild west of early American history.

A recent article speaks for itself.

As a long time InfoWorld.com reader when it was print, then converged to online, this is about as bad as IT journalism can get.

*Delete* from the bookmarks.

For those of you who spend time around me, you are used to the REI water bottle I carry around 24/7. A few weeks ago I was using an older bottle and it made a mess in my bag, destroying my macbook pro.

My options were to wait 2-4 weeks for it to dry out and hopefully start back up, pay a $1250 flat rate water repair fee to Apple, or buy a new unit.

Recently we’ve been huge fans of the Apple refurbished store. Brand new units discounted as high as 30%. I was able to locate a MBP / 2GB / 128GB SSD for an extremely reasonable price point. I had never used an SSD drive in a desktop and figured it would be a good change, plus I could take my 4GB memory kit out of my failed MBP and put it in the new unit. (An extra bonus was the refurb unit was mislabeled and actually came with 4GB).

To keep it short and sweet, I’ll never go back to a desktop that isn’t running an SSD drive. What a night and day experience, particularly when needing to cold boot Parallels/Windows for those apps that won’t run on OSX.

A few days ago the failed MBP had reached its 4 week point and I tried booting it up again — to my surprise all is well, just needs a new battery.

As the months and years progress SSD will likely be the defacto standard for mobile computing based on my initial experiences.

Pre-rant — why does the Gmail app have a better UI than the built in mail reader on blackberry devices? How is a third party capable of levering your development kits better than you, the designer of the platform, RIM?

Apple just released a discounted iPhone to lure unsuspecting users of reliable telephone service to their services. This is unfair and should be stopped. It can only be stopped at the source.

I’m an avid mac fan. Apple does quite a bit right. They are innovators and leaders in this current generation of technology. One of their brilliant competitive advantages is their ability to strongly leverage their innovations which are routinely and consistently superior to everything else on the market.

I refuse to go back to ATT mobile services because I cannot tolerate dropped phone calls several times a day. It would not matter if the iPhone was being provided free of charge on a month to month contract, I will not use ATT services. Speaking as a first generation iPhone user, I would rather struggle through the nuances of syncing my blackberry to my mac than put up with ATT. And yes, it is annoying to be hanging out with my friends while they navigate aircraft to landing on their phone while I’m barely able to pull up a web page.

There’s nothing new or fresh with regards to my ATT issues. When I switched to Verizon, all of the previous dropped call or terrible service areas went away.

Portlanders and those of you nearby, I’d like to introduce a conference coming up called Open Source Bridge.

If you are a Network Redux customer we can issue a $100 discount of the $250 registration. As one of the three sponsoring groups with commiter status we also have a couple of free passes reserved for the first two customers of ours to ask.

I will be speaking at a business focused seminar on the 17th called Bridging the Developer and the Datacenter. Huge bonus points for the Network Redux customers who attend the session!

There is quite a large array of fantastic sessions to attend. The first two customers to inquire will get the free passes!

Cheers

Thomas

A couple app failures from yesterday. The Dell one gets old, they can make it difficult to buy product sometimes. The openoffice one wasn’t as annoying in that I rarely use it. Admittingly I prefer Microsoft Office 2008 for the Mac…

Perusing an article discussing a civil suite against facebook by a Florida librarian and privacy advocate and came across the following:

Facebook spokesman Barry Schnitt wrote in an e-mail to CNET News. “He’s not going to get it but we promise to refund all the money he paid to use Facebook. Seriously, we’re glad to know how important Facebook is to Mr. Karantsalis but his account was not disabled, is currently active, and he is using it, so I’m not sure what the problem is.”

Bonus points to Barry for sliding in the reference to free services.

Again an excellent reminder: all of these services are provided free of charge.