First, couched in terms of the technology product adoption lifecycle the ‘acceptable’ downtime depends upon where you are on the curve. “Innovators” will put up with a certain amount of crashes and failures. Early adopters have less tolerance. The pain comes as you cross the chasm. The early majority simply wont ‘buy’ a service/product unless they believe its stable and reliable. Downtime at this point could kill the market. So the further you are along the curve the more you ought to invest in resiliance of the systems.

The second stream of thought is a consideration of how mission critical the ‘service’ is to end users and how ‘real time’ their requirements are. A system dealing with truck appointments and gate receipts at a container terminal (oneport.com) is going to need to be very resiliant as delays can cause major operational impact (and many dollars in consequential losses).

On the other hand a tshirt website has less need to be resiliant since failure is merely inconvenient and may cause the site loss of revenue, but not the customers. The tradeoff is loss of reputation, negative word of mouth.

Its good advice to develop practical and simple work around support if the system fails. This isnt feasible for all online services but if you can switch to a ‘manual’ mode to maintain service then customers will prefer that to being totally down.

This might mean creating static mailto HTML forms and having to maintain manual data entry and manual email response on the backend (we spoofed an auction in Japan this way for a week until a critical bug was resolved) or it might mean fail over to a professional (outsourced) customer service hotline (costs being kept down by it only being used in emergencies).

In a bootstrapping start-up role Ive had to trade off the risk of system outage (or overload at peak times) against the desire (and need) to grow the customer base. This was a prepaid calling card business in Hong Kong. We managed some of the loading by influencing customer behaviour through our pricing strategies. We didnt have capital to build a fully resiliant platform.

Even where we invested in resiliance there were still problems. We hosted our services in a state of the art telco facility and bought capacity from world leading telco providers. The FM building lost all 3 redundant aircon systems in a typhoon and their key switching systems overheated and shut down the facility. A separate software upgrade on a providers network tripped a major outage on the routers which rippled around the network and caused sporadic failure for a week. Even when you pay for the so called best you can face downtime.

Ive also been involved in projects where a Systems Architect has gone mad spending millions of dollars on redundant infrastructure way ahead of progress on customer adoption. When customer numbers / transaction volumes are lower the work arounds are obviously easier. As the numbers grow it becomes more necessary to build greater resiliance in the systems.

To assist in assessing how much to invest and when, Id recommend laying out a matrix of risk of failure, investment cost of redundancy or resiliance, risk/cost to customer/business, time to recovery.

Brad Feld, a Colorado-based VC, wrote an insightful piece on acceptable downtime for rapidly growing companies. It’s rarely the case, however, that executive management gets full disclosure on what root causes are responsible for the embarrasin…

Brad Feld wrote a post yesterday titled “Acceptable Downtime”, where he explains that he has a position on the board of a startup which is considering adding redundancy to their web based service to mitigate the possibility of a catastrophi…

Brad Feld wrote a post yesterday titled “Acceptable Downtime”, where he explains that he has a position on the board of a startup which is considering adding redundancy to their web based service to mitigate the possibility of a catastrophi…

2) I thought hardware and open-source software were supposed to be so cheap to be essentially free? At least that’s the hype or “spin”. If so, what excuse is there for a lack of 400% redundancy? Every business and technology plan should have a special section entitled “How Scalable are we Really?”.

3) If the business plan for a VC-funded venture targets major accounts which have critical uptime requirements, isn’t it the responsibility of the entrepreneur to gain full funding for the appropriate level of redundancy and fallback to meet targeted customer requirements? If so, what’s the issue here? A failure of the entrepreneur to do robust planning? A failure of due diligence by the VCs?

4) The important thing is that entrepreneur be up-front and spin-free with customers as to what levels of service they can expect and what levels of “drama” are likely.

– Jack Krupansky

Sometimes all this means is having good enough monitoring that you can graceully take the system down when it starts to fault, and put up a polite apology on the website while this is happening. This turns an “unexpected outage” into a “planned outage”.

Of course, if you have “planned outages” too often, your customers have good reason to be unhappy.

It turns out after doing a lot or research, our customers will accept a read only version while we get the other system back online. While we have never been down for very long, it is better to have someting up than nothing. Also communication is important. When ever we have had a problem, we shoot an email to all our administrators and call on the phone our major customers. The worst thing for our customers is to not be able to log into the system and not know what is going on.

To put it in VC terms, it is ALWAYS better to tell bad new right away to your customers, board, investors, than to wait and hope.

http://www.codesta.com/knowledge/management/uptime%5Frealities/

Any downtime in the online channel is unacceptable. This is the equivalent of putting a CLOSED sign in your window.

Do you accept downtime in your real-world channel? If a customer is in your store, with a product in hand, and cash at the ready…do you say sorry?

We are 10 years into online commerce. Firms have to stop factoring in downtime. It indicates that companies still think like old-school mainframe folks.

Stop. The. Madness.

smp

It seems like systems engineers and software architects usually want to design elegant, hi-performance s/w, and marketing people always want feature-rich applications so who is there to advocate for reliability?

-mjr