« swipe left for tags/categories
swipe right to go back »
Every major software or web company I’ve ever been involved in has had a catastrophic outage of some sort. I view it as a rite of passage – when this happens when your company is young and no one notices, it gives you a chance to get better. But eventually you’ll have one when you are big enough for people to notice. How you handle it and what you learn from speaks volumes about your future.
Last week, two companies that we are investors in had shitty experiences. SendGrid‘s was short – it only lasted a few hours – and was quickly diagnosed. BigDoor‘s was longer and took several days to repair and get things back to a stable state. Both companies handled their problems with grace and transparency – announcing that all was back to normal with a blog post describing in detail what happened.
While you never ever want something like this to happen, it’s inevitable. I’m very proud of how both BigDoor and SendGrid handled their respective outages and know that they’ve each learned a lot – both in how to communicate about what happened as well as insuring that this particular type of outage won’t happen again.
In both cases, they ended up with 100% system recovery. In addition, each company took responsibility for the problem and didn’t shift the blame to a particular person. I’m especially impressed how my friends at BigDoor processed this as the root cause of the problem was caused by a new employee. They explain this in detail in their post and end with the following:
“Yes, this employee is still with us, and here’s why: when exceptions like this occur, what’s important is how we react to the crisis, accountability, and how hard we drive to quickly resolve things in the best way possible for our customers. I’m incredibly impressed with how this individual reacted throughout, and my theory is that they’ll become one of our legendary stars in years to come.”
I still remember the first time I was ever involved in a catastrophic data loss. I was 17 and working at Petcom, my first real programming job. It was late on a Friday night and I got a call from a Petcom customer. I was the only person around so I answered the phone. The person was panicked – their hard drive had lost all of its data (it was an Apple III ProFile hard drive – probably 5 MB). The person was the accounting manager and they were trying to run some process but couldn’t get anything to work. I remember discerning that it seemed like the hard drive was fine but she had deleted all of her data. Fortunately, Petcom was obsessive about backups and made all of their clients buy a tape drive – in this case, one from Tallgrass (I vaguely remember that they were in Overland Park, KS – I can’t figure out why I remember that.)
After determining the tape drive software was working and was available, I started walking the person through restoring her data. She was talking out loud as she brought up the tape drive menu and starting clicking on keys before I had a chance to say anything at which point she pressed the key to format the tape that was in the drive. I sat in shock for a second and asked her if she had another backup tape. She told me that she didn’t – this was the only one she ever used. I asked her what it said on the screen. She said something like “formatting tape.” I asked again if there was another backup tape. Nope. I told her that I thought she had just overwritten her only backup. Now, in addition to having deleted all of her data, she had wiped out her backup. We spent a little more time trying to figure this out, at which point she started crying. I doubt she realized she was talking to a 17 year old. She eventually calmed down but neither of knew what to do next. Eventually the call ended and I went into the bathroom and threw up.
I eventually got in touch with the owner of Petcom (Chris) at his house who told me to go home and not to worry about it, they’d figure it out over the weekend. I can’t remember the resolution, but I think Chris had a backup for the client from the previous month so they only lost a month or so worth of data. But that evening made an incredible impression on me. Yes, I finished the evening with at least one illegal drink (since the drinking age at the time in Texas was 18.)
It’s 28 years later and computers still crash, backups are still not 100% failsafe, and the stress of massive system failure still causes people to go in the bathroom and throw up. It’s just part of how this works. So, before you end up in pain, I encourage you to think hard about your existing backup, failover, and disaster recovery approaches. And, when the unexpected, not anticipated, not accounted for thing happens, make sure you communicate continually and clearly what is going on, no matter how painful it might be.
Ever since I switched to the Mac, I’ve had N (where N is a suitably large number) tell me that I should switch to Gmail from Exchange. I finally decided to try it for a week and see if it works for me. Given my Mac experience – where I had to commit and really use it, I’ve decided to do the same on Gmail.
For now, I’m just going to use Gmail (instead of Google Apps) because I don’t want to go through the hell of switching the feld.com domain since I’ve got a bunch of other people (e.g. my family members) on it in a variety of configurations. That’ll limit me a little as I won’t be able to use the Apps Marketplace, but the benefit is I’ll be able to mess around with a variety of other Gmail stuff.
If you’ve got Gmail addons, hints, tips, and trick, leave them for me here. At the end of next week, I’ll either be switching to Gmail or heading back to Mac Mail against my Exchange server.
This morning, as I was waiting for my laptop to grind through its startup process I started wondering why I had a laptop. I travel a lot and had it with me in San Francisco and Los Angeles this week, but hardly used it. And, when I did, I was frustrated with how long I had to wait for it to “get started”.
Today, while I was waiting for my laptop to sync email (Outlook 2010) I grabbed my iPad, opened mail, and read/reply/deleted all of the email that came in over night. I was finished processing the email before my laptop was ready to be used.
I had this same experience yesterday morning in LA. Except then I processed all of my overnight email on my HTC EVO phone which was also acting as the hotspot for my laptop to connect. And, throughout the day, I just did email on my phone instead of firing up my laptop.
The only time I used my laptop last week was a three+ hour stretch in San Francisco when I was at First Round Capital’s office (thanks Josh for the use of your desk) in between meetings. I had turned on my laptop at 8:45am when I got to FRC’s office, did a board meeting from 9am to 12 (the laptop was in a different room), and then used my laptop from noon until I left around 3:30. By noon it had fully synched itself.
As I write this, I realize that Android and Apple both sync faster with my email on an Exchange data store than my Windows 7 laptop with Outlook. A lot faster. It doesn’t seem to matter whether I’m connecting over 3G or Wifi – my Android phone, iPad, and iPhone are ready to go right away whereas my laptop takes anywhere from 5 to 15 minutes to get into a fully usable state (where the disk doesn’t spin an slow things down, or Outlook is non-responsive, or something else funky is going on.) I’m on a Lenovo X300 with 4GB of RAM so it’s not the hardware.
I wrote this post on my iPad using the cute little iPad keyboard doc. It appears my laptop is once again useable, but it’s probably too late for me this morning. Time for a run.
Google gave all 5000 Google I/O attendees an HTC EVO (I guess it’s a Sprint EVO) running Android. For the past two years I’ve been using an iPhone and have become increasingly disgusted by AT&T’s service which is horrible (and deteriorating) in the cities I frequent – most notably Boulder, San Francisco, Los Angeles, Seattle, NY, and Boston. So – I decided to give the EVO+Android a real shot and use it for a week as my permanent phone.
When I wrote my post Open Android vs. Closed iPhone right after Google I/O a few folks took shots at me for pimping a free phone that I got at a conference. Given the amount of money I regularly shell out to screw around on hardware and software (I’m one of those guys who happily buys things just to try them out) I shrugged this off but figured it was worth pre-empting since I’m sure this nonsense will come around again. So – there’s the disclaimer – I got this phone for free (although I did sit on two panels and spent a day and a half talking to people at Google I/O.)
While there has been plenty of fan boy and anti-fan boy chatter about this phone, I can only find one thing to complain about – the battery life. It’s still running Android 2.1 so I expect there will be plenty of battery tune up in Android 2.2, but out of the box the battery only lasts about six hours. I’ve tuned my settings so I can get a full day out of it, but am still carrying my USB cord to grab some juice from time to time. There a few tricks (like charge it with it turned off) that help a lot, but it feels like the iPhone 3G did when it first came out where I was always paying attention to how much charge I had left. Fortunately this will get better with software (quickly) and – since the battery is removable, I can just carry a spare around.
Ok – that’s literally the only thing I don’t like. The screen is phenomenal. All of the apps I run on my iPhone are available on Android – I even found a few new ones. The camera is killer. The email client is much better than the iPhone. Search for anything is lightening fast. Voice recognition – er – recognizes my voice. I have a phone that tethers and – if I want – I have a hotspot (bye bye MiFi.) My applications remember their state and come up instantly because they are still running in the background. The browser is fast. Google Maps + Navigation is incredible, especially for someone who can’t read a map to save his life. I can dial a phone number, look up an address, and get directions from within the calendar. The weather app knows where I am. Google Voice works great and is tightly integrated.
And – for the payoff – I can make a fucking telephone call on this thing. I can’t remember the last time I looked back after a day and thought “wow – I didn’t drop a single call today.” Now the only dropped calls I’ve had are when I’m talking to someone on an iPhone and they drop.
I’m looking forward to iPhone 4.0 coming out so I can see how it compares. My guess is that I’ll get the Android 2.2 upgrade at about the same time so I’ll have both to play around with in June and July. The real result will be to see which phone I’m using when I get back from Alaska in August. In the mean time, the HTC EVO is a winner and – as a result – the smart phone thing is going to get interesting now that Apple has some real competition and can no longer just walk all over Microsoft and Palm.
Did I mention that I can’t wait to get my hands on an Android Tablet?
I’m at the Glue Conference all day. So far, it’s far exceeded my already high expectations. I’m now sitting in the API track and the first two presentations have been dynamite. Clay Loveless from Mashery just did a presentation titled “5 Things I Hate About Your API-TOS“. He nailed it. Here are his top five (most important last), along with some commentary from me.
For simplicity, I’ll call the company providing the API’s the “platform company” and the companies using the API as the “ecosystem partners.” Also – I’m not picking sides here as I’m an investor in both “platform companies” and “ecosystem partners”. Rather, I’m just trying to summarize Clay’s points, bring out a few ideas, and give you a sense of the kind of stuff we are talking about at Glue.
5. Do You Think My Code is Yours? While it may seem like a stretch that a platform company trying to create an ecosystem would try to assert this, the phrase “derivative rights” appears in a surprising number of platform company API’s. And I’ve run into people that actually believe they own the code (or rights to the code) developed by their ecosystem partners. The only thing I can say to this one is “be careful and don’t accept absurd assertions.”
4. It’s Just Tooooooo Loooooong. This one is related to the next one, but it’s what happens when the lawyers take over. See #3.
3. It’s Written in Legalese, But I Speak Geek. Thanks for the 14 page TOS – now what the fuck does it mean? Give me a one page summary in plain English and bullet points. Be “ecosystem friendly” – all the time. Don’t bury the lead on page 11. Just tell me the rules so I can play by them.
2. Commercial Use OK Or Not? I’m seeing this become increasingly contentious between some platform companies and their ecosystem partners. Until the platform company is successful, this is a mellow and happy situation. Once the platform company becomes successful, often in part to the adoption of their API by their ecosystem partners, the platform company starts trying to split out commercial and non-commercial use, at least in certain areas. If you are an ecosystem partner and you think this evolution should be against the rules, just check page 10 of the TOS (per point #4) where it says “Company reserves the right to change any aspect of the TOS at any time in the future.”
1. TOS != Product Roadmap Communication Platform. As an ecosystem partner, you should assume the platform company will change its roadmap over time to support its business goals. It can be painful when this happens in the context of a TOS change, although I think there are some cases where the platform company just has to say “ok – here’s how we are going to do things going forward – deal with it.” The solution to this one is clear and open bi-directional communication – as long as there is trust and no one is trying to hide the ball or do things that are clearly “over the line” in terms of the TOS, these situations are usually quickly resolvable with an appropriate commercial agreement.
Oh – and if you want to run Java on an Apple IIc, here’s how you do it.