« FriendFeed Adds IntenseDebate and BrightKite | The Bear Blogs »

I’m Done With Private Beta

I was going to call this post "Private Beta is Bullshit" but then I realized I might be wrong. Rather than decide, I’m looking for reasons to change my mind. Please help me. In the spirit of thinking out loud on my blog, I’m going to go through a history lesson from my perspective to frame the problem.

When I started writing commercial software in the 1980′s, there was a well-defined "beta process." Your first beta was actually called an alpha – when you had your friends and a few lead users play around with your stuff which was guaranteed to break every few minutes. But they were a good source of much needed early feedback and testing. Then came beta – you shipped your disks out to a wider audience, including a bunch of people you didn’t know but who were interested in your product, and they banged away looking for bugs. You had a bug collecting and management process (if you were really cutting edge you even had a BBS for this) and while there wasn’t a code freeze, you spent all of your time fixing bugs and hardening / optimizing the code. If you had a complex system, you started shipping release candidates (RCs); less complex went straight to a release (GA). Inevitably some bugs were found and a bug fix version (v x.01) was released within a week or two. At this point you started working on the next version (v x+1.0); if there were meaningful bugs still in v x you did small incremental releases (v x.02) on the previous code base.

This approach worked well when you shipped bits on disks. The rise of the commercial Internet didn’t change this approach much other than ultimately eliminate the need to ship disks as your users could now download the bits directly.

The rise of the web and web-based applications in the later 1990′s (1997 on) did change this as it was now trivial to "push" a new version of your app to the web site. Some companies, especially popular consumer sites and large commercial sites, did very limited testing internally and relied on their users to help shake down the web app. Others had a beta.website.com version (or equivalent) where limited (and often brave) users played around with the app before it went in production. In all cases, the length of time of the dev/test/production loop varied widely.

At some point, Google popularized the idea of an extended beta. This was a release version that had the beta label on it which is supposed to indicate to people that it’s an early version that is still evolving. Amazingly, some apps like Gmail (or Docs or Calendar), seem to never lose their beta label while others like Reader and Photos have dropped them already. At some point, "beta" stopped really meaning anything other than "we’ve launched and we probably have a lot of bugs still so beware of using us for mission critical stuff."

With the rise of the Web 2.0 apps, beta became the new black and every app launched with a beta label, regardless of its maturity (e.g. a whole bunch of them were alphas.) Here’s where the problem emerged. At some point every beta got reviewed by a series of web sites led by TechCrunch (well – not every one – but the ones that managed to rise above the ever increasing noise.) When they got written up, many of them inevitably ran into The First 25,000 Users Are Irrelevant problem (which builds on Josh Kopelman’s seminal post titled 53,651 – which might be updated to be called 791K.) During this experience, many sites simply crash based on the sudden load as they weren’t built to handle the scale or peak load.

Boom – a new invention occurred. This one is called "private beta" and now means "we are early and buggy and can’t handle your scale, but we want you to try us anyway when we are ready for you." I’ve grown to hate this as it’s really an alpha. For whatever reason, companies are either afraid to call an alpha an alpha or they don’t know what an alpha is. For a web app, operational scale is an important part of the shift from alpha to beta, although as we’ve found with apps like Twitter, users can be incredibly forgiving with scale problems (noisy – but forgiving).

So – why not get rid of the "private beta" label and call all of these things alphas. Alphas can be private – or even public – but they create another emotional and conceptual barrier between "stuff that’s built but not ready for prime time" (alpha), "stuff that getting close but still needs to be pounded on by real users and might go down" (beta), and "stuff that’s released" (v x.0). That seems a lot more clear to me than "private beta", "beta" (which might last forever), and ultimately v x.0.

In the grand scheme of things this might simply end up in "Brad Pet Peeve" land, but recently it’s been bothering me more than my other pet peeves so it feels like there’s something here. Call me out if there isn’t or pile on if there is.

June 12th, 2008 Categories: Computer Industry

Shane

You have to blame Google for this. They've almost single handedly screwed up the term beta. Gmail 'beta' that still asks me to 'invite' anyone I email to join Gmail a year after it's been publicly available to everyone?!? It's a total pet peeve of mine too.

I also think that web 2.0 startups should focus more on finding the right early alpha users and working with them more closely. Directly to Josh's point, the early adopter tech crowd aren't mainstream consumers. It's easy to get these folks to sign up and test your product and give you feedback that… "it should be integrated with Twitter and manage my blog posts, and RSS feeds"…when most of the folks in your target audience have never used any of these.

Maybe someone should create a community ranking service that allows people to quickly vote on the 'status' of an online app. Alpha – Beta – Release – Or Prime Time. Then we could ridicule Google when they call their Prime Time apps Beta, and put most of the Web 2.0 startups back in their place when they call an Alpha product a Beta.

I'll put some code together this weekend, and try to get a Private Beta out by Monday
Don Diegol

Fear of failure – I think I have read about this recently?
http://intensedebate.com/people/todd_sawic49431 todd_sawic49431

Brad –
In my experience with web apps – alpha's are really early stage concepts meant to illustrate a concept or feature in a live web setting. Private beta's are meant as the first working release while public beta's are meant as often short term, fix the bugs and test scaling in public (goog is the one violating this rule not the other way around). And the reality is that public betas are dying in my view so what's really bullshit are public betas as the private beta is to me true to the way the term was used in the good old software days (limited release, constant revisions, etc.)
Kosso

Hi Brad. Phreadz had a 'clandestine alfalfa mode' and we're now in the 'closed beetroot era'

Yes, I'm poking fun at all these 'alpha/beta' sites out there. But I definitely have the need to do this in this way – mainly due to securing the necessary funds to build out the hardware and bandwidth architecture required to reduce the chance of it falling flat on its face if I swung the doors wide open – for more testing.

There are sites out there which are fully open to the whole world, requiring no Wonka ticket invites to join, which still carry the 'Alpha Release' slogan. I'll agree that this irritates me.

My product works. And when I can afford to, I'll be opening it up, for all to use. And there will be no vegetation or vapor in sight.
http://intensedebate.com/people/mikepk mikepk

We had this debate inside Grazr, what does "Beta" mean in the new Web 2.0 world? The truth is, it doesn't mean much. I'm an ex systems / hardware guy and my partner has lots of experience with shipping software during the bits-on-disk days. We put grazr out there as a public alpha initially, then transitioned to beta thinking in the old paradigm of software development. We then removed the beta badge once we thought of it as "release". We even used version numbers when we had major changes, at each major point release we've had a "beta". We think of our current version as a beta of version number 3, but most people don't know what that means. Google set the bar for beta to mean "totally functional except we change things occasionally". Most people looking at web services only have a very general concept of the engineering alpha, beta, release development phases so I suspect that for many, beta just means "it works like gmail".

I had considered writing a blog post on this a long time ago but it never left draft form. The other interesting thing I see is the transition of users mental model of software from something physical. This is directly analogous to the change that occurred for documents on the web. Originally the only mental model was physical paper, so when people had to grapple with the constantly changing, dynamic nature of the web, they resorted to putting "under construction" on every page. Eventually people realized that "under construction" was meaningless, the page is never 'done', and that the medium itself was fluid. I think the same thing is happening with web applications. "Beta" is one way for people to deal with the transition from a static to a dynamic medium for the delivery of functionality.

Another quick point is that I think you're also overlooking what I believe is one of the reasons private betas are in vogue right now, marketing. Creating an artificial sense of scarcity with special passes and only allowing a select few to gain access can generate buzz for a product. Requiring people to ask for "membership to the beta club" also engages them early. Being a data nerd, I'd love to know if this early interaction actually increases the user engagement to the product. Posts like "Anyone have invites for XXX" can in and of themselves pique interest. I think this fad will pass though as people experience private beta fatigue.
http://intensedebate.com/people/kimm_viebro1980 kimm_viebro1980

Thanks for saying it much more succinctly than I did.

Oh, and based on the other comments I see here that also make sense, maybe what needs to go away is the public beta?
http://intensedebate.com/people/bfeld bfeld

from an email comment: "Simple: the CEO/CTO of an early stage startup is far more concerned with Impressing the Board than he is with the quality of the product. I've written a lot of code for startups — you've funded some of them — and these guys are scared to death of telling you and the board that there are too many bugs to release it right now. If you asked about bug find rates and fix rates, bugs found in alpha testing, beta testing, etc., then maybe these guys would do these things. In short, it's your own fault."
http://intensedebate.com/people/shane_jone42421 shane_jone42421

One clear line that can be drawn between Alpha and Beta in my mind is wether or not your data will persist through to the final release. Previous alpha releases that I've worked on have had a policy that all data and accounts will be wiped clean before entering the beta / general release process.

Not a big deal for traditional 'installed' software, but it's an important factor for web services. Has anyone come across a legitimate site that asked people to play around, provide feedback, but don't expect your data to persist past this release…I've never seen it.
Chris Treadaway

I think this is a pet peeve at most Brad…

Alpha means that the app could pretty much be rewritten based on feedback, and it likely doesn't have all necessary features. Closed Beta = testing real life usage in a sandbox. Beta = release.

We've learned a ton in our Closed Beta release of MinutesNotice that, quite frankly, I'm very glad we didn't learn in a Beta. Perception of what we're doing would be awful, we'd probably have been panned already in major blogs, and we'd be sunk.

Someone above nailed it when they said to blame Google. They are to blame, not folks trying to get their apps off the ground.
http://intensedebate.com/people/deva_hazari2084 deva_hazari2084

A lot of great comments already. I wrote a very similar post in 2006 and linked to posts with the same issue from 2004 – this is nothing new and unlikely to go away. I do think that it's two different issues. One is things not really beta being called beta. The other is how to reach users for a real beta in a controlled fashion. More detailed discussion and one potential solution on my blog post on this topic: http://www.emaildashboard.com/2008/06/the-beta-ex…
http://intensedebate.com/people/steve_bergs2127 steve_bergs2127

My experience with small organizations is that they rarely have good metrics. They're too busy doing things like writing code to analyze data like bug find/fix rates.

OTOH, perhaps board members could drive better behavior by insisting on getting the right facts.
owen

it's new for me!
John Furrier

Brad, been a while since i posted a comment on your blog.. this is one sweet post… your'e right on

i'm 42 so i'm with you. there isn't anything wrong with coming out of the closet with 'private beta' which to me is pre alpha. It's an internet platform – hell why not…

I do think private beta confuses the capital markets…

years ago (not to long ago) – you pitched a vc with powerpoint- aka slideware
now – you pitch with private beta

private beta is the the new 'slideware' –

relevance, real customers, scale matter – not many vcs get that…

great post!
http://intensedebate.com/people/dgcohen dgcohen

The real question is whether or not doing a beta "privately" makes any sense. I think it can in certain situations. But for me the peeve is more about startups who use it as a marketing gimmick. getting an "invitation" is supposed to seem really hard. they're trying to make you feel special. the reality is that if almost all (98% of?) private betas were simply open betas, their user counts would be only marginally different. and without all the hoops they make their users jump through. there's no "cool factor" to this unless you really do have a million users on your waiting list. the upside is for startups that do have a long wait list, it can be really useful to ensure you can scale over time, doing so slowly as you let users "in". So, for me the answer is just let users in and stick the beta label on it along with appropriate warnings. then if you really have too much demand before you're ready (and thus are in the 2%), you can start taking names and letting them in slowly. otherwise, you're simply creating an annoyance for your potential users which you might also call the dreaded "friction". but with popular up and coming services like brightkite, it does seem to work well for them. people are seeking the rare invitation (i've sent about 100 out). so like most things in the world, the answer is of course "it depends."
http://intensedebate.com/people/bfeld bfeld

The difference here is the notion that "alpha" is the internal test. I've always used alpha for an external test with a small number of friendly and active users. The internal stuff shouldn't get to be called alpha since the organization is still "self testing" (important, but not in the release cycle).

The more I read the comments, the more I realize this is semantics (as you correctly point out in your first sentence.) My struggle is that there isn't uniform agreement on what each of the alpha / beta / whatever means, so different companies use them different ways. While the notion of an IEEE standard for a naming convention is nonsense, the variance in the different naming usages is making me tired.
http://intensedebate.com/people/arinewman120 arinewman120

Some great insights here. As a company currently in private beta this is topic near and dear to our hearts. At Filtrbox we did a private beta for a number of reasons and it has worked out well for us. A lot of this is semantics, I agree, but there is truth in the core statement.

When Filtrbox began the transition from friends and family testing to opening it up to a broader audience, we were in no way ready for the mass public and our goal wasn't to see how many users we could handle, it was to learn what needed to be improved before going public. We leaned A LOT, without getting slammed or dismissed as a "not ready for primetime" app by the public when we knew we weren't ready. We are very focused on delivering a usable, quality product that solves problems and going right into public mode wasn't not going to align well with that objective. We also did successfully build up a bit of demand, and our early users have been engaged and helpful in testing the product and giving feedback.

When we go public, there will be no "beta" label on the product, and we will stand behind Filtrbox as a commercial application.

Perhaps we should have called it "Alpha" or "preview" to more accurately reflect the app's current state at the time but in the end I feel like we actually evolved from alpha to beta while operating the service under the "private beta" banner and it worked out.

Related: My partner Tom put up a blog post last year about Google's widget funding program…seems even finance is on the beta bandwagon. http://tomchikoore.wordpress.com/2007/06/29/you-a…
http://intensedebate.com/people/dgcohen dgcohen

you don't need to make it private to communicate that though.
http://intensedebate.com/people/bfeld bfeld

Kimm – outstanding thoughts. This is really helpful perspective – there isn't anything here I can argue with.
http://intensedebate.com/people/bfeld bfeld

Yeah – I always confuse ironic and coincidental along with then / than and accept / except.
http://intensedebate.com/people/edward_h50271 edward_h50271

I agree with the posters that think this is as much about marketing as it is about defining a new or real distinction. Private means exclusive. Alpha may be construed a laborious. As you correctly point out, the word beta has lost some of its meaning, or it has evolved to mean something else. Something new was needed.

I love this post though. I studied English instead of something useful. I cringe when I hear people describe something as "ironic," as 90% of the time they are describing something that is "coincidental." While there is nothing nefarious about Private Beta I am sure, changing the meaning of words for the purposes of marketing is dangerous. Ask George Orwell

Amen for being anal.
JungleDave

Well, here's one differentiator in my book – if no issues came up in the private beta I would ship the same code as the public beta. You would never do that from alpha->beta. Of course many folks have abused the "beta" and "private beta" labels, so that when they are used in their traditional sense folks such as yourself aren't sure what to expect.
http://intensedebate.com/people/bfeld bfeld

Dave – while this makes sense, it still feels like it’s alpha land to me. Presumably you are selectively letting in some people you don’t know but have expressed interest. However, you are still throttling (or metering) the invites to control the usage to test for your obvious edge cases.

I guess there an “alpha”, “private beta”, “public beta”, “release” pacing. My struggle is that I’m confused as to how long private betas last and some of them now stretch on for months!
http://intensedebate.com/people/bfeld bfeld

That’s an interesting perspective, especially given the Googleness of “public beta”. Several folks have commented that it’s “public beta” that has gotten perverted by the Google usage – that’s an interesting thing to ponder.
http://intensedebate.com/people/bfeld bfeld

Shan – that’s a good example. If I synthesize several of the comments defending “private beta”, they all come back to the notion that there is a role for something between “alpha” and “public beta”. I still struggle with calling it “private beta”, but as someone who used Xobni during this time period, it was a logical positioning.
David Reinke

At StyleHop we are staying "private alpha" – it actually took a bit of discussion for us to land there since it's not the current convention. Glad to see others are thinking the same way we are.
http://twitter.com/fithappypro @fithappypro

I realize I'm a little late to join this discussion, but I just happened to be googling 'Private Beta' and this post ranked high on the return. As I respect a lot of your work Brad.

I'm just wondering if 'Private Beta' could be a good way to get a little market research done as a small start-up?Even if it does work perfectly, could you not use a 'Private Beta' (or whatever you like to call it) to test the waters before a full launch? It seems to me that it would be which is why I'm leaning towards giving it a try.
http://twitter.com/fithappypro @fithappypro

I realize I'm a little late to join this discussion, but I just happened to be googling 'Private Beta' and this post ranked high on the return. As I respect a lot of your work Brad.

I'm just wondering if 'Private Beta' could be a good way to get a little market research done as a small start-up?Even if it does work perfectly, could you not use a 'Private Beta' (or whatever you like to call it) to test the waters before a full launch? It seems to me that it would be which is why I'm leaning towards giving it a try.

I’m Done With Private Beta

Related posts:

Popular Tags

Categories