Comments on: I’m Done With Private Beta

By: @fithappypro

@fithappypro — Tue, 02 Nov 2010 06:19:04 +0000

I realize I'm a little late to join this discussion, but I just happened to be googling 'Private Beta' and this post ranked high on the return. As I respect a lot of your work Brad.

I'm just wondering if 'Private Beta' could be a good way to get a little market research done as a small start-up?Even if it does work perfectly, could you not use a 'Private Beta' (or whatever you like to call it) to test the waters before a full launch? It seems to me that it would be which is why I'm leaning towards giving it a try.

By: @fithappypro

@fithappypro — Tue, 02 Nov 2010 06:19:04 +0000

I realize I'm a little late to join this discussion, but I just happened to be googling 'Private Beta' and this post ranked high on the return. As I respect a lot of your work Brad.

By: David Reinke

David Reinke — Wed, 07 Jan 2009 01:55:54 +0000

At StyleHop we are staying "private alpha" – it actually took a bit of discussion for us to land there since it's not the current convention. Glad to see others are thinking the same way we are.

By: jungledave48931

jungledave48931 — Wed, 07 Jan 2009 01:55:53 +0000

I'm a regular reader here.. first time poster
This topic was one I felt compelled to reply to as I think you're on track, but I do think there is still a place for private betas – which I'll describe below.
You're right that what many companies call a 'private beta' really is an alpha – incomplete with a long list of known issues (including scalability). A true beta should be feature complete with a minimum of known issues and represent the final phase of testing before release. So where does a private beta fit in? In my experience, when you transition from internal/F&F testing to public testing there are always a number of issues that come up quickly that were not found internally – things like conflicts with 3rd party software or plugins, firewalls, browser versions, dependencies on internal servers, etc.
If you go straight to a public beta, you run the risk of these issues being seen by thousands of users before you can fix them. A short private beta period with a few hundred users across a wide mix of environments helps shake out these issues.
As an example – we just launched a private beta for the Workgroup Edition of Jungle Disk. Within a few hours we had identified an issue that was affecting a fair number of the users that we had missed in testing because it was specific to their Amazon S3 accounts. Had we launched straight to public beta it's likely that several thousand users would have run into this problem and had a bad first experience with the product. Our private beta period will likely be short – two to three weeks max, but that additional testing will give us confidence that when we open up the public beta we haven't missed any embarrassing issues.

By: Jim Pollock

Jim Pollock — Wed, 07 Jan 2009 01:55:53 +0000

One of the issues is the fluidity of the semantics. I started writing commercial software in 1981 and in several companies I've always pushed and adhered to a methodology and nomenclature of:

Engineering test: internal - engineering team tests their own code
Alpha: internal - test audience expands to people that didn't write the code (other engineering teams, marketing, customer support staff)
Beta: external - limited number of customers, early adopters

We rigidly defined alpha as an expanded INTERNAL audience. Beta was for the user community.

Of course in the 80's, my software experience was scientific/engineering desktop software so there were no scalability issues. (Except could we handle the phone calls if we shipped buggy software.) Which is why in my most successful experiences, we let customer support have full veto power on any rev shipment decisions. They were the most impacted group in the company if bad software went out.

So I am going to vociferously agree with JungleDave. In the "new world of software" scalability and interaction with other software plug-ins/layers is very important and nearly impossible to test internally. You NEED a friendly external audience to excercise it. You want hundreds of testers, maybe thousands, but not hundreds of thousands which could destroy your reputation or ability to sell the REAL product and inundate your development team with noise.

I'm all for the private beta... when it's used correctly.

Jim

By: david_duey

david_duey — Wed, 07 Jan 2009 01:55:53 +0000

From a user's perspective I've always assumed that Private Beta means the product is beyond alpha (i.e. not terribly buggy) but the ability to scale is still in question. I don't mind the Private Beta designation because it implies that the developers are being thoughtful about product roll-out.

By: eddie_lebr12311

eddie_lebr12311 — Wed, 07 Jan 2009 01:55:53 +0000

Honestly, Brad, as a long-time reader I enjoyed the post, but when I step back it just looks like a semantic argument to me. As Web 2.0 continues to evolve, so will the lexicon. True, there is ambiguity in the phrasing at the moment, but I imagine it will sort itself out and, even if it doesn't, I doubt anyone is being misled by the disparate use of "alpha," "beta," "limited beta," etc.

By: Dave Schwartz

Dave Schwartz — Wed, 07 Jan 2009 01:55:53 +0000

Jim Pollock brought up semantics which was my visceral reaction. No reason to get upset about how someone describes their QA / UI optimization process. To each his own and if the product turns out stable and desirable, then who cares what the beta was called! And as a beta partner for Google Friend Connect right now, I can tell you that regardless of what Google calls it, what we're involved with is a QA and Dev request process which is turning out great. Hope all is well!

By: kimm_viebro1980

kimm_viebro1980 — Wed, 07 Jan 2009 01:55:53 +0000

I was going to say that you're wrong straight out, until I understood that you were suggesting that these private betas be called alphas. I'll give you that, but from a support professional's experience, my take on it is that even if it's been used inappropriately recently, private beta is still an important stage in development.

If scalability is a question, then certainly that is one of the things you want to test, so private beta shouldn't be used to mask a lack of scalability. There are plenty of other reasons, though, for private beta and part of it has to do with managing expectations and risk, which can't happen as well if it's a public beta, even with a more forgiving public. I've always counseled that it's important to define a set number of desired beta participants and a set list of criteria for what makes a good beta participant and that no beta testing happens without falling in line with those parameters. A public beta can't do that and whether your customers are paying with money, eyeballs, or goodwill, it's not smart to squander that.

In private beta, you're in shakedown mode and not just for the code. You're testing other post-release aspects of your operation too and how they might be impacted by the release. You're growing your support knowledge and expertise in real-world scenarios, not the la-la land of ideal lab conditions that occur during alpha. You don't really want support to be learning that stuff (and showing their lack of knowledge) in the full fray. As JungleDave pointed out, you're also looking for more issues to address by getting a better cross-section of users than you might have had during the alpha phase. In fact, in recruiting private beta participants, I look for operational platform diversity. Sure it's more trivial these days to roll out a fix for a problem you might discover during this phase, but do you really want to have do expose that problem in front of the entire user population, especially if it's one of those ugly gotchas?

I also like to recruit beta participants based on how they use the product (are they using it in depth, breadth or some other way unique that gives me more data than I would have otherwise?) and their willingness and availability to partner not only on feedback but on troubleshooting too, without risking damage to the customer/provider relationship. This also can't be said of a public beta. Sure, you'll probably get a diverse set of feedback, but that doesn't guarantee you'll get people who experience issues AND are willing to work with you appropriately to sort them out. Plus, you'll get all sorts of people who get noisy over stuff you already know about or don't want to know about and you'll have to deal with that as well as the people who just go snark on you behind your backs. This is not managing risk well.

To treat it any other way than a shakedown cruise is to be pushing (alpha) code out publicly before it's time, expecting beta code to solve issues when you don't actually know if that's true or not or, as Steve points out, using the term 'beta' inappropriately as the new New for marketing.

I realize that some aspects of this may be less true for a smaller operation, but that doesn't make it untrue, even for SAAS. I've had many occasions proving the most important reason to come in at 6am is to be available for the onslaught of customer yelps when code got pushed out overnight without letting us know in support. Private betas done right greatly minimize that and it keeps customers and other parts of your operation much happier when it works the way it's supposed to.

By: Nick Harris

Nick Harris — Wed, 07 Jan 2009 01:55:53 +0000

I agree with Jungle Dave and also with Nick Bradbury’s comments about beta software in the world of desktop applications now a days (http://nick.typepad.com/blog/2008/05/beta-buggy.h...

With Inbox 3.0, I did a short private beta period which I used to both shake out configuration bugs as Jungle Dave points out (especially important with a Outlook add-in) as well as flushing out exactly what the new feature set would look like. This proved to be very helpful since I ended up adding a handful of “helper” features to compliment the new features in 3.0.

With what Nick B. points out, I had a private tester who seemed more than a little ticked off that Inbox 3.0 wouldn’t load in his Outlook configuration and that it took me a few days to identify the issue and fix it. If I had released publicly, I would have left a bad taste with thousands of users instead of just a few.

Which brings up bug reporting. Managing bug reports in a beta period is hard enough, and having the same issue reported by thousands of users makes it even tougher. Being able to flush out the easy to find / common use case bugs with a small audience gives you the chance to clean those up before the avalanche of bug reports that come with a public beta.