Engineering test: internal – engineering team tests their own code
Alpha: internal – test audience expands to people that didn't write the code (other engineering teams, marketing, customer support staff)
Beta: external – limited number of customers, early adopters

We rigidly defined alpha as an expanded INTERNAL audience. Beta was for the user community.

Of course in the 80's, my software experience was scientific/engineering desktop software so there were no scalability issues. (Except could we handle the phone calls if we shipped buggy software.) Which is why in my most successful experiences, we let customer support have full veto power on any rev shipment decisions. They were the most impacted group in the company if bad software went out.

So I am going to vociferously agree with JungleDave. In the "new world of software" scalability and interaction with other software plug-ins/layers is very important and nearly impossible to test internally. You NEED a friendly external audience to excercise it. You want hundreds of testers, maybe thousands, but not hundreds of thousands which could destroy your reputation or ability to sell the REAL product and inundate your development team with noise.

I'm all for the private beta… when it's used correctly.

Jim

If scalability is a question, then certainly that is one of the things you want to test, so private beta shouldn't be used to mask a lack of scalability. There are plenty of other reasons, though, for private beta and part of it has to do with managing expectations and risk, which can't happen as well if it's a public beta, even with a more forgiving public. I've always counseled that it's important to define a set number of desired beta participants and a set list of criteria for what makes a good beta participant and that no beta testing happens without falling in line with those parameters. A public beta can't do that and whether your customers are paying with money, eyeballs, or goodwill, it's not smart to squander that.

In private beta, you're in shakedown mode and not just for the code. You're testing other post-release aspects of your operation too and how they might be impacted by the release. You're growing your support knowledge and expertise in real-world scenarios, not the la-la land of ideal lab conditions that occur during alpha. You don't really want support to be learning that stuff (and showing their lack of knowledge) in the full fray. As JungleDave pointed out, you're also looking for more issues to address by getting a better cross-section of users than you might have had during the alpha phase. In fact, in recruiting private beta participants, I look for operational platform diversity. Sure it's more trivial these days to roll out a fix for a problem you might discover during this phase, but do you really want to have do expose that problem in front of the entire user population, especially if it's one of those ugly gotchas?

I also like to recruit beta participants based on how they use the product (are they using it in depth, breadth or some other way unique that gives me more data than I would have otherwise?) and their willingness and availability to partner not only on feedback but on troubleshooting too, without risking damage to the customer/provider relationship. This also can't be said of a public beta. Sure, you'll probably get a diverse set of feedback, but that doesn't guarantee you'll get people who experience issues AND are willing to work with you appropriately to sort them out. Plus, you'll get all sorts of people who get noisy over stuff you already know about or don't want to know about and you'll have to deal with that as well as the people who just go snark on you behind your backs. This is not managing risk well.

To treat it any other way than a shakedown cruise is to be pushing (alpha) code out publicly before it's time, expecting beta code to solve issues when you don't actually know if that's true or not or, as Steve points out, using the term 'beta' inappropriately as the new New for marketing.

I realize that some aspects of this may be less true for a smaller operation, but that doesn't make it untrue, even for SAAS. I've had many occasions proving the most important reason to come in at 6am is to be available for the onslaught of customer yelps when code got pushed out overnight without letting us know in support. Private betas done right greatly minimize that and it keeps customers and other parts of your operation much happier when it works the way it's supposed to.

With Inbox 3.0, I did a short private beta period which I used to both shake out configuration bugs as Jungle Dave points out (especially important with a Outlook add-in) as well as flushing out exactly what the new feature set would look like. This proved to be very helpful since I ended up adding a handful of “helper” features to compliment the new features in 3.0.

With what Nick B. points out, I had a private tester who seemed more than a little ticked off that Inbox 3.0 wouldn’t load in his Outlook configuration and that it took me a few days to identify the issue and fix it. If I had released publicly, I would have left a bad taste with thousands of users instead of just a few.

Which brings up bug reporting. Managing bug reports in a beta period is hard enough, and having the same issue reported by thousands of users makes it even tougher. Being able to flush out the easy to find / common use case bugs with a small audience gives you the chance to clean those up before the avalanche of bug reports that come with a public beta.

There's still a place for private betas (or more accurately, alphas) though. As Dave said, there's always issues when going from internal testing to public testing, and a private beta / alpha version can help to weed some of these out before releasing to the general population.

So here's a different take on the beta question: We offer a free version and those users act like our beta users (there's lots more of them and they're early adopters). Our paid customers tend to suggest the more sophisticated improvements based on their real working requirements. Both are very valuable for different reasons.

I've been involved in the delivery of software products since the (very) early 80's. Alpha was always internal test. Beta was really the opportunity to test "nearly final code" and the preparedness of the organization to deliver and support the code.

Scalability is not a new issue. Neither is integration with other applications, APIs, widgets, etc.

What is "newish" is the growing desire to push more responsibility out to the end user for nearly every facet of product evaluation, education, and use. There are striking similarities to simple cost reduction efforts underway in nearly every business. The underlying truth here is that many web companies; too often funded on the belief that it only takes a few hundred k to launch a product, severely underfund the efforts required for going to market in responsible fashion and apparently feel compelled to share their shortcomings with the prospective market.

Like you Brad, I've decided to abandon participation in private betas. I soon discovered there was no need to hide from the paparazzi after enrolling in a private beta and my ego was ill served.

However, let me propose one legitimate use of the term "private beta" that I think has emerged as particularly unique to our day and time. It is how we are expecting to use our private beta when we launch in early fall.

Because we are no longer "shipping bits", we have an opportunity to get software up and running more quickly and test value propositions earlier. In the "shipping bits" world, it was very difficult & costly to get a high scale test of your core value proposition. Surveys? Focus groups? All inadequate.

However, in today's day and time, it is exceedingly easy to get up and running one value proposition and test whether that is the correct one, at some levels of scale. By running a private beta, you have an opportunity to pull together a group of users who can really help you decide what direction your product needs to go.

That's not to imply that the level of quality shouldn't be at or near "beta quality". One example to point to is Xobni. They started off as email analytics and moved to Fast Email Search as positioning. It's harder to do that when you are in the "public" phase.

Testing that value proposition and iterating is to me the key distinction on the use of a private beta.