How To Fix Obamacare

Now that our federal government is back at work and the short term debt ceiling thing is resolved, it should be no surprise that the news cycle is now obsessed with Obamacare and its flawed implementation. Over the weekend I must have seen a dozen articles about this online and in the NY Times, and then I woke up this morning to a bunch of new things about the site underlying tech, how screwed up it is, and what / how the Health and Human Services agency is going to do to fix it.

The punch line – a tech surge.

To ensure that we make swift progress, and that the consumer experience continues to improve, our team has called in additional help to solve some of the more complex technical issues we are encountering.

Our team is bringing in some of the best and brightest from both inside and outside government to scrub in with the team and help improve  We’re also putting in place tools and processes to aggressively monitor and identify parts of where individuals are encountering errors or having difficulty using the site, so we can prioritize and fix them.  We are also defining new test processes to prevent new issues from cropping up as we improve the overall service and deploying fixes to the site during off-peak hours on a regular basis.

From my perspective, this is exactly the wrong thing to do. Many years ago I read Fredrick Brooks iconic book on software engineering – The Mythical Man-Month. One of his key messages is that adding additional software engineers to an already late project will just delay things more. I like to take a different approach – if a project is late, take people off the project, shrink the scope, and ship it faster.

I think rather than a tech surge, we should have a “tech retreat and reset.” There are four easy steps.

  • 1. Shut down everything including taking all the existing sites offline.
  • 2. Set a new launch date of July 14, 2014.
  • 3. Fire all of the contractors.
  • 4. Hire Harper Reed as CTO of, give him the ball and 100% of the budget, and let him run with it.

If Harper isn’t available, ask him for three names of people he’d put in charge of this. But put one person – a CTO – in charge. And let them hire a team – using all the budget for individual hires, not government contractors or consulting firms.

Hopefully the government owns all the software even though apparently violates open source licenses. Given that, the new CTO and his team can quickly triage what is useful and what isn’t. By taking the whole thing offline for nine months, you aren’t in the hell of trying to fix something while it’s completely broken. It’s still a fire drill, but you are no longer inside the building that is burning to the ground.

It’s 2013. We know a lot more about building complex software than we did in 1980. So we should stop using approaches from the 1980s, admit failure when it happens, and hit reset. Doing a “tech surge” will only end in more tears.

  • ipostelnik

    A stop and reset would be absolutely the worst possible option. Leaving out the political implications, it’s the wrong approach technically. Without users trying to hit the site, they will learn nothing. The rewrite/reset/whatever you call it is going to be the same failure, regarding of who’s in charge. They need to iterate on the changes, see the issues and fix them as they come along.

    There’s no way a team can go into a room and build a new system of this scope that works on the first try. They have to learn from real-world usage, real integration problems. Focused iteration is right way to manage risk.

    • I’m not suggesting that they don’t release early and iterate. That’s the essence of the existing problem – they did a big bang launch that seems fundamentally flawed. The exact opposite of an agile approach.

      I didn’t say “rewrite” – I said “reset”. 80% of what they have might be useable. Who knows. But someone needs to look at it from that perspective, rather than the perspective of “let’s through more bodies at this and try to fix the mess by overpowering it with a surge.”

      • RealityEngineer

        re: “80% of what they have might be useable”. That might be true, though if that 80% is too intertwined with the rest it risks major spaghetti code and an unmaintainable system and perhaps more time than it is worth to pull out the 20%. Hopefully they do solid API definition walls between the different government entities that would let some systems be salvageable

        Given the scalability problems and talk of loads of different browser side code being loaded it is likely they need to re-architect it from scratch. This should have been built in a way that there could have been private exchanges, which would also force a clean interface wall between systems. What they should have is each insurer providing an API to get plan information which *any* “exchange” can use to allow private entities to create more user friendly exchanges, and also a sign up API. Then the government either provides a hook in where the private site hands off to the government site to take the plan and calculate a subsidy (so whatever background information they need isn’t handed off to the private entity). The exchange gets a processing fee for each signup. If they insist, government could provide its own exchange (though that would deter competition for more user friendly exchanges).

        • Well said!

        • ipostelnik

          Your API idea is a pipe dream. This is hard to pull off in a situation where everyone is really eager to work together. In the case of ACA, I’m not sure how much insurance companies want to be integrated. I suspect their technical capabilities are much more limited than a typical sw shop. Throw in regulatory compliance, and need for public/industry review for government-pushed API and we’re talking about years.

  • Switch to the scheme Massachusetts uses. You go to the web site and they let you compare all of the policies. Since there are a finite amount of policies all of the combinations can be precomputed. Fill out some income data and you get an estimate of the subsidy. You pick the plan you like and they give you the phone number for the insurance company. You don’t buy anything on the site. When you leave nothing is stored. It only takes a few minutes. Sign up is handled by the phone call to the insurance company.

    Subsidies are then handled by adjusting your tax exemptions. At the end of the year your tax return is used to reconcile everything and to compute your true subsidy. It is easy to do at that point since they have all of your income data and the insurance company files a 1099-like form to show that you have paid for the insurance. There may even be a way to file quarterlies for people that are paying negative taxes due to credits.

    No giant government bureaucracy needed to manage this.

    • BTW, building the needlessly complicated system they are working on now makes sense if it’s true purpose is to go into competition with private insurance and offer a single payer option. If that is the true purpose then the MA style system would not have met their hidden goals.

      • Yup – my dad (Stan Feld – believes that this is the ultimate goal.

        • RealityEngineer

          Most government healthcare regulations work to limit competition to protect existing players. The *threat* of single payer, without the reality of it, helps deter new insurers from starting. If there is a strong chance a business might be nationalized in the next ten years, or even twenty, it isn’t clear that it is worth the risk (since after years to build the company, whoever buys your stock in a cash out will only do it at a decent price if they think the company will be around for some number of years afterwards).

          In reality they *should* get rid of employer coverage in the sense that they should change the tax break to individuals so they buy their own insurance. Since the payouts won’t change, on average given the same policies competition should force the prices down to be comparable. Except they’ll likely be better since people are more likely to buy higher deductible policies which are a better deal. Employe coverage helps insurers, and it helps big business at the expense of small since they have a recruiting advantage when they negotiate better benefits at the same cost.

          For many details on what is really wrong with the healthcare system, “crony capitalist” favors to industry at the expense of the public, rather than a real free market, this provides details with links to sources:

        • was wondering what Mr. Feld thought. Missed him at Stocktoberfest this year.

      • tonyspencer

        That was always the goal. Collapse the system by applying pressure on private industry. Premiums will skyrocket, public will demand a fix, hello single payer. On occassion they’ve told exactly this:

        Obama: “I don’t think we’ll be able to eliminate the employer based coverage immediately…”

        This whole mantra of “if you like what you have you can keep it” was a lie.

      • aaronbrethorst

        House progressives would’ve loved to have seen a single payer bill passed in 2010. Wouldn’t have happened. Single payer, if and when it happens, won’t happen for another decade or two.

  • tonyspencer

    I have to disagree with ipostelnik and fully agree with Brad. I’ve attempted to use the product and found it so incredibly broken it needs a full reset. For instance, after one long wait upon attempting to create an account I received the cryptic error that it failed and I should try again. I tried once more using the same username and was given the feedback that ‘this username is already taken’ despite being previously told my account could not be created. I have a hunch the current app is creating dirty data.

    • Definitely dirty data. That’s a huge part of the issue – they are now going to have to write a bunch of code to clean up the dirty data, which has been propagated through the various (not very well) interconnected systems. Ugh.

  • John Fein

    Agree with resetting the team but wouldn’t take it offline unless it’s substantially broken (majority of users can’t use it as intended). Related: the Harper Reed Wikipedia entry is great.

    • Early data shows that it is substantially broken – hence my recommendation.

  • Kris Tuttle

    Another right on post. Bonus points for citing Brooks. That is an awesome book I read as a kid and should be required reading for everyone in development. Big web systems are no different than the massive IBM MVS, IMS and CICS systems of old. Every major release got hung up and IBM called together the same handful of their best software engineers, locked them in a room for two weeks, they patched up all the gaps created by the large disconnected development teams and made it work. Sounds like the government approach to this system with contractors and consultants was destined to be a disaster.

    • Rosey

      Along with Brooks (whom I was potty-trained on), I remember, “The simplest things at high volume are the most complex.” Wish I could remember who said it.

  • laurayecies

    Interesting post. The CA system seems to be basically working – a bit slow but usable. I find it amazing that there were all these separate efforts but then perhaps it is fortuitous – can’t one of those working systems be repurposed.
    On a related note – I’m dealing with a similar issue with a poorly engineered site not ready for prime time – my son is applying to college and the “Common Application” web site is also breaking – many colleges are postponing their deadlines as a result.

  • Dan Tienes

    Best solution I’ve seen so far. The key is to delay it so they can fix it (or start fresh, which may be better). But trying to fix it live is like changing the wheels on a car in motion. And Brooks’ Law has never, as far as I know, been wrong.

  • kermit64113

    The reset concept also makes sense from a marketing perspective. To use the Bill Aulet (“Disciplined Entrepreneuship”) terminology, the target persona (there are two – the uninsured, and the 20-30 yr old healthy early adopter) are disillusioned. They cannot assess the value proposition and are moving to other alternatives (which the second group has). You have to start over from a market perspective – too many have tried. (Can you imagine the outrage if we discover a privacy breach in the middle of this?)

    The second issue is legal. If a fully workable system cannot be implemented until December (see today’s NYT article), can the government continue to implement the individual mandate? I’d hate to see the court case – “I’m being fined, your honor, for something I could not participate in and did not have enough time to correctly evaluate. I thought I was going to have 90 days and only got 30.” Talk about getting the last laugh – Congress skewered for trying to delay the individual mandate for a year, only to end up delaying the individual mandate for ______ months.

    • Rosey

      Hi Jim. I’d love to run a “pizza party” triage on this mess. But who would be the content experts on the front row?
      Who REALLY knows what the workflow must know and remember (logical data elements) in the order it must be CRUD’ed (Create, Read, Update, and/or Deleted)?

      (In the weeds here: Then, CRUD transaction by CRUD transaction, triage which data elements are being handled correctly and which are not — then Agile/Scrum them in a surge.)

      I’m with Brad, this is absolutely doable — IF the right people are on the front row. And that, unfortunately, sounds political.

  • Gearing up the #hireharper campaign as we speak.

    • That is a good campaign.

  • We really need to do a reset on government contracting procedures so that a small set of companies like Booz Allen, Accenture Federal and CGI can’t continually sell massively overpriced, inept solutions just because they know how to work procurement systems and can lobby the appropriate parties. Government contracting is so convoluted that I couldn’t recommend most startups go after it… you’d run out of cash before you got your first check.

  • Shutting it down, even temporarily, would be a terrible political mistake. Those who seek to kill Obamacare would use that shutdown as a sign that “Obamacare is broken. Oh well, you tired. Now, let’s get on to impeaching Obama.”

    They need to band-aid the existing system long enough while they rebuild it from the ground up, properly, and with contractors who can be held accountable.

    I’ve been trying to get my application approved since Oct 1. It appears I am stuck in some kind of loop that’s preventing my application from being received. I can’t get past the “In Progress” state, and I’ve verified my identity numerous times. It doesn’t help that when I go back into the application, my wife’s SS# and my daughter’s SS# keep switching places. Some of their data-checking logic is severely broken or incomplete. It’d be great if I could actually see how much each plan costs but right now I’m stuck in some sort of limbo. It has not been a pleasant experience, but I understand how complex this system is so all I can do is wait until they’ve fixed it enough that I can get the plan data I need to make a decision about health care and insurance for my family.

    • “Terrible political mistake” – this is probably true, but is also a horrible reason to continue with the current plan. We’ve seen that story so many times in the history of humanity.

      • Rosey

        So, “Crow best eaten warm”?

        Agreed. But politics is not a rational enterprise.

  • RealityEngineer

    This New York Times article claims:
    “According to one specialist, the Web site contains about 500 million lines of software code.”

    It definitely sounds like it was created by a mass horde of warm bodies in a poorly designed system leading to way to much code rather than hiring a smaller group of the best people to design things better. Then they want to throw more hordes at it to fix it, likely leading to more of a mess. The budget they had to begin with was of course far more than it likely would have taken to see something like this done in the private sector. Unfortunately a large part of the code&mess sounds like it is within existing government computer systems (which is probably why the line count is so high, since there are a number of systems to deal with) but at least the core exchanges should have been produced competitively.

    • It’s inconceivable to me that a system like this should be 500 million lines of code. That just spells disaster.

      • It’s the government, and they are here to help.

      • Appeos

        Yeah, that’s ludicrous, being an order of magnitude more that a complex operating system.

        Of course, maybe they meant they were counting the line items of the expenses the government contractors put in 😉

    • ipostelnik

      I’m having hard time believing the 500mm lines of code line. Wikipedia article on lines of code ( says that Windows Server 2003 had 50mm lines of code. No way is 10x that.

      I’m going to guess they made up this number to justify the price tag.

  • phil swenson

    this keeps happening. Anyone remember the great Colorado Welfare System disaster?

    And the Airforce ERP $1B mess:

    the people running these projects are clueless.

  • The political fall out if he did this would be amazing. Never going to happen despite being a great idea.

  • Great post from Clay Johnson (of DBOT – Department of Better Technology) on what he’d do to fix this –

  • Jeff Jones

    Seems they have already fixed the issue…only catch is you need a Commodore 64 to run the program,34294/?ref=auto

  • aaronbrethorst

    two words: single payer.

  • Appeos

    I have decades of experience working on hyper-complex software systems and it is easy to see how these things spiral out of control. It is usually because too many people get involved, never too few, so the “tech surge” is ludicrous and shows just how uninformed the President is (not his fault, of course, but his advisers should know better).

    The answer to these situations is always to stop, admit there’s no chance to deliver what was promised, then move on with a small team of people who know what they’re doing, hopefully without the full glare of the media looking over everyone’s shoulder.

    I’d be happy to step up and fix this for free. I’ve been working on some pretty good new technology that allows extremely complex IT systems to be designed, built and maintained, without the usual scenarios of going over-budget, running out of time, etc.

    Actually, I already contacted the President about this technology, but I doubt he’d be interested, as his advisers have already decided who’s going to get paid. I’d do it for free, as I think it is a great initiative. I also offered to help the UK government (their Health IT system project burned through tens of billions before it got cancelled), but again, the government contractors seem to have everything stitched up.

    I did wonder what kind of politician would willingly spent hundreds of millions, or even billions, of dollars when there was a much better value (essentially free) alternative, but…

  • DJ

    Well… maybe. One thing to note is that is not truly national. For example, if you live in Colorado you are redirected to the Colorado exchange web site – is simply an intermediate that lasts for all of one click. I don’t know how many other states this is true for, but there are also some shenanigans going on in various states. Some states are intentionally dragging their heels on implementation. I believe one has a requirement that you have to use a “certified assistant” (a person), but is requiring an exorbitant number of classroom hours to get certified. So only a handful are available to service the entire population.

    Also, I’m not sure I buy that it’s as screwed up at the code level that some say. I suspect instead that there a lots of little things they need to do deal with the scale. Even if you shut it down and tried to iterate, you would still have enormous demand to contend with once it reopened.

    Imagine if Twitter or even Amazon had launched with tens of millions of users out of the gate.

    I think their better off leaving the “disaster” up and having it slowly become less disastrous over time rather than risking everything on another big bang.

  • Kerey Roper

    While the removal of the license is pretty bad, the real headscratcher about that DataTables library is that they are using a non-minified version of the JS in production amongst other best practice misses. There are tools to automate that without removing the license info.

  • mikeduffy
    • Yeah – I saw that. We had some email back and forth also.

  • Andre

    One point that has not been discussed enough is the procurement process. As technologists, we have all criticized the failed implementation and cost, but we can gain a lot more from this conversation if we take time to examine the government procurement process. How is it that 3 people can build almost a fully functional site with similar functionality in a weekend ( for little more than the cost of their time and we paid private companies > $450M for essentially crap.

    One suggestion would be to have an independent council of technology leaders (with no conflict of interest) to vet projects and give estimates of time and reasonable expense required to implement said projects. The government should then use that as their reference and baseline for accepting and approving contracts. The current process is fundamentally flawed because most, if not all, contractors charge the government more than they would non-government firms or agencies. Additionally, the seniority and previous contract advantage that contractors have should be removed completely. This builds a system of more senior contractors that sub-contract work out to other firms that would not have a remote possibility of winning the contract.

  • The politics of shutting down the site —- oh, the Repubs would have a field day. “Look, we managed to shut it down!” 🙂