Accidentally Deleting Everything

The first time I experienced someone accidentally wiped out a full set of data was on an IBM PC with a Tallgrass  Hard Drive and Tape Backup. I was at PetCom Systems, my first real job. It was a Friday evening and I was the only person around. The phone rang and I answered it, “PetCom Systems, how can I help you?” It was a user from somewhere who was trying to back up for the weekend and didn’t know what to do next. I asked her what she had done. She walked me through what was on her screen, which basically said “Are you sure you want to delete all the files on your disk.” She had already pressed Enter in a panic, it had finished whatever it was trying to do, and her PetCom software (and nothing else) worked anymore. She had somehow, from within Tallgrass, wiped out all the files (except for DOS) on her drive when she was trying to make a backup.

I told her what I thought had happened. I was 17. She cried. She told me her boss was going to fire her when he found out. She cried some more. I tried to say something soothing but I didn’t really know what to say. She eventually stopped crying, told me thanks for trying to help (I think she knew I was 17) and we said goodbye.

I went into the bathroom and threw up.

Since that time, I’ve typed some version of del *.* and answered Y more times than I’d like to think. I became good friends with Norton Unerase. After deleting the wrong directory a few times in the mid 1980s, I started always typing the full path name when I wanted to delete a directory. After doing this wrong a few more times and having MS-DOS eat files I wanted, I started making a backup before I deleted a directory. At some point I became pretty paranoid about backups.

And then I got casual again. About five years ago I decided I didn’t really care about any of the data that I had and if it all went away, I’d be fine with that. Fortunately, I’m not responsible for any data at work so I can’t really do any harm there. The only data that really matters to Amy are her photos so I’m extra careful with them (and have plenty of backups).

As I started down the Great Photo Organization of 2016, I made a backup. Yay. As I fell in love with Mylio, I gave myself the illusion that it was backing things up correctly because of its “Protection” approach.

Early yesterday morning, I set up Mylio on Amy’s computer, feeling ready to get her rolling with it now that I had organized 22k+ photos and was very happy with them. I installed Mylio, set it up, pointed it at the photo directory, and hit Enter. It did something different than I thought it would (and that it had done when I added on my second computer – or at least I think it was different.)

I then tried to set it up correctly again, the way I wanted. It created a second photos folder in Amy’s instance of Mylio and started adding all the photos to the library again, doubling the photo count. I highlighted the first folder in Mylio that I had set up and hit delete. I told it to only delete from this local computer. However, I’d pointed it at the Dropbox file share of our photos.

Two minutes later as one folder was counting up and the other was counting down, I realized I’d fucked myself. My heart rate and blood pressure went up and a giant “Fuck, fuck, fuck!” emerged from my lips.

It took me about ten minutes figure out what state things were really in and how bad it was. I knew I had a backup pre-organizing folders, so my worst case was that I lost four hours of moving files around from folder to folder. Mylio grinded away for about 30 minutes synchronizing all machines to the same state, which was one where there were no photos anywhere.

Amy said soothing words to me during this stretch of time. She had only asked me 351 times the previous few days if all her photos would be ok, so I put her behavior in the heroic category. Heroic calmness. “I feel bad for Brad” was soothing said by her, instead of “You fucking asshole – you deleted all our photos!” which would have been appropriate for her to say.

After everything settled down, I went into Dropbox, clicked on Deleted Files, and clicked on Restore next to the folder that said “Photos.” Dropbox is happily doing its thing, giving me back the four hours I might have lost.

Dropbox wins today. You get huge karma points for saving my bacon without me really deserving it. Thank you.

Mylio – you need to clean up a little of the UX around Dropbox.

Brad, ok, you’ve blown it once – be more careful now.

  • 4 hours – ouch. And another glimpse into why Amy and you work so well together. She trusted you to fix the problem – which you did.

    Did you read the story of the startup Code Spaces? They had everything in AWS (including backups) and it was all deleted by a single person. Game over for the company.

    https://threatpost.com/hacker-puts-hosting-service-code-spaces-out-of-business/106761/

    It’s just amazing how fast data can be deleted.

    • OUCH!

    • I remember the story, but don’t remember how it happened. I just sent a note to everyone in our portfolio about this.

  • I said this yesterday – I have 3 levels of backups of most stuff + 4 for the repos. I use my own webdav folder, local physical hard drives + memory sticks. I have some original source code printed out, too.

    The last time I got burned was back when I still ran my own mail server + the drive w/ the root volume – and importantly, the backup jobs – began to suffer CRC-style errors. This apparently started before it exhibited visible errors, because a non-trivial portion of the mail backups covering the first few years of fp + my consulting before it was just garbled junk.

    Backups of backups of backups ever since.

    • Wisdom on your part!

  • I’m extremely paranoid about backups myself, but I also deleted some of photo library accidentally about two years ago. My son recovered a bunch of photos from SD card data, but some were just plain gone.

    Still makes me sick when I think about it.

    I’m glad you recovered Amy’s photos!

  • war story alert – jbminn

    Relatedly – hard drives can sometimes be ‘repaired’. Everyone knows abt Norton’s (or whatever is best for windows today) + Drive Magic and similar applications.

    Back in the pre-internet days, when mastadons + XTs roamed freely, I was frequently faced w/ recovering data from non-booting customer machines out in the middle of nowhere. (I repaired PCs + data center equipment)

    I figured out that the super old-school 17xx errors could often be fixed by opening up the hard drive casings (yes, really) and carefully bending back the small spring-loaded latch that locked the head(s) in place when the system was powered off.

    That trick worked easily 20x over the years + allowed me to recover 100s of thousands of lost files.

    • Do you remember Ontrack and Mace? They were a regular part of my non-mechanical toolkit.

      • I’ve heard of Ontrack but not Mace. I bought a copy of Norton’s way back + was able to use it on most of the windows boxes I encountered.

        I learned the spring-latch trick before I began using software to repair drives. The first customer who saw me try this freaked out. I had to remind him that the drive *already* didn’t work + that by opening it up, I would a) not make that worse and b) maybe determine what’s wrong + be able to fix it.

        That was the key to a whole other dimension. I quickly developed a reputation for fixing dead hard drives + saw *lots* of very upset people become very, very happy as a result.

        • Mace was the silent killer of the bunch. Paul Mace was heroic. It’s mostly gone from the Internet, but this was a fun article.
          http://articles.chicagotribune.com/1987-02-01/business/8701080849_1_hard-disk-files-unformat

          • Nice. I’ve heard of that technique, so maybe I did hear of Mace at some point. I don’t specifically recall the name, tho. That was in late 80s, right?

            I first used my spring-latch trick sometime in ’84. This was after the first of the Compaq Plus’ were stating to fail w/ the hard drives that wouldn’t boot back up after powering off.

            The click-click-click sound that it made convinced me that it was a mechanical problem + prob had a mechanical solution.

          • Yup. Late 80s.

            My hardware approach was to take the drive out of the drive back, bang it once hard on the table, and reinstall it. That worked a bizarrely high percent of the time.

          • yeah, the ‘bang it’ technique probably freed up the same stupid latch.

          • Still does. a 1TB SATA drive is sitting on my desk having recovered with just such tough love. All other soft/hardware attempts to resuscitate had failed until I banged it on my desk and the vibration on the desk told me that it worked. Again.

          • How about the freezer trick? Put the drive in a zip-lock bag, and put the whole thing in the freezer. Saved more than a few drives back in the day.

  • rm -fr *

    Shudder

    • i’m sure you know abt aliasing commands like that to remove the switches

  • Same thing happened to me. Lucky me, a combination of dropbox, bitbucket and github got ~99% of my work back

  • So so glad you were able to restore. Multi device + cloud sync is powerful but can be dangerous as it’s natural to want to delete parts of the data in one location and easy to forget to unsync first. It’s incumbent on the software providers to work extra hard on that UI and be sure there’s easy restoration functions. I’ve been reading your photo organizing/storage posts with great interest and am planning to try Mylio. Interestingly it reminds me of Sharpcast Photos from years ago – another example of sometimes good ideas just being too early.

  • Viva la Dropbox. I love that product. Gives me so much peace of mind.

  • DaveJ

    Great scene in Silicon Valley where Russ puts the bottle of tequila on the keyboard and it deletes all the client’s video content (porn).

    • Yes – that one was hilarious.

  • Rob Ryan

    Ha! You have a great ability to humor, comfort and scare the living shit out of me all at the same time. And PS by the way, WTF is with photo apps? When someone creates a simple, straight forward program, I’ll eat my hat!

    • Try Mylio. I’m loving it.

  • In 2016 I want to learn to be more like Amy, calm.

    • Calm she is, that one.

    • Sam

      Amy: “I feel bad for Brad”
      Brad: “Mylio – you need to clean up a little of the UX around Dropbox.”
      Mylio is lucky Brad has Amy.

      • Yeah – no kidding. More importantly, Brad is lucky he has Amy.

        • StevenHB

          Yup.

  • Nick Ambrose

    In an internship almost 20 years ago now, one of my tasks was to write a “live point-in-time” backup for the Oracle DB (goodness knows why we didnt use a tool written by an actual professional!)

    This database was the result of a multi-year effort involved with gathering data in every format known to man (email, spreadsheets, txt files, DB dumps) about the inventory of a large phone service provider (very large).

    Godknows how many man-hours had been expended on collecting it, validating it, scrubbing it and importing it into the master DB.

    A few weeks after my internship, I got a call from my old boss, and in a shaking voice he asked me “That database backup software you wrote when you were here …. does it actually work ?”

    Yup, he had basically typed something the equivalent of “Drop database XXX; COMMIT” on the LIVE (and only full copy) database.

    The only backup was the one I’d made on my last day.

    I told him how to run the recover tool and never heard from him again.

    It *should* have worked but … I guess I’ll never know for 100%

    • Pretty amazing that he didn’t call you. I take that as a BAD sign …

      • Nick Ambrose

        Hmmm, could be (hope not). I was definitely not going to call him!

        • Wise on your part!

          • Nick Ambrose

            I am choosing to believe it worked just fine and he was too embarrassed to call me back. Didn’t impact my final review from them either so …

  • Eli

    “I went into the bathroom and threw up”
    Thank you for a few good seconds of pure laugh, this line brilliantly fitted in the situation, good one.

  • I’ve given a couple friend/family advice to follow a simple strategy I follow for the the limited amount I have that I don’t want to lose (and in some cases don’t want to store in the cloud). For almost a decade now I’ve used Apricorn drives, which are and affordable, self-contained and reliable part of my back-up solutions/mix:

    http://www.apricorn.com/products/external-storage-backup-1/portable3.html

    I use it to keep a periodic back-up of that data (and actually have a second unit offsite in storage too, in case of fire, theft, etc.). Although I use time machine and other tools, this helps for the stuff that’s too important to fail. While it’s not always current to the day, week month, it’s a rock solid approach. It basically ensure that I never end up in tears or puking because of this sort of situation — a lesson I learned the hard way once many moons ago.