I ingest a ton of information on a daily, weekly, monthly, quarterly, and annual basis. My process for doing it today is entirely manual. I’m starting to look around for a way to automate this using the metaphor of a “personal dashboard”, not dissimilar to the idea from the 1980’s of an EIS (“executive information system”). Let me explain.
- Daily: I have an information processing routine each morning that is web-based. I open a folder in Firefox that contains 14 tabs. I then go through all of them – most, but not all are news related. A few are interactive and require data from me. I then scan through my tweets from the previous night. I then review my “Daily” email folder – most of the items are “daily reports” from a variety of companies I’m an investor in. Next up, my RSS feeds. Finally, I process whatever email came in from the previous night.
- Weekly: I have a weekly tab in Firefox. There are only 5 tabs here and they shift around a little. But – they reference a variety of text and numerical data that I check on a weekly basis.
- Monthly: I get financial statements (balance sheet, income statement, cash flow statement) along with board packages from all of the companies I’m an investor in along with all of my personal financial information.
- Quarterly: Similar to monthly, but for the quarter.
- Annual: Similar to monthly, but for the year. I also generate a variety of other “annual data” much of it to do with either money or fitness.
My Daily routine takes around an hour. Weekly, which includes reviewing my upcoming calendar, takes about 30 minutes. I don’t know how long Monthly, Quarterly, or Annual take as they are usually spread out over multiple days.
In theory, I’m using Firefox and Outlook as my personal dashboards to get to this data and then viewing it in a variety of apps including Excel, Adobe, and Word. However, this is really unsatisfying as the data is (a) in different formats, (b) impossible to search effectively, (c) not persistent, and (d) difficult to handle or manipulate.
My guess is I need both an (a) ingestion and (b) presentation layer. The ingestion layer seems straightforward – the software I’d use for my personal dashboard should be able to generate an XML template for each “type of data”. I should be able to configure this (or – optimally – the ingestion layer should be able to figure this out automatically). The ingestion layer should be able to handle different types of inputs – html files, xml files, emails, or some other quasi-API. So – “Glue”.
The presentation layer is a little harder for me to get my mind around. A year ago I would have said “hmtl is fine – just give it to me in Firefox via a web page.” In some cases this is fine, but I want finer grained control over how this stuff is displayed. Some of the web pages I look at are formatted worse and are less flexible than the DEC-based EISes I played with in the 1980’s. In many cases we haven’t made any progress on the presentation layer not withstanding all the efforts of Edward Tufte. So – “HCI”.
I’m hopeful that in a decade I’ll have a much more effective way of dealing with my periodic information routine. Until then, I’m searching for companies working on both the ingestion layer and presentation layer (preferably both). Feel free to give me a shout if this is something you are working on.
hi brad,
you might check out http://www.lifeIO.com, now in public beta. it's a start in the direction you suggest and i'd very much appreciate your feedback.
lifeIO™ – my life my way.
lifeIO streamlines your day to day
lifeIO is a personal portal that simply and flexibly aggregates and integrates all of the elements of your daily online life in a single browser window. mail, chat, social networks, calendar, todos, notes, rss, shopping.
In an exponentially growing and fragmenting information stream lifeIO aggregates, filters, organizes, stores and displays what I need, want and am interested in real time, in an easy to use, customizable interface, accessible in any browser from your computer or phone.
Comment by Bruce Spector — October 12, 2009 @ 1:49 pm
this is also the key to the entertainment and info industry. Transmedia Interactive structure both in story(cube) and the technical architecture… and interface from the couch.
Comment by @im2b — October 12, 2009 @ 2:14 pm
weaving some of the "personal dashboard" talk into the defrag discussion here:
http://defragcon.com/Blog/?p=443
Comment by eric norlin — October 12, 2009 @ 2:17 pm
Building something like this for 1 person would be super simple to do. The problem is making it so general that anyone can use it.
Couple of questions:
1) How are you receiving financial information for your Monthly, Quarterly, and Yearly reports?
2) The sites that require information from you, are requiring what type of information? And does it change? If so at what frequency?
I wish I knew more about the type of data you are consuming, this would be an very cool project to work on…
Comment by JChauncey — October 12, 2009 @ 3:32 pm
This will be hard to improve upon in the near term, but a post you made quite some time ago still sticks in my mind – about a scale with wifi that goes to dailyburn. That has opportunity.
Comment by Dave M. — October 12, 2009 @ 3:38 pm
Bruce,
I almost never give personal details on a site without some sort of background info on what software does. Why doesn't the site have a brief over view of the software that people can look at before signing up?
I would be surprised if anyone signed up just seeing your site with no info on what your product does.
Comment by Dawn — October 12, 2009 @ 3:40 pm
having just gone from private to public beta we're just working on a new splash page that will do exactly what you suggest. should be ready next week.
Comment by Bruce Spector — October 12, 2009 @ 3:57 pm
[...] thing from Gist to Socialcast, and folks like Brad are pretty actively seeking out how to use “personal dashboards” to sort all of [...]
Pingback by Is the “Email Era” Over? | CloudAve — October 12, 2009 @ 9:03 am
having just gone from private to public beta we're just working on a new splash page that will do exactly what you suggest. should be ready next week. not intended as an excuse but explanation, we're a very small team and lifeIO has many moving parts. i believe part of the reason that you haven't seen the "personal dashboard" that you want is that it's hard to do right and to scale. lifeIO is an ambitious project and should have been the province of yahoo (after they acquired WebCal) and google or ms but all of them are too saddled with their legacy baggage to do the rethink/innovation that what you want requires. that's why i built lifeIO.
Comment by Bruce Spector — October 12, 2009 @ 4:09 pm
Uhhh…have you heard of MindTouch? This is what we do. Well, to be honest what you have outlined is something specific, but it is indeed precisely our value prop. That + collaborative.
Why don't you help MindTouch to flesh out the specifics of what you're after. In the end we'll have a new solution and module for EIS and you will have your dream executive dashboards realized.
To be clear, building this on MindTouch is relatively easy. To get a sense of more general use cases watch the vids to the right of the page here: http://www.mindtouch.com/solutions/intranet_solut... (first one is a tad hyperbolic, 2nd is overly basic, but it gets good after that.) Also, here is another one that provides the high level concepts:http://www.mindtouch.com/Customers/success_storie...
Comment by Roebot — October 12, 2009 @ 4:32 pm
cool post. I do something similar but you sound more disciplined than me! I do a pretty good job of mashing my cycling and running data from power meters and heart rate monitors but when I get into the business data it gets a bit trickier.
we are working on part of the solution at indicee.com. Our initial use case is starting with the frustrated spreadsheet user who manually cobbles data from a bunch of different systems to create a view/dashboard that can be used for decision making and collaboration. Our initial target is SMB, business users who use packaged applications (like ERP and CRM). I assume your portfolio companies fire you a bunch of reports and spreadsheets that we could consume and then present. Like I said…we don't solve all of your problem but it might be fun to gather some requirements to see where we can take it. As we progress we are going to start looking at scraping and gathering unstructured data that can be "munged" together with structured data. This is a tricky problem to solve in a simple, point and click way for general use. There are all kinds of expensive, enterprise apps that you can pile a bunch of IT guys and consultants on to solve your problem but that isn't what the masses like you or I need. What is need is a simple, low cost, service based app that you can upload and connect your data to.
Comment by @MarkCunningham — October 12, 2009 @ 4:56 pm
Did a lot of work with this sort of thing over the last few years, mostly CEO/CFO of huge multinationals. I saw a lot of vendors lead with technology rather than strategy, something I suspect you are better positioned to avoid. Execs wanted personal dashboards but were often arbitrary about why they wanted them and what actions/decisions would be possible as a result. @MarkCunningham is on the right track–get the requirements and uses right before working on the tech.
Comment by Joseph Logan — October 12, 2009 @ 5:07 pm
I have to agree with Aaron, MindTouch is leading the charge in this space. We can help build this out for you.
Comment by Mark Fidelman — October 12, 2009 @ 5:19 pm
My take is that the daily routine and the less frequent weekly.monthly/quarterly routines require very different treatment.
The daily routine is more about managing actual messages than it is about managing data. Unless you are getting hundreds of messages a day in various places, it's more efficient to look at the messages themselves then it is to look at metadata about the messages. We learned this the hard way at threadsy, where we started with a more data rich page and ended up with something more like a super-inbox. Check it out at http://www.threadsy.com we combine email, facebook and twitter into something very new.
The longer period info is really more about reporting changes in data, and not actual messages that you need to attend to. I think they will end up being two completely different products…
Comment by Rob Goldman — October 12, 2009 @ 6:12 pm
I am in the process of building a Personal Dashboard.
I am in the infant stages of building a Personal Dashboard. The one I have in mind will not only do what you are requesting, but also function as a more robust PC component working in conjunction with other PC resources and programs, on and off-line.
The Dashboard I am building will have a revenue model built into it. It will be 100% free to the client and have no advertising. Our revenue model is transparent to the user.
I believe with the right team I can have this model up and running in the next 6 months. It should be well received in the personal and business world as a tool that simplifies daily task or simply navigating your digital world.
If you would like to be updated when this goes live or would like to suggest features, please feel free to email me at ike@logobids.com.
Isaac Hill
Founder
Logobids.com
Comment by Isaac Hill — October 12, 2009 @ 6:57 pm
Take all of your inputs and multiply for working moms – add in kids, schools, schedules, teams, extra-curric activities, volunteer, boards, committees, bar mitzvah prep, college apps and visits, dentists, specialists, field trip forms, menu plan, grocery list, household stuff, and don't forget the dog's playdates and vet visits (the dog isn't on Google calendar yet). Reminders, newsletters, requests, calendars. Looking for a team to help me build the dashboard of a working mom's dreams.
Comment by Cheryl Fellows — October 12, 2009 @ 7:38 pm
Hi – I signed up but can't get in. It timed out after I submit the user info. Looks like it got something saved b/c I then tried to login, did the "reset password" and got the email to do that. . . changed pw, still can't get in, am getting 504 Gateway time-out.
Comment by Cheryl Fellows — October 12, 2009 @ 7:41 pm
heh I saw the threadsy article on TC before I saw Brad's comment on the dashboard and thought about posting a link to the article… Cnat wait to get my invite.
Comment by JChauncey — October 12, 2009 @ 10:45 pm
Bruce – thanks for the note. I set up an account and got started. This is more of a lifestreaming app than I’m looking for – I’m much more interested in a wider variety of unstructured data along side my structured (XML / RSS) data. In addition, at first blush, I have to explore all my contacts from Outlook (you don’t include a way to get them without a one time export) and you didn’t manage to ingest my 800+ record OPML file. I’ll keep playing around – it’s a neat app – but much too narrow for what I’m looking for.
Comment by Brad Feld — October 12, 2009 @ 10:46 pm
isnt that what the wall chalkboard is for? =)
Comment by JChauncey — October 12, 2009 @ 10:46 pm
1. Financial information comes in either .XLS files or .PDF files.
2. Some daily, like the data I enter at http://www.daytum.com/bfeld or http://www.dailyburn.com. Others are weekly. Mostly it’s date + data_point (1..n)
Comment by Brad Feld — October 12, 2009 @ 10:48 pm
Yeah – no kidding. I’d love that.
Comment by Brad Feld — October 12, 2009 @ 10:48 pm
Brad, this is exactly the problem we've been working on at Mentations, Inc. (http://www.mentations.com) over the past 6 years. In fact, I'd be happy to send you a link to my "Mentations Mini" dashboard which is a limited, read-only version of my full Windows dashboard that models all the areas I consider important in my life including news, my kids, career, relationships, etc. While we don't have the Outlook integration currently, I'm confident I could get something hooked up in short order. As already mentioned by others previously, creating a generalized solution robust enough to handle the variances in people's myriad lifestyles is tough, and hence we are still working on the out-of-the-box user personalization experience (should be done within the next month though). That being said, I'll be so bold as to offer to work with you over the course of a week free of charge to get something customized for your life (as well as to demo my own Mentations dashboard to you) as long as you have a decent Windows machine to run the Mentations software on!
Comment by Brian Schneeberg — October 12, 2009 @ 10:52 pm
Brad, I've been thinking a lot about this one as well. I have even put into my bucket of "Next Startup Ideas".
However, my problem/solution is much simpler than that. I want something that can cut 15-20 minutes of my daily routine and make sure I don't miss important inflection points on things that I care.
On my case I check Feedburner readers, Twitter followers (on multiple accounts), Google Analytics, Compete, Alexa, Google PageRank/SEOs-metrics, etc. Each of those for each of the sites/projects/businesses I care about. I'd love to open a Daily Dashboard that shows all that info into a single page and flags inflection points/large deviations for me to drill down a bit.
Comment by @calbucci — October 12, 2009 @ 11:49 pm
Sounds like a job for Google Wave…
Comment by Scott Loftesness — October 13, 2009 @ 2:13 am
I thought filtrbox was doing a lot of that for you…
I agree with the other Scott, I bet there's a Google Wave solution coming down the road for you, and for all of us.
Comment by Scott Yates — October 13, 2009 @ 2:50 am
Brad,
I would agree @RobGoldman that it depends on how structured the data is. If it's stuff that follows a predictable format (ie. can be scraped) check out Dapper.net. That's the acquisition piece of the equation that allows you to basically make an API for any web page. For the presentation side I second @Aaron Fulkerson's recommendation of Mindtouch. You can also do a lot in terms of data operations and presentation with Yahoo Pipes and Google Spreadsheets. Both can massage data and consume and generate XML (which Dapper.net is also able to output). I used Dapper to reformat Hacker News and give me a sortable way to read it the way I like: http://www.hnsort.com
Sean
Comment by Sean Tierney — October 13, 2009 @ 4:00 am
Brad, this is the digital nirvana question, but I would suggest that what you think you want, isn't actually going to be what you want…at least not yet. (Certainly in terms of investment areas!)
I have exactly the same dilemmas, and suspect we could both use existing tools to facilitate most of the above. There is one key issue, though highlighted of late, which is 'trust'. I only have the privacy of one business (about to launch a highly disruptive model and with some key players) to worry about, but you have many. Are you really confident in a future where all your dialogue, social interaction, facts and figures are cloud hosted by one provider? I'm certainly not. It is that reason I stick with Exchange not Gmail and store my financials et al locally.
I think the next big business model is for the provision of a tangibly private hosted solution, perhaps with a new universal encryption model which convinces me as a consumer that all my data can only be accessed by myself or those I choose. Only then might I be tempted to lock my life into one dashboard. Maybe Dell et al, could have a new lease of life, providing personal servers, which operate in a cloud environment, protected by firewalls and multiple encrypted offsite back ups. In other words, the box becomes your personal privacy wall.
The other big thing is filtering, which again I think needs to come first. Tons of news and entertainment sources will inevitably die and then a handful of feeds will provide personally filtered versions of what matters to us. We are working on that at Famebook, where apart from our 'famous people & famous brands' tag, we are also focused on subjective and not objective content, which we believe is the next huge shift in media.
Hope that makes sense…
Comment by Jan Simmonds — October 13, 2009 @ 10:57 am
Having spent some time with Google Wave, I find it completely incomprehensible. I know that I don’t yet get it and I’ll keep trying, but it doesn’t feel anywhere close to addressing this issue.
Comment by Brad Feld — October 13, 2009 @ 11:19 am
Per my note to Scott, Google Wave just doesn’t seem like a fit. Filtrbox is one of the 14 tabs and gets me a lot of the “news”, but I’m trying to deal with way more than just the “news.”
Comment by Brad Feld — October 13, 2009 @ 11:19 am
Dapper.net looks really interesting. I hadn’t ever really paid attention to the underlying tech (I knew the ad app, but not the data manipulation open source project). Thanks – definitely on the right track.
Comment by Brad Feld — October 13, 2009 @ 11:22 am
Makes perfect sense. Security is a huge issue for me also for exactly the reason you state. We also have a documentation retention policy (everything goes away on a rolling 12 month basis) so I never have anything that is older than 12 months (nor do I really need it as I’ve discovered). Ultimately I think this is a solvable issue but it requires a lot more effort to be scalable and bulletproof (vs. a one off solution for a dude like me).
Comment by Brad Feld — October 13, 2009 @ 11:52 am
Aaron – as you know I’m a huge fan. I’d be happy to play around with you with this and see if it can do it. Are you going to be at Defrag? If so – let’s sit in the corner and hack for a while. If not, we’ll figure out another way.
Comment by Brad Feld — October 13, 2009 @ 12:51 pm
I took a look at http://indicee.com/ and it definitely seems relevant for Excel. I’ll play around.
Comment by Brad Feld — October 13, 2009 @ 12:53 pm
Yup – agree. This was the basic issue with EIS’s back in the 1980’s (and every other iteration of this). You have to start with your strategy and then find / build the tools.
Comment by Brad Feld — October 13, 2009 @ 12:54 pm
Sort of true, although I have a lot of daily transaction data that I care about (daily status, user metrics by company) that I care about. From my last look at Threadsy, you seem to address the same category of stuff that Gist addresses, which is some of it, but only some of it.
Comment by Brad Feld — October 13, 2009 @ 12:56 pm
Definitely keep me posted on your progress.
Comment by Brad Feld — October 13, 2009 @ 12:59 pm
Great example – yes – it’s not just my little corner of the universe.
Comment by Brad Feld — October 13, 2009 @ 1:03 pm
Send me the link – I’d be happy to take a look.
Comment by Brad Feld — October 13, 2009 @ 1:10 pm
Yes!
Comment by Brad Feld — October 13, 2009 @ 1:16 pm
It's really exciting to read about your information consuming routine and the need for a better solution because that's what i have been planning to design at Cognizr.
While most of the products/services on offer are aggregators of xml, the real solution for this will come from the use of semantic technology particularly for the ingestion layer to reduce the information overload upfront. For the presentation layer, i think a spatio-temporo-content aware flexible interface native app would be much better than a browser based one.
Comment by Dharm — October 13, 2009 @ 1:56 pm
Awesome. Happy to play around whenever you are ready.
Comment by Brad Feld — October 13, 2009 @ 2:00 pm
I'm having a hard time understanding your a-d points and why a 'dashboard' is going to solve them, could you break down a specific use cases of how a 'dashboard' is going to do to help you manipulate your data more effectively than current tools?
Comment by Kyle — October 13, 2009 @ 5:50 pm
A few months ago, a friend and I threw around some ideas for something along these lines. Our sticking point was a third layer you didn't mention, an output layer. Most of the advantages of a dashboard disappear if after reading something in the dashboard you have to go somewhere else to comment or reply to something. It's probably nothing insurmountable, but it seemed to be the most difficult of the tasks that we were looking at.
Comment by Matt_Cullen — October 13, 2009 @ 6:28 pm
[...] My Quest For A Personal Dashboard (tags: productivity information) [...]
Pingback by Knowtu » links for 2009-10-13 — October 13, 2009 @ 6:05 pm
Totally agree with the need for the output layer.
Comment by Brad Feld — October 14, 2009 @ 2:51 am
Right now I have to use a variety of different UI / apps to look at for each chunk of data. In addition, I don’t get any trending data – each “data” is a once in time snapshot of the data. Re: a use case, I guess I didn’t do a good enough job in the post – I’m not sure how else to explain it.
Comment by Brad Feld — October 14, 2009 @ 3:29 am
Interesting – a dashboard that would save High net Worth individials 15- 20 minutes per day. The challenge being what would a High Net worth individual pay for such a service, how to target them – being those that have a specific challenge of mutliple forms of data including financial, non financial structured and structured and time sensitive and just plain sensitive material.
Working back from how many & how much would be interesting.
Comment by david — October 14, 2009 @ 5:37 am
[...] love of data visualization and the quantified self. Brad Feld has a post detailing his quest for a Personal Dashboard. I’ve thought about this for a while now myself and have had visions of a few items all [...]
Pingback by Personal Dashboard | Smart Home › joshua paul premuda | man of leisure — October 14, 2009 @ 7:49 am
at least you find time to reply to all the comments on your blog =)
Comment by JChauncey — October 14, 2009 @ 3:52 pm
Hah! I try.
Comment by Brad Feld — October 14, 2009 @ 4:13 pm
Brad, you might want to give http://www.Gathr.me a look. It combines your networks, feds, friends, communications, media into one site, without making you learn a new interface.
Comment by Michael — October 14, 2009 @ 4:57 pm
Thanks – I just signed up for the private beta.
Comment by Brad Feld — October 15, 2009 @ 5:43 am
Have you tried Gist? It's awesome for this kind of stuff.
Comment by @marmarko — October 15, 2009 @ 8:06 am
I love Gist. So much that I'm one of the investors. It is one of the 14 daily tabs but it doesn't yet dealnwith some of the information that I want to ingest.
Comment by Brad Feld — October 15, 2009 @ 11:38 am
I've personally created a dashboard for my business which has all the metrics that I want to know about all areas of my business. It's still under development, but the idea is to narrow down to the metrics that will enable me to know how things are going, and then check into that area of the business if needed.
I'm personally using google spreadsheets for this dash board, not because I particularly like them, but because they are useful when multiple people in the company are adding data.
In terms of daily reports, i'm developing software which will send you a report from everyone in your company that reports to you with: A list of their priorities, a list of exactly what they did for the day, and how many hours they worked, and what percentage of time they worked on their priorities. The key being that it's one email sent once per day for your entire organization, so that you can see in one email 10 or 20 people, and it doesn't take so long because of the presentation/ organization of the information.
Comment by Rob Rawson — October 24, 2009 @ 7:30 pm
Nice! I’d love to see a demo of it when it’s done.
Comment by Brad Feld — October 24, 2009 @ 8:34 pm
The ultimate control over presentation can come from a desktop application. Everyone says desktop is dead, web is where everything should be now and in the future, and while I mostly agree, desktop will continue to serve very specific needs for a long time to come. That may be something to consider in light of not being happy with some of the results you've seen thus far with web based dashboard systems.
Comment by Club Penguin — October 25, 2009 @ 2:46 am
Nice post on web scraping, simple and too the point
, I use python for simple html web scraping, but for larger projects like the web, files, or documents i tried website scraping which worked great, they build quick custom screen scrapers, web scraping, and data parsing programs
Comment by Candice — October 29, 2009 @ 8:51 pm
[...] talk a little about how I use Gist since I’ll use it every day to help me deal with a shortened daily information routine. As you probably know, we are investors in Gist and I’m on the board so – while this is [...]
Pingback by How I Use The Gist Dashboard View — November 17, 2009 @ 4:06 am
[...] talk a little about how I use Gist since I’ll use it every day to help me deal with a shortened daily information routine. As you probably know, we are investors in Gist and I’m on the board so – while this is [...]
Pingback by How I Use The Gist Dashboard View — November 17, 2009 @ 6:48 am