I spent the last two days at the Defrag Conference.  It was awesome on so many levels including the content, the venue, seeing a bunch of great friends, and meeting a bunch of new people.

The conference originated out of an email exchange that Eric Norlin (the amazing guy who puts on the Defrag and Glue Conferences with his even more amazing wife Kim) and I had as a result of a series of blog posts that I wrote in 2006 starting with There Is A Major Software Innovation Wave Coming and Intelligence Amplification.

Over the past three years there has been an incredible amount of innovation around this theme (which we originally called the Implicit Web.)  While lots of it is still messy, sloppy, or ineffective, that’s just part of the innovation cycle.  A consistent discussion point at Defrag was “how to deal with this overwhelming amount of information” – there is no debate about the (a) amount, (b) need to deal with it, or (c) value of dealing with it.  However, a lot of the subtext was that there was too much information and we needed better ways to deal with it.

I agree with the conclusion, but not the premise.  I don’t think there is too much information.  I want more.  More, more, more, more, more.  MORE.  I don’t want to stop until I have all the information.  MORE!  You can’t give me too much information!

I don’t believe the issue is too much information.  This is an independent variable that we can’t control.  For the foreseeable future, there will be a continuous and rapid increase of information as more of the world gets digitized, more individuals become content creators, more systems open up and provide access to their data, and more infrastructure for creating, storing, and transmitting information (and data) gets built.

Yeah – I know – that’s obvious.  But there are a few ways to approach it.  My desired way is to accept the thing you can’t control (more information) and drastically improve the methods for consuming it.  I spent the better part of two days having this thought over and over again. 

By the end of the second day, I’d decided that my original premise was correct – there continues to be a huge innovation wave in software that addresses this.  And we are just starting to deal with it.  And while software is at the core of it, we’ve learned an enormous amount in the past few years about the power of people to help curate it, both directly (by doing things to it) and indirectly (by software interpreting the broad signals of what a large number of people are doing to it.)

The user interfaces – and user interaction model – for all of this stuff still sucks rocks.  And I love things that suck, because that creates huge opportunities for innovation.

A few weeks ago I wrote a post titled The Maturing of the Implicit Web.  In it I talked about new releases from AdaptiveBlue and OneRiot.  As I sit here in my hotel room in Seattle waiting for TA McCann (CEO of Gist) to show up for our pre-board meeting run, I’m pondering how much work I’m starting to get the web to do for me.

For example, as part of my morning information routine, I go through my Gist dashboard.  This is a list of all the new information that Gist has found from a wide variety of data sources about people and companies in my social network.  It derives the social network from my email inbox, integrates it with my Facebook, LinkedIn, and Twitter social graphs, and then presents it to me in a way that is prioritized by what it thinks I find most interesting.  The level of relevance to me is amazing now that I’ve had it running for a few months.  While Gist is still in closed beta, if you want an invite just email me.

Gist is synthesizing the data for me from a variety of other web services.  At our board meeting today we have a long list of potential partners and data services that we prioritize based on (a) quality of data, (b) availability of data, and (c) ability to integrate the data.  Exactly one year ago I wrote a post titled No API?  You Suck! I feel so strongly about this; if I wrote the post again today I would have titled it “No API?  You Really Suck!”

One of the data sources that has a strong API layer is our company Gnip.  They recently announced that they will be integrating data from the Facebook Platform into their data set.  This comes on the heals of their announcement about adding WordPress as a data publisher to their system.  Gnip now has over 30 data publishers actively flowing through their system and have found rapid adoption from a number of interested customers.  Oh – and Gnip’s API is well documented, public, and evolving rapidly.

So – it was with great pleasure when I saw Alex Iskold’s announcement that there is now a Glue API. There is a tremendous amount of interesting semantic data in AdaptiveBlue’s Glue system – the API liberates it for anyone that wants to put some energy into working with it.

But wait, there’s more.  OneRiot also just released their API, which – while not public yet – is available by request.  OneRiot has a fascinating set of real time data available via a search interface that gets better and more relevant every week.  They’ve also demonstrated that they can build to search scale as they have some superb technical search folks on their team.

Gang – thanks for not sucking.  Y’all are helping set things up so the web does more of the work for me!

I’ve been fascinated with the notion of the Implicit Web since I determined that I was tired of my computer (and the Internet in general) being stupid.  I wanted it (my computer as well as the Internet) to pay attention to what I, and others, were doing.  Theoretically “my compute infrastructure” should learn, automate repeated tasks (automatically), figure out what information I actually want, and make sure I get it when I want it.

In 20 years, I expect we will snicker at the idea of having to go search for information by typing a few words into a text box on the screen.  It’s way better than 20 years ago, but when you step back and think about it, it’s pretty lame.  I mean, I’ve got this incredible computer on my desk, a gazillion servers in the cloud, this awesome social network, yet I find myself typing the same stuff into little boxes over and over again.  Ok – it’s all pretty incredible given that it wasn’t so long ago that people had to rub sticks together to get fire, but can’t it be amazing and lame at the same time?

Several companies that I’ve got a personal investment in that play in and around the implicit web recently came out with new releases that I’m pretty excited about; each addresses different problems, but does it in elegant and clever ways.

The first – OneRiot – came out with a new twist on using Twitter for search.  OneRiot’s goal is to provide a search engine for the real time web.  To that end, they’ve historically gotten their data on what people are looking at from a collection of browser-based sensors (anonymous, opt-in only).  They’ve built a unique search infrastructure that takes a variety of factors, including number of people on a specific URL in a particular time period, freshness of the content, and typical content weighting algorithms.  A little while ago they realized that people were tweeting a huge number of URLs, mostly via URL shorteners (which are loathed by some very smart people.) Twitter search addresses keywords in the tweet, but it doesn’t do anything with the URL’s, especially the shortened ones.  So, OneRiot built a pre-processor that grabs tweets from Twitter’s API that include a URL, tosses the shortened URL into OneRiot’s search corpus (which expands the URL and indexes the full page text), and then references it back to the original tweet.  It also correlates all tweets with the same URL (including re-tweets) across any URL shortened service.  Now, imagine incorporating any URL data that’s real time that has an API, such as Digg.  Aha!  It’s alpha so forgive it if it breaks – but give it a try.

The second – AdaptiveBlue – has released their newest version of Glue.  Glue is a contextual network that uses semantic technology to automatically connect people around everyday things such as books, music, movies, stars, artists, stocks, wine, and restaurants.  It uses a browser-based plugin to build this contextual network implicitly.  When you are on a site such as Amazon, Last.fm, Netflix, Yahoo! Finance, Wine.com, or Citysearch, the Glue bar automatically appears when it recognizes an appropriate object, categorizes it, and let’s you take specific action on it if you want.  Glue has been evolving nicely and now includes the idea of connected conversations between friends (e.g. talk about whatever you like regardless of the site you are visiting), smart recommendations (e.g. implicit recommendations), and web wide top lists of the aggregated activity of all Glue users.

In addition, we’ve finally found a company that we think is attacking a wide swath of the problem of the Implicit Web the correct way, at least given today’s technology. We hope to close the investment and start talking publicly about it early next month. 

For now, I expect the applications around the Implicit Web to continue to fall into the early adopter / you need to see it to believe it category (where it’s harder to explain than just to show).  In the near term, if you are interested in this are, try out OneRiot and Glue – they are both evolving and maturing very nicely.

Josh Kopelman doesn’t blog that frequently, but almost all of them are worth reading carefully.  His latest post – Feed Frenzy – is great.  Josh is facing the "multiple news feed problem" as he joins more and more services that publish a news feed.  He takes on the notification side of the equation – the opposite of what FriendFeed and SocialThing do.

All of the social network sites continue to use email as a notification mechanism.  When something happens on the social network that pertains to you (including messages), you get an email.  Anyone that has a meaningful volume of social network activity quickly learns how to turn these notifications off.  This defeats part of the real time value of social networks – now I have to go check and see what’s going on to see if anything relevant to me has happened.

As the "too much email" meme continues to circulate, someone is going to realize that one of the drivers of it is the endless notification cycle and the least common denominator – namely email – that is the mechanism for the notifications.

The solution – as Josh points out – is analogous to SNMP and network operations.  Josh wants an SNMP enabled dashboard for all his news feeds.  Aggregate everything into one easy to monitor dashboard, take action automatically on critical things that I’ve told the dashboard it can take action on, and organize the rest of the notifications in a way that I can deal with.

As an extra special bonus, this dashboard would help me connect all the atomic data (namely – my friend data) on the various social networks I’m getting news feed data from.  Fred Wilson would be "Fred Wilson" across twitter, his blog, his tumblr, facebook, linkedin, myspace, disqus, intense debate, etc.  I’d be able to interact with "Fred Wilson", not each of the discrete Fred Wilson’s.

There was a moment in time where I thought RSS might be the solution for this.  But it’s not – there’s a second order problem (and opportunity) here that requires something additional, especially given that new API’s are appearing for handling specific services news feeds. 

Stuff like FriendFeed and SocialThing address part of the problem, but not all of it (and – ironically – often create additional data as anyone who was been notified by email that a new friend has signed up to follow them on FriendFeed has discovered.)

I love recursive problems.

My partner Ryan put up another post on the Foundry Group blog about our Implicit Web theme.  This is a theme that is particularly interesting and useful to me since I spend so much time in front of my computer trying to deal with and synthesize a never ending stream of data and information.

I want my computer to do more of the work for me.  I want the web to figure stuff out for me.  I want my computer to learn.  I want my friends’ behavior and interests to inform mine.  I want people I don’t know but am connected to through other people I trust to help me find things.  I want a pony.

A couple of years ago my path crossed with Eric Norlin.  Out of our interactions emerged a conference called Defrag.  The first version of Defrag happened last fall – as is typical of my schemes, I always have an evil plan.  In this case is was to get in the middle of a bunch of really smart people and hear what they think about the wide range of problems we addressed at Defrag (which was a proxy for the concept of the Implicit Web.)  Eric talked about this in his post today titled On Community-driven tech conferences.

My evil plan worked great and I learned a ton.  We’re doing Defrag 2008 in – surprise – 2008 (11/3 and 11/4 to be exact.)  I’m continuing to spend a lot of time looking at, thinking about, and investing in the Implicit Web between now and then.  And reading as much Alex Iskold as he will write.

If you have been following Defrag from a distance, Sean Ammirati at Read/WriteWeb has an awesome summary up titled Five Themes From the Defrag Conference.

1. The Attention Economy
2. Next-Level Discovery
3. Social Graph
4. User Centric Identity
5. Enterprise Customers Arrive

In addition just summarizing what was covered, Sean has very insightful paragraphs after each one under the heading “My Take.”

Also, a very thorough summary of many of the sessions was blogged by Charles Knight at AltSearchEngines.

John Markoff suggests there is a looming battle of semantics between Web 3.0 and semantic web.  Markoff summarizes by saying:

“There is no easy consensus about how to define what is meant by Web 3.0, but it is generally seen as a reference to the semantic Web. While it is not that much more precise a phrase, the semantic Web refers to technology to make using the Internet better by understanding the meaning of what people are doing, not just the way pages link to each other.”

I’ll take it up a notch.  Let’s toss “implicit web” into the mix and expand it beyond just “what people are doing” and “the way pages link to each other.”  It’s not only about people and pages – it’s about getting the computer to figure out a lot of stuff for you based on what you – and people that you “trust” or “find relevant” do.  You could be cynical and just assert that this is the way our friend the “social graph” finds its way into the mix of Web 2.0.  Or – you could keep playing – the way we are planning to at Defrag – and say something all encompassing like:

“Defrag is the first conference focused solely on the internet-based tools that transform loads of information into layers of knowledge, and accelerate the “aha” moment. Defrag is about the space that lives in between knowledge management, “social” networking, collaboration and business intelligence. Defrag is not a version number. Rather it’s a gathering place for the growing community of implementers, users, builders and thinkers that are working on the next wave of software innovation.”

It doesn’t really matter what you call it – a wave of startups tackling “the next problem” are starting to appear.  Look for a panoply of new and exciting buzzwords – and look past them to the meat.

John Markoff has a fun little story about the origin of the phrase What-You-See-Is-What–You-Get in his article titled The Real History of WYSIWYG.  WYSIWYG appeared on the scene 30+ years ago – it seems like the wikipedia page on it needs a little updating.

Now that WYSIWYG is part of our collective consciousness (and a deep expectation of most computer users, even though there are many apps that don’t quite cut the mustard), I’m less concerned with WYSIWYG these days. I’m more intrigued with WYWIWYG – “What-You-Want-Is-What-You-Get.” 

Come to Defrag on Nov 5 and 6 in Denver and talk to me (and a bunch of other smart people including folks like David Weinberger, Clay Shirky, JP Rangaswami, Jerry Michalski, Ester Dyson, Doc Searls, Bradley Horowitz, Paul Kedrosky, Chris Shipley) and a long list of others about WYWIWYG.

You know you’ve been thinking about registering, but your computer won’t do it for you (at least not yet.)  You still have to click here and type some stuff in.  Maybe someday WYWIWYG.

In his post 30 Thoughts At 30,000 feet, Fred Wilson referred to Alex Iskold as “a freak of nature.”  Fred supports this by saying “He writes code, runs a company, and does amazing blog posts for Read Write Web that are better than most Gartner research reports. I’d ask how he finds the time to do all of that, but I know the answer. Full disclosure – Alex’ company, Adaptive Blue, is a Union Square Ventures portfolio company.”

I’ve gotten to know Alex over the past year (I’m also a small investor in Adaptive Blue) and I’ve begun referring to Alex as “the big brain.”  Almost all of our communication is via email (I see him every now and then) and I’ve started envisioning him as a gigantic brain suspended in some funky liquid with a cable coming out of it that is connected to a computer (if I was a little better with Photoshop I might even draw you a picture of it.)

As Fred says, Alex “does amazing blog posts for Read Write Web that are better than most Gartner research reports.”  I agree and read each of them carefully as part of trying to increase my cognitive reasoning functions around the theme I call The Implicit Web.  Alex’s post today titled The Structured Web – A Primer is another excellent one.  In this post Alex does a great job of linking together four things that I spend plenty of time thinking about – APIs, semantic information, microformats, and RSS. 

Keep them coming Alex.

Scoble has a 20 minute interview up with Eric Norlin discussing Defrag and other conferences that Eric has helped start.  If you haven’t heard of Defrag, it’s a new conference being organized by Eric taking place in Denver on November 5 – 6.  The sound bite overview is:

Defrag is the first conference focused solely on the internet-based tools that transform loads of information into layers of knowledge, and accelerate the “aha” moment. Defrag is about the space that lives in between knowledge management, “social” networking, collaboration and business intelligence. Defrag is not a version number. Rather it’s a gathering place for the growing community of implementers, users, builders and thinkers that are working on the next wave of software innovation.

In short – “the Implicit Web.”  The agenda and lineup of speakers is incredible.  Eric has done an incredible job and has been blogging furiously about it.  

Now that we are less than a month out, it’s time to register if you want to join us.  The conference will sell out because we’ve capped the total number of attendees so sign up now and come play.