"Real-time" is for machines. That's a thought rolling in my head today. I'm thinking of possibly doing a long post some time soon, but in the meantime, I'll share a few "random" thoughts.
Professionally, I design and develop real-time trading systems. I see strong parallels between the evolution of real-time trading systems on wall street and evolution of the real-time social web.
The social web is at an inflection point, as it deals with "real-time". "real-time" search in Twitter. "real-time" aggregation in FriendFeed and so on. There appears to be enormous value, but we're struggling to harness that value, and it's quickly overwhelming us.
I'll begin with a very brief evolution of real-time trading systems.
Trading
In the beginning, users are amused by real-time data. In fact it does add a lot of value. We show real-time prices in trading systems, User reacts, points, clicks... trades.
Users realize the advantages of its immediacy, so naturally they want more. Follow more companies, get more prices and other company data in real-time, trade more etc.
At some point there are too many companies, too many prices, too many decisions, not enough screen real-estate,and slow humans:)
Now we introduce the machines... In my case, trading systems that consume massive amounts of real-time data, make decisions, while hiding the complexities and details.
Social Web - same path
For the social web, you might start with Twitter Search. You follow a hash tag or two (2 companies). Or maybe you follow a few friends.
Seems interesting, so you want to track the same hash tag in other social sources, say Delicious. Or maybe you want to follow more people.
You're impressed so you want to track more tags without compromising speed. No one wants stale data.
Suddenly, you find yourself following too many topics, too many people, you can't decide which conversations to join etc. You can't possibly process each bit, tweet, message...
So you turn to the machines.
But now you're
further away from the humans... and therefore anti-social.
Is this bad? Not necessarily. Moreover, it doesn't have to be anti-social. My experience in finance shows me the way.
Algorithmic trading systems do 99% of the heavy lifting. They trade behind the scenes according to rules that react to real-time information. They're effectively complex event processing engines. Now this is
key. Great Algorithmic trading systems are designed to interrupt humans when they need help. In other words, where humans add value. These would be "exceptions" or situations where there's a lot of ambivalence.
Possible solution and future of Real-time Social Web
The Real-time social web will, at some point, need it's version of an "algorithmic trading system". Effectively, hiding but still processing all the real-time information of interest... then interrupting you when it needs your help or when you can add value. Some of this is happening today with tools like Twitturly, that aggregate, filter and rank stories/links mentioned in twitter.
Like trading engines, social engines might have "behind the scenes rules" and "exceptions"
Behind the scene rules
- Retweet link every 2 hours until 25 percent of my readership has read my post
- Tweet link only after 10 percent of my followers are online, based on their last tweet times
- Introduce followers that ReTweet my links
- Retweet old stories during down times
- Tweet posts with tags that match trending topics
- Retweet until your post gets a comment
Exceptions
- Notify me to create new post if ReTweet count is too low
- Have me choose a specific post to tweet if too many posts tags match trends
- more than 5 people ask me a question that matches keywords
- let me know which one of my followers is posting the most
- Notify me when when my 10 favorite friends are online
- Alert me when any of my friends mentions a vacation or wedding
The Technology might already exist where Trading and Social Web intercept - "News"
Interestingly, the technology exists, albeit, too expensive for the average social networker... but probably priced right for corporate brands. Financial Services shops already use complex event processing engines like StreamBase and Progress/Apama to process price signals but also News stories in real-time. Processing news is a really tough problem that I won't get into here. However, I have a hunch that harnessing the real-time social web is this way is actually "easier" and less "risky".
Anyone want to team up on this bad boy? could be fun!
Afterword - Ultra High Frequency Trading - Ultra High Frequency Social Networking
In the past few years there's been an increasing focus on Ultra High Frequency Trading or Low Latency Trading in Finance. Sub milli-second trading. Faster data, Faster reaction times. The basic idea is that there's value in reacting first or taking advantage of sub milli-second behavior or phenomena. Things are developing quickly in this space. This could also be where the real-time social web will head toward. Perhaps I'll speculate on what the ultra high frequency social web might look like, in a future post.
What do you think about the Real-time Social Web?
