Open Data Spiders?

Since 2009 I've hoped that the semantic web - that is my computer dealing with suppliers' computers - would replace the need for price comparison sites. Following a discussion last night at the CtrlShift Explorers' Club, I'm confident that we don't have much longer to wait.

If suppliers publish their product data in computer-readable format, I could then programme an application or 'spider' to search the provders' open systems to find the product that's right for me - ideally a bespoke product assembled from a menu of optional components. This spider would use my personal data to conduct its search without disclosing that data to any product providers, at least until the time of purchase (and disclosure might not even be necessary then). It could also collect, say, public sector Open Data related to my desired activity, and analyse it in the context of my relevant personal transaction history. This could vastly improve my choice of car, holiday or home improvement and how it's financed. Or it could save me money by keeping me on the right energy or mobile phone tariff.

This is not about 'intent-casting' or 'demand-casting' in order to encourage suppliers to send me thousands of offers. My spider would not announce to the world that it's looking for anything. It would simply run around the web looking at openly available product codes and report its findings to me. Ideally, the product provider will have no idea that it's actually me who's looking until I make a purchase, if ever.

And I would not need to read any screens or physically enter any data until my spider reported its findings - or it could save me the trouble by calling my mobile.

In a machine-to-machine world, the marketing challenge is to ensure that anyone's 'spider' can always find your product data, and that data is accurate and up to date. Perhaps it could be somehow 'spider optimised', but it seems to me it's the job of the spider developers to make sure the spiders are good at finding product data, even when it's in a sorry state.

My sense is that an Open Data approach to the market takes such a different corporate mindset that it is unlikely to sit comfortably within traditional suppliers, where "Big Data" is the latest buzzphrase. In the Open Data world the challenge is to enable your products to be directly embedded in the ecosystem, helping to solve problems as customers encounter them and their machines or 'spiders' look for an answer. The traditional product approach is not 'connected' in that way, or at all. And, as I suggested recently, "Big Data" approach to behavioural targeting of advertisting seems fundamentally hamstrung by the fact that personal behavioural data is highly contextual and not really 'predictive' from one scenario to the next. Why spend all that money on what is ultimately a shot in the dark?

Those who ignore the Open Data option could well be spending their way rapidly into oblivion. 


Image from Data.gov.uk