Humans, Not Big Data, Must Benefit From Opening Up The Banks

One of the 'good news' items in last year's Autumn Statement was the endorsement of an Open Data Institute report on how innovators could make better use of bank data on our behalf (instead of on the banks' behalf). Last week, the Treasury followed up with a call for evidence "on how best to deliver an open standard for application programming interfaces (APIs) in UK banking and... whether more open data in banking could benefit consumers."

This is important, because enabling our own machines to start crunching our financial data on our behalf is key to humans winning the race against Big Data.

'Open data' involves enabling public access to data by connecting data sets using uniform resource identifiers ('Linked Data') and publishing data about them in machine-readable formats. While the government and other public institutions have done a good job of opening up public data sets so far, our private institutions are lagging behind, as discussed here. That's because they benefit from making sure we know less than them about our own requirements - 'information asymmetry'.

The Midata initiative was the first to aim at restoring the balance in favour of consumers, by taking aim at energy utilities, telecoms providers and banks. Now the energy companies face being opened up like so many tin cans by smart metering, while the telecoms market looks like it will go from bad to worse. Banks have also proved a tougher nut to crack. Yes, some people can download their current account data via internet banking. But in addition to consumers, there are 4.5 million small businesses out there that need a hell of a lot more than that. So far, it's taken regulatory action just to start the process of improving access to small business credit data and making a market in rejected small business loan applications. Now the Treasury is trying to force the pace on the technology necessary to support all that.

The ODI's recommendations for opening up banking are: 

• banks should agree an open API standard to support third party access to bank data - basically firms that can help you make sense of the data (but Big Data firms will try to persuade you to share it with them too);
• independent guidance should be provided on technology, security and data protection standards that banks can adopt to ensure data sharing meets all legal requirements; 
• an industry wide approach should be established to vet third party software applications and publish a list of vetted applications as open data - this would allow visibility of firms that are acting on consumers' behalf and those who are not;
• standard data on Personal Current Account terms and conditions should be published by banks as open data; and 
• credit data should be made available as open data.

My main concern is that requiring agreement on 'standards' as a precondition for opening up banking will enable the banks to delay the whole process by a decade - as they did with Faster Payments. As was recommended by the security working group in the Midata initiative, I would prefer to see banks required to immediately make their data available to each consumer/SME in whatever open format the banks choose, while adhering to common data security protocols, and leave it to the open data community to figure out how to re-format and display whatever rubbish they dish up.

Apart from complaints by the banks, Treasury officials expect to hear positive contributions from consumer groups, other financial services providers, financial technology firms and app and software designers.

Let's not disappoint them. This is a good opportunity to ensure the government clears the way for innovation that puts you and me at the heart of financial services, without mistakenly creating further barriers in the process.

The Future Is Not Behind Us, It Lives Locally

For every old saying there's an opposite. Today's conflict lies in the adage that "those who don't know history are destined to repeat it." Yet "our history is not our destiny." This leaves a fine line between useful historical insight into how the world works and steering solely by what we see in the rear-view mirror. The distinction becomes critical as we lurch ever more quickly from one financial crisis to the next.

So what lessons from recent history will help us move foward, and which should go in the scrapbook? Here are 5 that I think are worth taking forward, in no particular order:

1. Clearly, the Internet is not a fad. Consumers are the winners, aided by facilitators in their battle against our creaking institutions. Yet e-commerce is still only 10% of all retail. And we are still in the 'primordial soup' phase in the evolution of our tools for extracting meaning from the great, chaotic swirl of data. So this trend has a lot more mileage in it yet and will sweep across more service sectors - if you can buy it online and have it delivered it won't be sold in high volumes on the high street. In fact, it might even be made locally...

2. In addition to democratising services, the Internet is also returning the means of production to local communities through e-enabled machines. Remote/home-working is replacing central workplaces and 'factories' are getting closer to customers, requiring a rethink of corporate systems, processes and supply chains. Huge production plants and sole-occupancy office towers will gradually become a thing of the past. High streets may well regenerate to support this trend, or as a result of it.

3. Cities with at least 3 private workers for every public sector worker see the most growth. Give that some thought if you live in a city near the bottom of this chart. This chimes with findings that economies (regional and national) whose public spending exceeds 35% of their GDP struggle to grow. You have to feel sorry for Northern Ireland...

4. The public sector is very inefficient and needs to act locally - public institutions are already being relentlessly affected by lessons 1 and 3 above, and 2 should increase  the pressure. Civil servants really have no alternative but to spend public money more wisely. Meanwhile, we'll help drive down the cost of government by dealing with the government online. Big government offices must surely go eventually. Fortunately, the moves to devolve more power to local government coincide here, but I don't think we should be fooled into thinking that's part of any real plan to future-proof the UK...

5. The UK's financial system is seriously inefficient at allocating money to people and businesses. The fact that we all rely heavily on a few major banks for whom lending to small businesses is not a core activity is part of the problem, but innovation and competition are constrained by our outdated and rigid regulatory framework and the related incentives. Crowdfunding, or peer-to-peer finance, platforms are springing up all over the country, and are increasingly focused on specific sectors, activities and locations.

In short, if I were a civil servant (see 4) based outside the south east of England (3), I would start an Internet-based service (1) that efficiently provides low cost finance (5) to help localise the means of producing stuff using e-enabled machines - or buy my own 3D printer and start renting it out (2). Now. 

Image from KC Anderson.

Will Consumer Transaction Data Drive New Online Marketplaces?

I should begin this post by explaining that I'm involved in the Interoperability aspects of the midata programme. I was invited to participate on a voluntary basis and have no client in that process. I donate my time. I'm independent of the dozens of other participants. I didn't shoot JFK. And I don't even own a pet, let alone one I believe to be a reincarnation of Elvis Presley. 

The only conspiracy in which I might be accused of involvement is the mass collaboration by consumers known as Web 2.0, which has evolved into Web 3.0 using "linked data" that computers can read (aka "the semantic web"). Lots of information - including government data and data from online bank accounts - is now available in this format because it makes analysis so much easier. Analysing the data is necessary to convert it from being merely information into useful knowledge. This is why the government is keen that banks, telecoms providers and big energy start making all your transaction details available to you in machine-readable form - if you want it. Gaining insight into your finances, communications and energy use will enable you to make better spending decisions and even negotiate new, bespoke products - if you want to do so.

In an explanatory post on the Which? website, Consumer Affairs Minister Jo Swinson sparked a number of comments by people who do not want to be empowered or have their consumer experience made more efficient. They are happy to make their own product searches and to use price comparison services. Evidently, they are not concerned at the number of problems detected by the Office of Fair Trading in its work on the price comparison sector. They also appear to believe that information security and privacy safeguards around existing transaction databases are adequate. I am not amongst them.

It's perhaps unfortunate that this debate seems to be centring on price comparison services, when it's really the mainstream product providers who are to 'blame' for the lack of consumer bargaining power in key markets. In fairness these services have evolved into a prime marketing channel through being more nimble, internet savvy and committed to transparency than the mainstream product providers themselves. This has rendered the services useful to consumers up to a point, but they are limited by their deals with the product providers as to how far they can really empower the consumer. In this sense they occupy the battleground created by consumer rebellion and institutional resistance.

I have expressed my own frustration with the current model of price comparison service for many years. That so many engage in extensive television advertising campaigns tells you that being a price comparison service provider is a really great business to be in. But consumers are paying for those big advertising campaigns in the same way they're paying for vacuous bank advertising - through the price of the products they buy, which in turn generates commission and/or ad revenue for the comparison service providers. So, while these services are intended as a tool for consumers to use, it would be naive to assume that price comparison service providers are acting on the consumers' behalf. Product providers are paying good money to ensure these marketing channel ultimately work for them, not you. As a result, these services are not so much about 'price comparison' as simply 'comparative advertising'.

Critically, however, the current crop of price comparison sites also only rely on the entry and/or display of personal and product information in human readable form. They have you by the eyeballs, at least for as long as you're able to keep your eyes on the screen. Yet the product providers are able to rely on their computers mining and analysing a wealth of yours and other customers' transaction data to work out the most profitable product to offer you. True, in some scenarios - like insurance - this 'information asymmetry' appears to favour you, but product providers have giant data sets that can overcome any advantage you think you might have. And when in doubt they simply charge you more. In The Undercover Economist, Tim Harford shows how information asymmetry works to our detriment in buying a car - when the dealer has all the information - as well as in the market for health insurance, because we have the information and not the supplier.

So while a price comparison service might enable you to buy a cheaper traditional product from one provider versus another, you are by no means able to negotiate from a position of real knowledge about the product or price that's right for you.

This makes a human-readable interface an enormous waste of your personal time for the tedious yet important task of ensuring you spend your money wisely. Instead, our own computers should be analysing our transaction data and interrogating product providers' systems directly, not only to find a product at the right price, but to create the right product for the right price. This might be as simple as relying on your rate of energy use to always ensure you're on the right tariff, even if that means switching providers daily. Or it may open up opportunities for collaborative consumption amongst consumers with whom you share similar interests or behaviours.

Of course, most of us won't have the time, skill or resources to do all this for ourselves, anymore than we can build our own cars. So new intermediaries are springing up to store and/or crunch the data for us. I'm not going to name them because I don't want this post to be perceived as some kind of advertisement. These intermediaries have been variously called 'data stores', 'personal data vaults', 'personal information managers' and, most recently for the purposes of the Midata programme, 'midata stores' and 'midata service providers'.

The services provided by these intermediaries vary according to whether they just store, display and/or transmit the data at your request without otherwise processing it; whether they receive data from you or your current supplier; and whether they analyse the data or combine it with other data to produce a result on which you might rely to purchase an alternative product or change your behaviour in some way. 

These intermediaries need contracts with consumers that permit them to access the consumer's transaction data with relevant permissions restrictions, and which agree some form of remuneration. Such contracts would need to go further than the basic service terms of price comparison sites. You might allow them to receive a disclosed commission from a product provider, but at least this would be transparent to you. This marks the line between whether the intermediary is acting for you or the product provider, and there will need to be clarity on this point.


Some people say they don't want suppliers to store any of our transaction data. They want it deleted as soon as it's no longer needed. But it should be clear by now that your transaction history could be very valuable to you, and you should have the option of downloading and storing it and giving it to another provider. Most businesses insist on such 'data portability' when moving from one outsourcing service provider to another, for example, and this can be just as important for consumers and small businesses.

Identity and authentication are also important features of a world in which transaction data is being transmitted. Perviously, I've suggested that proof of identity should be momentary, based on much wider behavioural data than just the static datasets that fraudsters can replicate, and the data used to establish your identity should be discarded straight away, rather than held for re-use. But that is not to say that transaction data itself should be deleted.

The impact of all this on mainstream product providers cannot be underestimated. I've previously drawn the distinction between 'facilitators' who exist primarily to solve customers' problems and 'institutions' who exist primarily to solve their own problems at their customers' expense.  Clearly this new environment will favour product providers who are aligned with consumers' day-to-day activities rather than those who make life awkward because it suits their own profitability. A gulf may well begin to open between product providers as they are tested by this distinction. More confident consumers should be more prepared to spend money with facilitators than those faced with institutions they distrust.

However, I believe that this trend will most likely result in a series of digital platforms on which consumers and suppliers in various markets directly negotiate products and pricing in a transparent way, based on each consumer's transaction data, wherever that is stored. Such platforms have already arrived in so many other consumer markets as part of the Web 2.0 phenomenon, that it's only a matter of time that they arrive in the markets targeted by the Midata programme in any event. And it's only natural that consumers would want to leverage their own transaction data in that context.

When Will Control Truly Shift To The Consumer?

For those engaged in the process of empowering consumers, 2012 is already a fascinating year. So it was timely that a bunch of us met at Ctrl Shift's "Explorer's Club" to try to map the timeline for when 'customer relationship management' truly inverts and firms finally acknowledge their customers control them. 

The output of the session is being converted into an 'infographic' that will be available as a reference soon. In the meantime, here's an excellent drawing that Joel Cooper produced during the session to reflect the various themes:

Augmented Reality and Linked Data

I enjoyed a far-ranging conversation with @StGilesResident today, and am indebted to him for the reference to the following TED talk by Pranav Mistry. I do not know how I've missed this to date and am looking forward to seeing how this technology evolves with 'midata' initiatives to support our day-to-day activities in future. The folks at Microsoft have also been playing around here (you can follow them here).

So Many Targets, So Few Bullets

Okay, it's been a looong week, it's 2:20am, my cab leaves for the airport at 4:00 and I haven't packed yet. So there are two chances of a decent post this week...

In all fairness, the dazzling array of targets hasn't exactly allowed one to focus one's thoughts - I mean, I've been hammering away on the Tweet button like a chimp on speed. So all I can usefully do is commend that frenetic feed to you, as a parting gift.

But if I had to choose, these would be my top 8:

1. Zero Hedge: on whether the US and Europe will track Japan's 21 year slide (and counting);
   
2. Spectator: that Rebekah Brooks statement on phone-hacking in full;
   
3. Naked Capitalism: Partying on the Edge of the Eurozone Volcano - I love that title;
   
4. Naked Capitalism: Fukushima Cover Up Unravels;
   
5. Guardian: What Twitter thinks of the News of the World, visualised;
   
6. P2P Banking: The US Government Accountability Office's Guide to US Regulation on P2P Lending;
   
7. Schumpeter: On being sacked for telling tales 'through' the media; and
   
8. Chris Marsden's observations on what the Dead Parrot Sketch and UK Media regulation have in common.

And, hey, I have a whole hour left to pack. So time for a little light entertainment:

Linked Data: An Energy Credit Rating?

I received a love letter from EON recently, telling me they love me so much they're going to increase my direct debit. That way, I will "build up a credit over summer" and enable EON to spread its bill over the year.

Sadly, this is a case of unrequited love. I do not want to leave any surplus cash with EON, especially when they haven't mentioned paying me any interest on the credit - or even putting the interest toward my bill.

But on the upside, EON's letter did prompt me to call @Oikonomics, who is fairly heavily involved in an alternative energy proposition, amongst many other things. We got talking about Green Deal and feed-in-tariffs and the challenges of comparing the official option for financing your personal alternative energy generating/saving with alternative means of financing. One idea is to enable you to generate an "Energy Credit Rating" to use as a base for comparing the overall benefit to an alternative energy option. It could combine your MPAN billing number, linked to some data from the Met Office about average sunlight/wind in your area, local energy prices for what you buy and what you generate, the Feed-in-Tariff scheme, and your own credit rating.

Linking some of this data may also help reduce the complexity/cost associated with making energy price changes.

Oh, and Bruce also told me Ovo credits your bill with the equivalent of 3% interest on any credit balance (see under "Earn from Energy").

Image from Energy Saving Trust.

Mydata And Consumer Empowerment

On Thursday, work began in earnest on the 'mydata' project featured in the government's "Consumer Empowerment" strategy (see Better Choices: Better Deals, on which I posted earlier).

As with the overall consumer empowerment policy, the primary goal of the ‘mydata’ project is “to put consumers in charge so that they are better able to get the best deals for themselves, individually and collectively.” Achieving that involves enabling consumers to access information about their purchases, analyse it according to their own preferences and use that information to make better purchasing decisions.

As a long-time critic of the European Commission approach to facilitating e-commerce, I'm overjoyed the government is convinced that new legislation is not the best way to achieve consumer empowerment. Instead, it's relying on "a wide range of new programmes that have been developed in partnership with businesses, consumer groups and regulators" against a background of normal regulatory enforcement.

The mydata development work is being fostered through a series of 'boards', chaired by Professor Nigel Shadbolt. There's a Strategy board made up of private sector retail businesses, consumer bodies and government representatives; four Sector boards comprising representatives of suppliers in the energy, financial services, telecoms and retail sectors; and an Interoperability board of private and public sector representatives. The focus of the Interoperability board is on maximising synergies among the sector groups' work; addressing barriers common to all groups; maintaining a balance amongst key issues of practical utility, privacy, security and data portability; and offering suggestions on where ideas and solutions in one sector might link up (or mashup) with others to better reflect consumers' day-to-day activities.

I know what you're thinking, but these boards are not designed to be exclusive talking shops or symbolic meetings of 'the great and the good'. The idea is to be as pragmatic as possible, using the boards to draw in as many interested parties, ideas and resources as possible to achieve rapid progress. It's a fascinating challenge, and I'm thrilled to be helping out on the interoperability front.

There'll be plenty of project communications, of course, which I'll be doing my best to retweet etc, and send to people I know who may be interested or able to help. I also plan to share material that I happen across outside the project. For starters, I've included below the links to Sir Tim Berners-Lee's TED talk on the semantic web (or Linked Data) and a few of the items he mentions. I'm very interested to receive any comments, referrals, ideas etc you may have.

Sir Tim cites some key examples of Linked Data and its uses. DBpedia is the fascinating "community effort to extract structured information from Wikipedia and to make this information available on the Web." And here is the TED talk by Hans Rosling to which he refers:

Image from 1Million1Shot.