'The Hydrogen Sonata' and the ethics of Big Data

Earlier this summer we lost Iain M Banks, one of the most imaginative writers in modern literature.  Living in Edinburgh and sharing a number of good friends, I was lucky enough to have met him on several occassions over the past 25 years or so, in bars (mainly), SF conventions (sometimes) and even bobbing about in a swimming pool. He was every bit as wonderful and generous a man as the many appreciations of his life have deservedly highlighted and I'll miss him very much.   

 

But Iain's significant body of work remains to inspire and I was reminded of a section from his latest, last Culture novel, The Hydrogen Sonata when I clicked through to a link about Predictive Analytics in the Cloud from the LinkedIn Big Data group this morning.

The article - which reads exactly as most over excited vendor press releases read and is concerned with a market I just know Iain would have the greatest disdain for - talks about tools which, and I'll quote directly from the CEO "can literally just say, 'Here's every event that happens in the world.' Earnings, economic indicators, seasonality, cyclicality, political events, and so on'...And they can literally model every single stock in the AMEX and the NYSE in relation to every single event, and can gain that kind of precision around it, which obviously helps them in terms of making their investments."

In The Hydrogen Sonata, Banks writes about an even grander ambition ambition.  The story muses about the challenges encountered when trying to predict how an entire civilisation will react when confronted with evidence which may reveal their founding myths to have been a lie.  Other civilisations - including Banks' great utopia, The Culture -  have attempted a particular simulation modelling technique to try and predict the reaction but these have all been found wanting:

"The Simming Problem boiled down to, How True to life was it morally justified to be?"

A longer history of the challenge is also presented: 

"Long before more species made it to the stars, they would be entirely used to the idea that you never made any significant societal decision with large-scale or long-term consequences without running simulations of the future course of events, just to make sure you were doing the right thing.  Simming problems at that stage were usually constrained by not having the calculational power to run a sufficiently detailed analysis, or disagreements regarding what the initial conditions ought to do.

Later, usually round about the time when your society had develped the sort of processal tech you could call Artifical Intelligence without blushing, the true nature of the Simming Problem started to appear.

Once you could reliably model whole populations within your simulated environment, at the level of detail and complexity that meant individual within the simulation had some sort of independent existence, the question became : how god-like, and how cruel, did you want to be?"

These considerations develop over several pages.  To simulate life, life must first be created to the extent that it recognises itself as life, leading to the thought that we might all be in a simulation ourselves.  All fantastically, entertaining stuff but it's perhaps not surprising that Banks' Culture Minds end up concluding that "Just Guessing" is ultimately as effective. 

What I believe will endure about this passage is not just the wit and vision but the necessity to assess the ethics of data usage.  OK, 'simming' life is not where we are right now but larger and larger data sets are being use in increasingly 'sophisticated' models such as the one hyped by the Business Insider article. We've all been reminded of 1984 and The Minority Report in recent weeks and months thanks largely to the no longer secret efforts of the NSA. Mainstream commentators are falling over themselves to express concern and outrage about the uses to which data could or, with greater alarm, 'is' being put to.  Data Professionals should start developing some answers to the more common concerns. 

Questions such as what individuals need to know about the data being collected about them, how it is impacting the choices available to them, who their data can be sold on to and whether attempts should be made to identify individuals from analysing a variety of aggregate data sources are all relevant and live today.  And there is surely a role for Data Governance to play here?  Data Governance should help organisations understand what they are connecting and why.  Data Governance should help organisations understand what they can do legitimately - ethically - do with their data. This will require particular attention as organisations look to exploit secondary usage of data.  If nothing else, establishing a Big Data Use Assessment every time you want to use your data sets for a purpose other than that they were originally intended to will help reduce the risk of costly law suits later down the line.
 

Later in The Hydrogen Sonata one of the characters warns that “One should never mistake pattern for meaning" which is text book Big Data best practice and another indication that data practitioners should take heed of this, the last, of the great Iain M Banks' Culture novels.  Recommended now and forever.  Whichever simulation you find yourself in.

Three Idiots Sat Babbling

At the start of the week there was an interesting piece in The Guardian about Big Data.  There’s been a lot of this sort of thing over the past few years of course but I’ve really been noticing a ramp up in press attention over the past couple of months.  Perhaps that has something to do with the recent release of Big Data: A Revolution that will transform how we Live, Work and Think by Kenneth Cukier and Viktor Mayer-Shonberger.  It's certainly a provocative read and one I’ll return to in a future post but for now I wanted to focus on another text mentioned in that Guardian article – The Minority Report by the great Philip K Dick.

This is a (very good) short story written in 1956 by Dick and undoubtedly gets referenced in quite so many Big Data articles because of the 2002 (not bad) film adaptation which definitely made analytics sexier than it probably deserves to be.  It was of relevance to the Guardian article primarily because the PreCrime unit it revolves around so closely resembles the “Crush” (Criminal Reduction Utilising Statistical History) policing approach being adopted in various parts of the planet. 

You can get a precis of the plot here (though I’d recommend reading it yourself because it is good) but I was interested in reading it to see if there was anything on top of the basic concept of using predictive analysis to reduce crime that can relate to the business of data management as I know it fifty eight years after the story was written.   

Given the era it was written in we can surely forgive the eccentric systems architecture it describes.  Chapter 1 gives a useful summary of what the PreCrime unit looks like...

"In the gloomy half-darkness the three idiots sat babbling. Every incoherent utterance, every random syllable, was analysed, compared, reassembled in the form of visual symbols, transcribed on conventional punchcards, and ejected into various coded slots. All day long the idiots babbled, imprisoned in their special high-backed chairs, held in one rigid position by metal bands, and bundles of wiring, clamps. Their physical needs were taken care of automatically. They had no spiritual needs. Vegetable-like, they muttered and dozed and existed. Their minds were dull, confused, lost in shadow.

But not the shadows of today. The three gibbering, fumbling creatures, with their enlarged heads and wasted bodies, were contemplating the future. The analytical machinery was recording prophecies, and as the three precog idiots talked, the machinery carefully listened."

No doubt a familiar experience to anyone that’s worked in Business Intelligence or Data Warehousing environments in the past decade but let’s look past the punchcard technology and call our “precog idiots” the equivalent of a Big Data statistical correlation engine.  As the story progresses, Dick offers an insight into how the three work together…

"...the system of the three precogs finds its genesis in the computers of the middle decades of this century. How are the results of an electronic computer checked? By feeding the data to a second computer of identical design. But two computers are not sufficient. If each computer arrived at a different answer it is impossible to tell a priori which is correct. The solution, based on a careful study of statistical method is to utilise a third computer to check the results of the first two. In this manner, a so-called majority report is obtained"

At a stretch, we could call this in memory, parallel processing? 

OK, I’m stretching the point here.  Perhaps Dick was not that much of a systems visionary?  Maybe his real strength was in predicting some of the data management issues that commonly arise today? 

1)      A Data Quality issue causes serious problems – Specifically it's the data quality dimension of timeliness that is revealed as having dropped the hero into difficulty.  It is the fact that each of the precog reports are run at different times – therefore using different ‘real time’ parameters – that leads to the different result of  the minority report.  And that’s the simple version of the plot. Nevertheless, the lesson is clear - don't compare apples with oranges.

2)      Engage current data providers in any delivery enhancement project - Driving the plot behind the pre-cog mistake is a power struggle between the Army who used to impose order and the PreCrime Unit that has usurped that role.  This seems to me to perfectly reflect the challenge any new Business Analytics solution faces when having to earn the credibility required to replace existing solutions.  Maybe even today there are people who claim to see little difference in Big Data solutions other than scale? Successful implementation projects will aim to bring in the existing providers of information into their stakeholder engagement.  These resource typically have much wisdom to impart and should be encouraged to find benefit from the new solutions,

3)      Knowing what data is held and how it is reported is key – It is only because of his position in the PreCrime Unit that the hero get placed in the tricky situation in the first place but it is also only by understanding the nature of the data held about him does he change his behaviour to escape the trap (sort of). Not only do I think it’s sensible data governance practice for organisations to know what data they actually have and how it is reported, I also think it is important for all of us as individuals to educate ourselves about what data is held about us, where, by whom and, critically, how we all contribute to it’s creation.

From an initial review of Cukier and Mayer-Shonberger I note that the authors are advising we all stop worrying about the causal why’s? and focus instead on the what’s? that the data shows us.  Clearly the lesson from Dick’s The Minority Report is to continue asking ‘why’ and ‘what’ but also start asking 'what if’. 

I’m aiming to post some thoughts on Big Data: A Revolution once I’ve finished it.  Hopefully it will be interesting to compare that vision of the future we’re living today with Dick’s vision of the future he envisaged back in 1956.