Searching for Data: The Ignorance Explosion

Intkhab Ali, Senior Solution Architect

August 29, 2018

About 25 years ago (literally on the cusp of the popularity of the Web), my boss at the time ​did a symposium presentation on the “ignorance explosion”, which was a play on the phrase “information explosion”. He was not referring to people becoming stupider, but rather the increasing gap between information available and people’s capability to process it. That is, with all the web sites, data, social information, news, etc. that is provided to people, their ability to stay on top of it all declined – exponentially.

The graph below highlights this phenomenon with the red area clearly illustrating information overload relative to our ability to process information in blue.

The real issue it causes is not information obesity but rather “signal-to-noise” or the relevance issue. There’s a lot of information out there, but how much of it is relevant to you for a particular need at a particular point of time? Google has been one of the most successful companies to address this issue, but even they fail to resolve the problem of getting the right data at the right time for the right purpose.  Although Google can provide information relevant to you by searching the internet, the results may not always be complete, accurate, or trusted. Inquisitive people can get thousands of pages of results, but they rarely go past the first page because the relevance of what they’re looking for drops dramatically. They get overwhelmed by the amount of data they need to wade through to get the information they’re looking for. Google also misses a lot of sources as well. Furthermore, it can’t always prioritize what’s important relative to what’s happening on the world stage.

The talk a couple of decades ago would refer to digital agents, assistants or butlers that crawled through infinitive sources of information found in newspapers, our inbox, and other media sources providing a customized feed of what’s important to us – in a concise prioritized digestible chunk. The problem with these agents was that not all information could be connected. Moreover, even learning agents would only discover what’s important based on our behaviour. They’d filter out an article about the proverbial price of tea in China which, although not apparently relevant, could have some impact on our life. 

Today, the promise of AI has never been more compelling (see AI is the Future, Always has Been). With IBM’s Watson, Amazon Echo, and Microsoft’s Cortana, some of these technologies are now being given life and making big promises. However, many of us architects have seen some promise, but still mostly disappointment in the results. We know that there is the potential to do much better. With all the search engines out there – some now using improved AI techniques – Google still seems to provide the most relevant results but still only on the first or second page.

Organizations are taking some very big steps to reign in and make our structured and unstructured information more easily discoverable, useable, and trustable, but the battle is Herculean. We have information overload from various data sources including the following:

• 1. Content management systems
  2. Dynamic aggregation and search technologies such as Google appliances
  3. Mail folders
  4. File Shares
  5. External web sites
  6. Log files
  7. Audit/Event trails
  8. Master Data systems
  9. IoT streams
  10. Media catalogues and video streaming companies
  11. Online courseware
  12. etc…ad nauseam

This is just the tip of the iceberg. However, the problem is that the ignorance gap is real and growing not only in our organizations, but around the world.  I used to be on top of Internet memes and trends, but it seems that every day I find out new ones and have to dash to knowyourmemes.com to investigate. My contention is not only that there’s an ignorance explosion based on human ability to process information, but also on system limitations to deliver comprehensive and reliable data on demand. Our search systems only touch parts of the above which – in case anyone remembers the parable of the blind men and the elephant – could skew our knowledge and behaviours in the wrong direction. The reality is that we don’t and can’t get a complete picture of knowledge in most topic areas.

AI-powered Enterprise Search will probably advance us, at least a little bit, but it’s important for us to continue to recognize the ignorance gap that will continue to grow as we progresses through time. Let’s acknowledge that gap and continue to look for solutions that can better help us carve out our relevant knowledge – recognizing that there is no complete solution yet. As architects, our goal is to help our organizations and clients steadily, but incrementally, climb the knowledge ladder fully acknowledging that the growing ignorance gap is not something we can fix.  All we can do is help extract the best knowledge we can with constantly improving technology that will never really catch up to the growth of information.  In fact, the best we can do is recognize that the information explosion has the corollary effect of creating an ignorance explosion and that we carry this recognition to our organizations and clients.