Data Recovery

Data Addiction and Data Recovery

The addiction to data is real and like many addictions it isn’t necessarily healthy. It is not that data is bad, but rather that data is infinite. Arbitrarily trying put constraints around data can lead to unexpected and suboptimal results.

Infinity Has Its Limits

Big data is one thing, but infinite data is quite another. In the effort to control and constrain data, we make a lot of assumptions and oversimplifications. Most of the time, this works well enough to programmatically make a model of the data, but this model is not necessarily reality.

Data starts with infinite possibility and expands in multiple dimensions from there. It gets infinitely bigger and infinitely smaller simultaneously. So, you have both the big picture and the minutia expanding simultaneously.

Conceptually this makes sense, but from a practical standpoint it is hard to constrain. Big data isn’t just the sum of all the little data. It is also made up of all the interactions between all of the data that feeds into the system. There are systems for small data, big data and meta data, but it is the unknown information in between that causes all of the mischief.

Creating Infinite Data

Data creation has its own flywheel. Data creates data, that creates more data which creates even more data. Just like shampoo. It is wash, rinse, repeat. But not all of this data creates value. Algorithms are creating more data, but not necessarily more knowledge.

Data can sometimes be a catalyst to create new knowledge and innovation, but new knowledge often needs some sort of external catalyst.

Functions Follow Forms

There isn’t a job function in the modern world that doesn’t require some type of information to be collected at some point. And the truth is data collection is time-consuming, tedious and error prone so the temptation of the data collectors to create a standardized form (yeah, I have definitely been on that side of the equation) to simplify and standardize the intake of data is great.

Initially, these forms seem like magic, but over time the form starts to break down. User fatigue sets in, new information is needed, something changes somewhere which makes this form and this data becomes obsolete, so a new form is born.

This cycle repeats and repeats. The data goes to new database and the old data is left for abandoned except that it is probably archived somewhere for future reference just in case.

The Expanding Universe

Like the universe, data is constantly expanding. Databases get bigger and bigger, storage capacity grows and grows, processing power gets faster and faster and bandwidth increases, and new data is perpetually generated.

Decision makers like to have data, so there is not a lot of incentive to curb this addiction, but in all likelihood the information that you need is not captured in your systems. It is the data that you don’t know about that contains the truly valuable information to guide decision making.

Get Over IT

Information technology will only take you so far. It is fine for processing what you already know and creating some efficiencies, but it is going to be an insight from outside your information systems that is going to generate the next big thing in your organization.