Inadequate Data Systems Equal Opportunity

Albert Einstein once said “Not everything that can be counted counts, and not everything that counts can be counted”. I agree with the first claim. Not all data is relevant or necessary and it takes a certain skill to filter out the noise and be able to identify what is required to complete your analysis. The second claim I don’t entirely agree with. In today’s modern age, it is imperative for companies to measure everything. Allocating resources to lengthy observations, manual counting and recording of activities are important investments required to execute a pragmatic approach to problem solving in today’s competitive environment.

Once this valuable information has been obtained, the next step is ensuring adequate upkeep and perpetual maintenance. As companies grow, maintaining a cohesive database becomes increasingly difficult. Often, database systems become fragmented. This could be the result of organizational changes, personnel changes or iterative ‘Band-Aid’ solutions to name a few.

As a consultant, a fragmented and disorganized data system is music to my ears. Often, I will hear executives say: “That data file is a mess, you won’t find anything of value in there”. More times than not, this response means that I have stumbled upon gold. The reason is simple; if the data set is large, confusing, messy, labelled irrelevant or requires a complex analysis, it usually means that I am venturing into uncharted territory. Opportunity that is hidden deep within this type of data has not yet been discovered, evaluated or remedied. The ability to successfully execute this type of analysis, seek the gaps that others will not and formulate a solution is exactly what makes consultants so successful.

Typically, I follow five steps before conducting an analysis on this type of data set:

1. Adjust the data so that it is in a working format.
2. Spot check random data points to ensure that the data set is accurate.
3. Determine the exhaustive list of variables to observe and make a hypothesis for a potential finding or opportunity.
4. Re-organize the variables to best isolate the finding or opportunity.
5. List out the potential ways that the analysis could be disputed or misconstrued.

These steps help to lay the foundation for the remainder of the analysis and they can also help to identify any red flags in the data set before beginning.

All things considered, the next time you are faced with 500,000 data points in 5 different formats across 10 different systems, think of yourself as a pioneer in uncharted territory in search of the next great discovery that could be worth millions of dollars.