Why do managers and workers still struggle to find the information that they need to make decisions or take action despite advances in digital technology? That is, what causes data deficiencies?

Corporate infrastructure and decision support systems have evolved over decades. Over this same period, organizations have endured management changes, shifting priorities and differing perspectives on the role of IT. Data silos, lost or bypassed data, poorly designed interfaces, nonstandardized data formats and chronically in flux requirements further compound the natural system and organizational challenges brought on by progress,

In my opinion, organizations can begin to combat some of the corporate infrastructure and organizational behavior issues by having a clear vision and mission when it comes to information systems. Management changes will happen, but an organization that has a clear vision and mission regarding the value of data and information will stay focused on strategic objectives amidst regime change. Everyone within the organization should be viewed as a stakeholder and a benefactor. There is an education process that needs to take place; all the parties concerned need to have the right reaction to the blue button moment. Data silos, poor user interface design, etc… persist because a wrong choice is made when a blue button moment occurs. The ability to changes the future depends on the decisions we make now.

The wrong blue button moment:
System doesn’t love embedded images so here is a link: https://goo.gl/L1XDLk

Source: Alex Cowan – Getting Started: Agile Meets Design Thinking, University of Virginia

The right blue button moment:
System doesn’t love embedded images so here is a link: https://goo.gl/fFdjnh

Source:  Alex Cowan – Getting Started: Agile Meets Design Thinking, University of Virginia

I believe we are lowering the barrier to entry when it comes to how we transform data into information. For years the industry spent time trying to force data into a common data model for business intelligence (BI), this normalization process usually consisted of one or more ETL (extract, transform, load) jobs. These jobs were typically batched, and the end state was a normalized data set pushed into a relational database management system (RDBMS), the relational SQL database schema was rigid and comprised of tables consisting of columns, rows, and fields. We called these DSS (Decision Support Systems) data warehouses and data marts. Fast forward a few years and many of these information systems which leveraged historical data as the primary predictor of the future are pivoting towards NoSQL databases where key-value pairs have replaced SQL relationships. NoSQL information systems are meant for massive real-time ingest; these systems are being used to build data lakes. The ability to use key-value pairs removed the need for a rigid schema often removing the need for an ETL process. The field of Data Science, NoSQL platforms like Hadoop, applications like Elastic Search for indexing, Kibana for visualization and programming languages like RPythonOctave, etc… make capturing data and performing analytics easier than ever before. With the advent of public cloud and platforms AWS EMRAWS Data PipelineGoogle Cloud DataflowGoogle Cloud Dataproc and many of the issues like data silos, lost or bypassed data, poorly designed interfaces, and nonstandardized data formats are being addressed.  The technology is being adapted to address matters that have persisted for a very long time; these new technologies are streamlining processes, increasing the time to value and solving some of the issues mentioned above.

References

Data Lake vs. Data Warehouse: Is the warehouse going under the lake? (2016, July 22). Retrieved September 06, 2017, from https://www.dezyre.com/article/data-lake-vs-data-warehouse-is-the-warehouse-going-under-the-lake/283

NoSQL vs SQL- 4 Reasons Why NoSQL is better for Big Data applications. (2015, March 19). Retrieved September 06, 2017, from https://www.dezyre.com/article/nosql-vs-sql-4-reasons-why-nosql-is-better-for-big-data-applications/86

Turban, E., Volonino, L., & Wood, G. R. (2015). Information technology for management digital strategies for insight, action, and sustainable performance. New Jersey (Estados Unidos): Wiley.