How to get the enterprise data strategy 'right' (Part I) : The roadmap to reliable data integration

-
There is immense untapped potential in the data universe 

There is data flowing relentlessly resulting from the ever expansive information exchange between its creators and consumers .Organizations are more or less baffled by the spooky fluctuations in businesses processes,workflow and enterprise routines that is created by this massive flux of data in the conduct of businesses.Indeed its not a thing of coincidence that the lifecycle of every business processes can be punctuated as a function of data.However, how to qualify, evaluate and improve the multifarious and heterogeneous data for a comprehensive data management and governance strategy is the essential question.

Organizations mostly take a very broad strategic approach to data management.To gain access to specific business information as opposed to unrefined bulk data streaming through API interfaces, there needs to be robust data formatting and integrity assessments.A more prudent way is to take a tactical approach to data integration, wherein the goal is to build a target data ecosystem and then establish cross-departmental links .This implies that each functional silo of business, focuses on its specific user-value to streamline.

A good starting point is asking how well conducted and managed is your ETL pipeline. Rather than over-focussing on best tools (choices like redshift vs kafka) to modify data based on advanced AI-driven cloud infrastructures, you should first get your basics dialled aka deeper understanding your business challenges and conforming to the best practices of ETL.This will only establish the right conducive environment for big data systems to take charge of producing quality analytics. While detecting anomalies its in best interest to devise more expressive integrity constraints and automate creation of dynamic master data catalogues.A few of the most critical criteria to consider are -

1.Data maintenance scoping from request for new features, broken data fixing, addition of new connections etc

2. Understanding transformation needs and keeping it user-oriented( in this case the various business functions)

3. Scale the ETL process for future anticipated changes and needs. The problem with combating these challenges is fundamentally rooted in highly volatile and changing nature of input data, changing requirements and business rules,.

Now having done the ground work, we need to transition towards the objective of setting up the control landscape for data management.Here in this series, I analyse the data pipeline through 3 data-qualifying concepts- Lineage and provenance , metadata consistency, integrity and interoperability (based on conceptualization of specific ontologies).

I shall discuss lineage and provenance and its paramount impact on data management and comprehensive data strategy in the next blog post in the same series.

Stay tuned!

Comments

Post a Comment

Popular posts from this blog

Becoming a Leader : A growth based maturity model

-
Image
The matter of leadership has enchanted possibly every individual, in his/her trail for excellence.Most of time, its seen as an over-night event of transformation that magically results from a spontaneous recipe of a few popular ingredients like courage, confidence and favourable externalities.People need to realize that sustainable and emulatory leadership goes much deeper than that level.I strongly feel there is a need for life-time value based maturity model of leadership, as a pathway which could be structured from very basic steps,so that it is accessible to a much wider audience.In the model that follows, I propose six phases of growth of individual leadership as a maturity mode l. These phases are as follows : Phase I :Instinctual bootstrapper Trust your gut-instinct and take the first step. The first steps land the first acquaintances Acquaintances turn into relationships through active community-level interactions Communication is key driver post-entry Phase II: Directed enthus
Read more

Value-added data-asset lifecycle management (Part II) : Enterprise capability development

-
                              Welcome back to the expository series on data assets and data governance! In my previous article, we learned about v alue-added data assets   and how and why you have to manage them strategically.In this episode, I discuss how these assets can be transformed for competitive enterprise capability development. Before delving into capability development from data assets   management, an important distinction claim attention here and needs to be delineated. That is between info quality and data quality , even though the two are used interchangeably , the former has user quality (known as information consumer) as its focus .The first major strategy shift required here is viewing information as a product to be managed as against limiting to data quality in databases and data-warehouses. Two models are considered industry standard for structuring the quality assurance which is the forerunner to information product management.They aid to assess info quality base
Read more

Blockchain, multi-agents and dynamic learning in supply chains

-
Image
Most of us in the supply chain professional community would not probably escape a day without noticing a story of how the blockchains are the talk of the 'supply chain town', with their ambitious claim to become the most promising step towards attaining the unvanquishable double T's of an agile supply chain network vis a vis traceability and transparency.There are two other wizkids of AI that promise to craft a more dynamic element to the plotline-:  intelligent multi-agents and dynamic learning . Here, I try to sketch a thread on possible interlude among these 3 game-changers in a relatively less explored context. Recently I was going through a research paper which was addressing the problem of supply chain disruption management with multi-agent based framework.Given that the treatment of problem was indeed , I found that the approach was silo-type with agents coordinating across enterprise legacy systems .The entire risk identification approach was centered on KPI benchma
Read more