geordee

A Case for Evolution of Data Warehousing

Data Warehousing has come a long way. The evolution of technology – especially in storage and processing – causes the technical solutions to evolve as well. It is time to revisit the traditional techniques of data warehousing, right from the data modeling aspects.

Most data warehouse implementations, from my experience, focused on quantitative aspects such as managing large volumes of data. There was an interviewer who almost mocked at me when I said I have not worked with terabytes of data. What we missed was the qualitative aspects of data, until it became an imperative. A large portion is then...

Trapped in Star Schema

Fact-less facts – an oxymoron. We had a requirement to track assignments of different individuals to different programs. At any point in time we should be able to retrieve the counts of assignments by various characteristics including the assignment status. Fact-less facts, obviously.

Here is a typical conversation.

“Let’s then design a fact-less fact table.”

“But wait, what’s the granularity for time dimension? The assignments can change any time, any number of times.”

“Probably we should use daily. That’s when the client’s reconcile the systems. Probably weekly would be fine, that’s when the reports are used. Or, monthly? That’s when...

Meta Data Warehousing

Imagine an ideal enterprise where the data is clean and consistent across. This means you can relate the data from one system to another without any transformations or lookups. This is probably the ultimate goal of master data management where the master data across the enterprise is managed for consistency, integrity and accuracy. Imagine ideal systems in that enterprise with data and functional services exposed so that those can be discovered, invoked and orchestrated. This is what service-oriented architecture is moving towards. Imagine these systems having unified authentication mechanisms and access controls. This is probably what enterprise directory services, single...

TIBCO, Integration, Visualization

TIBCO is an established Enterprise Application Integration product. Enterprise Application Integration came out from middleware technologies and right now about to get overridden by Service-oriented Architecture (SOA). In a Service-oriented Architecture communication between various services are fundamental and most of the Enterprise Application Integration patterns will be covered by various components of SOA. Will TIBCO then be an SOA platform? Yes. TIBCO has repositioned itself as an SOA platform already.

There is another paradigm shift that’s happening as service-oriented architectures get implemented in enterprises. Traditionally applications held the data and enterprise application integration tools helped these applications to communicate with...

Information Supply Chain

This is not a new thought. Over the years many people have thought about the information moving within an enterprise and created analogies to supply chains. This is an attempt to think independently on information supply chain.

Information Supply Chain, in my opinion, is to look at enterprise information from the perspective of a supply chain professional, with data replacing materials and goods. Once the supply chain is visualized and defined, it is possible to analyze value additions, optimize information flow and implement better governance.

From this perspective, applications in an enterprise IT landscape performs three functions. They produce data,...

Conductor, Coach, Commander

There is a difference between Information Technology (IT) and established industries in terms of maturity in operations. Producing IT outputs is still treated as a knowledge-intensive activity, while in other industries the production is labor-intensive. IT is slowly moving from knowledge-intensive to labor-intensive, and innovations such as software factory model are trying to replicate the best practices of labor-intensive operations in IT projects and operations. However, not all areas in IT are ready to adapt the best practices of labor-intensive industries. One such area is project management.

There is a general perception among software engineers, at least among the people...

Data and Computation

Business Intelligence (BI) has matured in the past 20 years or so of it’s existence. There has been a lot of developments in software and information technology during this period. One of the significant change is that computation has become cheaper than storage.

When BI was in the defining stages, querying the OLTP databases was not a viable option. This was mainly due to the fact that processing (computing) power was limited in OLTP systems and databases, and adding it would be costlier than moving the data to another place for OLAP analysis. If you tried to execute an analytical...

Architecture

Years ago, I architected a house. Being a Civil Engineer by qualification architecture is something that I was always fascinated with. Today I architect software, at least that’s what they say I do.

Architecture is one of the most misused terms in today’s software industry. There is an architect for everything – one for user interface, one for data, one for solution and one for the entire system. Then there are numerous technical architects who do all kinds of job – from design to coding. Then there are those obscure ones – lead architect, consultant architect, principal architect and so...

Sweet Dreams, Little One

It’s hard for us to conceive, Why on earth you had to leave. But if we listen, we’ll hear God say, “Here is where he at peace shall lay”. You’re already in beautiful heaven above, And can see nothing but God’s love. Even though we may not understand, We can be sure Jesus is holding your hand. You were the most beautiful baby, and though we despair We know God will look after you with tender loving care. You are safe in His hands now. Sweet dreams, Little One.

Beta

Recently I built a website that used beta version of an open source software. Obviously, I ran into problems and I had to contact the community to get help. One response I got was that the software is in beta and should not be used in production. I replied back, “my site is in beta version too!”

A lot of changes have taken place in the past few years. Traditionally software solutions were born in academic and research institutions and then brought to commercial and enterprise space. Once it is established it was adopted or made accessible to individual users...