Last night I was attending an after work meet up where the topic was “Clash Of The Titans”. Microsoft, IBM, SAP and Oracle was presenting their BI and analytic solutions, both what they can offer today and how their future releases will be.
Remembering back to the year 2000 when I started my first job as a data warehouse developer, we programmed SAS code without any intellisense on a dark blue background with a white font.
After writing 500 lines of code we said a small prayer and then we pressed F8. Usually we got lucky, other times we didn’t, and had to use the rest of the day finding that small typo or looking for that breach in logic that didn’t give us the result we wanted.
Yesterday, on the other hand, Microsoft just put a webcam up and pointed it at the audience and got instantly a facial reading on how happy the audience was, Oracle talked about their new mobile BI solution that is going to give you the most relevant reports depending on where you are, and what kind of meeting you were attending. IBM showed us how Watson just based on your data source made a dashboard and made suggestions about what you should look into, and SAP talked about how you magically could get all your BI needs just by scanning through your environment.
So what was I doing back in 2000? We didn’t have the technology that we have today of course, don’t think I even owned a webcam at that time. What we did was to Extract data, we Transformed and cleaned it, and Loaded it into a data warehouse. What struck me last night, was that none of the four “Titans” mentioned the data warehouse with a word.
So given my background and my 17 years in the BI realm I’m starting to get afraid that I’m getting old and cranky and don’t understand the new things with Big Data, analytics and IOT, etc.
I seriously don’t think I’m neither old or cranky, my children would probably disagree, but I think we might be heading for chaos if we don’t structure our data before we report on them.
What the “Titans” were saying was that you can just use Power BI, Cognos or whatever reporting tool you have directly on your source and magically you’ll get wonderful dashboards and reports. What about the cleansing, the business rules and the mantra we have been talking about for decades “One single version of the truth”. Did we forget it? If we did, we seriously need to start remember it again.
My last blogpost was about much of the same things that I am writing about here, but it kind of worries me that we are bypassing the data warehouse. So the question is why aren’t we talking about it? Is it because it’s “old school” like me? Or is it because it is easier to sell a fancy reporting tool or the new exciting possibilities in the cloud?
The question should be, how can the old realm and the new realm co-exist?
My thoughts are that the Enterprise Data Warehouse still will exist and the Big Data initiatives will come as a supplement. I also believe that Microsoft, Oracle, SAP, IBM and the other big platform solution vendors know this – that for enterprise analytics and reporting supporting business decisions you need a data warehouse, dimensional modelling, one version of the truth etc., but they struggle to make “EDW” and “ETL” as sexy as Facial recognition and tweets.
You will still need to compare your revenue with comparable days, you still will need to see the development in product margin over time. It seems strange to put those data and implement those business rules in an unstructured environment.
Use your data lake for low level data so your analysts can use those data to analyze. And here are the analytic tools from the “Titans” excellent. Use them for data discovery and if you find some gold, implement that back to your data warehouse, implement the business rules and make reports. And, to make data warehousing sexy again – use automation tools to speed up the process.
In my next post I will try to dig deeper into how we can make our core business “sexy” again.