9 days and 8 nights in London

9 days and 8 nights in London

I just got back from a long and exiting trip to London, where BI Builders has been sponsoring two big events. First Big Data World, where I was a speaker and then the Gartner Analytics Summit, where my colleague Anja was a speaker. And I would like to share my thoughts and key takeaways from this inspiring trip.

So firstly, Big Data World, a huge conference that was co-located with four other conferences; Smart IOT, Cloud Security Expo, Cloud Expo and Data Centre World.

The data warehouse is still sexy

As you probably can imagine the buzz words flew high here, Big Data vendors in all shapes and sizes had a booth and it was amazing to both hear about and get demonstrations of how cloud based, non-structured data could be utilized. There are a lot of upcoming smart software and technics that will help us utilise our data in new and exciting ways.

Our take on the Big Data World conference was, as I have written about earlier; the need for structure and control in this unstructured “big-data-world” that we are moving into, and even are in the middle of, is very important.

In my session, I talked about getting ready for the new era, and prepping for it before you run into it. Ready, Prep, Go! The analysis is only as good as the data quality and structure, and the need for integration with more structured master data entities is important when applying the analysis to business value.

The data warehouse isn’t sexy any more….

But what I have learned is that it’s the word itself and not the concept that is not sexy. But if you call it a data hub, discovery hub, central information factory or a data warehouse, it doesn’t really matter. The concept is still the same, with some variations of course. But if it’s modelled as a star schema or a snowflake, the end game is still to structure data for performance and flexibility and get the best possible fact-based decisions out of your historical data. Espen giving presentation in London

When we talk about big data and real time streaming data this type of data preparations seems old-fashioned and unnecessary. But after you have made the real-time reporting against your ‘IOT thingies’ the smart thing is to save some of that data into a prepared data hub so you can see development over time and connect it to your other dimensions. This was an area there were close to no information about.

 

Gartner 2016 logical data warehouseI talked about how our tool could be used to bridge the gap between the two worlds. The data blending between your EDW and your data lake, and how we fit into Gartner’s concept of the Logical Data Warehouse architecture. If you can use a self-service data preparation tool like Xpert BI to help you move and structure the data from your data lake and into your structured environment, you will get more value out of both structured and unstructured data. We have some great ideas on automating/accelerating this process, and we are working hard on getting it from concept to real world implementation, these are exiting times!

At Gartner our audience was a bit more in our main target area. Anja did a great session talking about her experience from some of our customers and the challenges they have met building and maintaining a data warehouse, or data hub. And how now, when everyone is doing analytics, you have to do it smarter by using automation wherever possible to gain and maintain competitive advantage. Our booth was swamped after the session with companies that told us they had encountered one or more of the challenges Anja described in her session. It can seem that the challenges involved when both building and maintaining a well driven EDW are global and timeless.

And many of the challenges comes back to the ETL tools allowing for creativity in prepping your data, not governing the process enough. With a tool like Xpert BI you force the development into a less creative state with out of the box technical documentation. Then it is up to you to move the creativeness out to the end user layer, where it belongs.

Anja giving presentation in LondonI attended a session with Teradata and Tesco telling about their new architecture. Tesco is huge, the complexity and variety in their technical portfolio was more than impressing. In my previous job I was a BI manager at a Norwegian retailer, but this was science fiction to me. But as Stephen Brobst, CTO at Teradata said: with all these technologies you will have to ingest, or blend some of the data in to your traditional data warehouse to be able to do reporting over time. It’s always good to hear one of the rock stars in our area say the same thing as yourself 🙂

And one other thing he said which can’t be said enough times, Data Lake does NOT equal Hadoop!

To sum it up, it was an inspiring trip and an extremely educational trip, and I hope that BI Builders managed to inspire some of you as well.

Are we heading towards chaos? Or am I just getting old?

Are we heading towards chaos? Or am I just getting old?

Last night I was attending an after work meet up where the topic was “Clash Of The Titans”. Microsoft, IBM, SAP and Oracle was presenting their BI and analytic solutions, both what they can offer today and how their future releases will be.

Write 500 lines of 500, pray and press F8Remembering back to the year 2000 when I started my first job as a data warehouse developer, we programmed SAS code without any intellisense on a dark blue background with a white font.

After writing 500 lines of code we said a small prayer and then we pressed F8. Usually we got lucky, other times we didn’t, and had to use the rest of the day finding that small typo or looking for that breach in logic that didn’t give us the result we wanted.

Yesterday, on the other hand, Microsoft just put a webcam up and pointed it at the audience and got instantly a facial reading on how happy the audience was, Oracle talked about their new mobile BI solution that is going to give you the most relevant reports depending on where you are, and what kind of meeting you were attending. IBM showed us how Watson just based on your data source made a dashboard and made suggestions about what you should look into, and SAP talked about how you magically could get all your BI needs just by scanning through your environment.

So what was I doing back in 2000? We didn’t have the technology that we have today of course, don’t think I even owned a webcam at that time. What we did was to Extract data, we Transformed and cleaned it, and Loaded it into a data warehouse. What struck me last night, was that none of the four “Titans” mentioned the data warehouse with a word.
Old and CrankySo given my background and my 17 years in the BI realm I’m starting to get afraid that I’m getting old and cranky and don’t understand the new things with Big Data, analytics and IOT, etc.
I seriously don’t think I’m neither old or cranky, my children would probably disagree, but I think we might be heading for chaos if we don’t structure our data before we report on them.

What the “Titans” were saying was that you can just use Power BI, Cognos or whatever reporting tool you have directly on your source and magically you’ll get wonderful dashboards and reports. What about the cleansing, the business rules and the mantra we have been talking about for decades “One single version of the truth”. Did we forget it? If we did, we seriously need to start remember it again.

My last blogpost was about much of the same things that I am writing about here, but it kind of worries me that we are bypassing the data warehouse. So the question is why aren’t we talking about it? Is it because it’s “old school” like me? Or is it because it is easier to sell a fancy reporting tool or the new exciting possibilities in the cloud?

The question should be, how can the old realm and the new realm co-exist?

My thoughts are that the Enterprise Data Warehouse still will exist and the Big Data initiatives will come as a supplement. I also believe that Microsoft, Oracle, SAP, IBM and the other big platform solution vendors know this – that for enterprise analytics and reporting supporting business decisions you need a data warehouse, dimensional modelling, one version of the truth etc., but they struggle to make “EDW” and “ETL” as sexy as Facial recognition and tweets.

You will still need to compare your revenue with comparable days, you still will need to see the development in product margin over time. It seems strange to put those data and implement those business rules in an unstructured environment.

Use your data lake for low level data so your analysts can use those data to analyze. And here are the analytic tools from the “Titans” excellent. Use them for data discovery and if you find some gold, implement that back to your data warehouse, implement the business rules and make reports. And, to make data warehousing sexy again – use automation tools to speed up the process.

In my next post I will try to dig deeper into how we can make our core business “sexy” again.