I just got back from a long and exiting trip to London, where BI Builders has been sponsoring two big events. First Big Data World, where I was a speaker and then the Gartner Analytics Summit, where my colleague Anja was a speaker. And I would like to share my thoughts and key takeaways from this inspiring trip.
So firstly, Big Data World, a huge conference that was co-located with four other conferences; Smart IOT, Cloud Security Expo, Cloud Expo and Data Centre World.
As you probably can imagine the buzz words flew high here, Big Data vendors in all shapes and sizes had a booth and it was amazing to both hear about and get demonstrations of how cloud based, non-structured data could be utilized. There are a lot of upcoming smart software and technics that will help us utilise our data in new and exciting ways.
Our take on the Big Data World conference was, as I have written about earlier; the need for structure and control in this unstructured “big-data-world” that we are moving into, and even are in the middle of, is very important.
In my session, I talked about getting ready for the new era, and prepping for it before you run into it. Ready, Prep, Go! The analysis is only as good as the data quality and structure, and the need for integration with more structured master data entities is important when applying the analysis to business value.
The data warehouse isn’t sexy any more….
But what I have learned is that it’s the word itself and not the concept that is not sexy. But if you call it a data hub, discovery hub, central information factory or a data warehouse, it doesn’t really matter. The concept is still the same, with some variations of course. But if it’s modelled as a star schema or a snowflake, the end game is still to structure data for performance and flexibility and get the best possible fact-based decisions out of your historical data.
When we talk about big data and real time streaming data this type of data preparations seems old-fashioned and unnecessary. But after you have made the real-time reporting against your ‘IOT thingies’ the smart thing is to save some of that data into a prepared data hub so you can see development over time and connect it to your other dimensions. This was an area there were close to no information about.
I talked about how our tool could be used to bridge the gap between the two worlds. The data blending between your EDW and your data lake, and how we fit into Gartner’s concept of the Logical Data Warehouse architecture. If you can use a self-service data preparation tool like Xpert BI to help you move and structure the data from your data lake and into your structured environment, you will get more value out of both structured and unstructured data. We have some great ideas on automating/accelerating this process, and we are working hard on getting it from concept to real world implementation, these are exiting times!
At Gartner our audience was a bit more in our main target area. Anja did a great session talking about her experience from some of our customers and the challenges they have met building and maintaining a data warehouse, or data hub. And how now, when everyone is doing analytics, you have to do it smarter by using automation wherever possible to gain and maintain competitive advantage. Our booth was swamped after the session with companies that told us they had encountered one or more of the challenges Anja described in her session. It can seem that the challenges involved when both building and maintaining a well driven EDW are global and timeless.
And many of the challenges comes back to the ETL tools allowing for creativity in prepping your data, not governing the process enough. With a tool like Xpert BI you force the development into a less creative state with out of the box technical documentation. Then it is up to you to move the creativeness out to the end user layer, where it belongs.
I attended a session with Teradata and Tesco telling about their new architecture. Tesco is huge, the complexity and variety in their technical portfolio was more than impressing. In my previous job I was a BI manager at a Norwegian retailer, but this was science fiction to me. But as Stephen Brobst, CTO at Teradata said: with all these technologies you will have to ingest, or blend some of the data in to your traditional data warehouse to be able to do reporting over time. It’s always good to hear one of the rock stars in our area say the same thing as yourself 🙂
And one other thing he said which can’t be said enough times, Data Lake does NOT equal Hadoop!
To sum it up, it was an inspiring trip and an extremely educational trip, and I hope that BI Builders managed to inspire some of you as well.