Actionable insights require a powerful data platform
Companies that successfully compete, innovate and win in modern business environments are data-driven. While enterprises have access to huge amounts of internal and external data, converting these raw figures to actionable insights is a complex process. All the raw data must be gathered in one place, then processed, cleaned, connected and structured. Next, it has to be made accessible to business people via reports, dashboards and analytical tools. Finally, machine learning and artificial intelligence are applied to learn underlying patterns and identify the next best actions.
Helping customers turn data into insights since 2008
We started our work in this field by helping clients build scalable data platforms based on high-performance computing technology. When the first versions of MapReduce and Hadoop became available, we saw great value in merging the concept of high-performance computing with big data. Since then, we have helped a number of large and small clients in the technology, media, retail and financial services industries build analytical data platforms based on open source technology stacks, both on premise and in the cloud.
Building blocks
Foundational platform
A solid base platform is the key to implementing modern business intelligence successfully. Building a base platform requires careful design of storage and compute fabrics, as well as the implementation of security, data governance, CI/CD processes and quality engineering. Nowadays, enterprises have two major choices to make:
- Build the platform in the cloud. Google, Amazon and Microsoft provide strong cloud-native options for building data lakes, data warehouses, analytics and machine learning capabilities. Sticking with the cloud helps reduce initial implementation costs and cuts the time to market.
- Leave data in private data centers. Any modern Hadoop distribution can serve as the foundation for a data platform. Open source technologies, like Apache Spark, Beam, Hive, NiFi, Kubeflow, ElasticSearch and many others provide a good ecosystem to cover all enterprise needs with an open data stack.
Data processing pipeline
Getting the first influx of data into the platform breathes life into it. However, raw data is rarely useful. To increase its value, data should be cleaned, processed and structured, and often data from different sources need to be joined together.
There are two major techniques to process data: batch and in-stream. In the world of microservices, importing data from systems-of-record into an analytical data platform is a challenge for many companies. Constructing modern architectural patterns, such as event sourcing and CQRS, helps with this difficulty. When integrations with legacy systems are needed, a number of open source technologies assist with implementing traditional batch importing and processing.
Last but not least, building data lineage and data quality capabilities ensures that the data stays up-to-date.
Analytics and machine learning
Once the data is in the platform, it needs to be utilized. There is a number of ways to turn data into actionable insights:
- Provide data analysts with convenient access to the data in the platform for manual analysis.
- Generate reports and dashboards with various metrics and KPIs, which the business may use.
- Build a machine learning platform, so that data scientists can implement modern artificial intelligence and machine learning algorithms.
- Implement a decision portal, where business users can utilize AI and ML algorithms built by data scientists to get actionable insights automatically.
Key features
Scalable data lake
Data processing pipeline
Data warehouse
Reporting and dashboarding
Data access layer
Data science and machine learning platform
Data governance
Data quality
Business decisions portal
Cloud or on-premise deployment
Security
Continuous delivery
How it works
Technology stack
Our engagement model
When helping clients with building data platforms, we first want to understand what components of the platform the client needs help with. The client may want to migrate from a legacy data platform based on proprietary technologies to an open source platform, migrate from an on-premise platform to the cloud, or implement a specific capability, such as data quality or a machine learning platform.
An engagement typically starts with an architecture and design phase, during which we analyze the current state of the platform, create a target architecture, and detail an implementation plan with estimates. Alternatively, we may embed an engineer in the team to help fix a specific problem that the client has. After the initial phase is done, and the improvement plan is prepared, we proceed with the implementation phases, which we deliver in an Agile way.
