Real-time analytics opens new use cases
Business problem
Increasingly, business intelligence systems that were once based on historic data, offline modeling and traditional reporting are being replaced by algorithms that operate on real-time data about customers and the world at large. These algorithms can power new applications that take advantage of real-time business opportunities and automated decision-making. Achieving this business transformation and new use-cases requires new technology that can process and analyze data in real time. If data is not processed in real time, businesses might be operating based on out-of-date data, which can lead to poor decisions.
Our In-stream processing experience
Over the years, we’ve helped many organizations jump-start their real-time projects. Quite a few of our solutions have grown into large-scale implementations, processing billions of events for applications ranging from fraud detection to real-time bidding marketplaces.
We’ve created a single reference architecture that details our complete end-to-end blueprint for an In-Stream Processing Service, based on lessons learned, best practices and proven configurations from our collective experience. It consists of 100% open source components, runs on any public cloud, and scales from developer sandboxes that can be spun-up at the click of a button, to always-on production configurations.
In-stream processing
In-stream processing blueprint
Our In Stream Processing blueprint is a preintegrated stack of these technologies and functions
- Persistent message queue system: Apache Kafka
- In-Stream Processing framework: Spark Streaming
- Lookup database: Redis
- Operational store: Cassandra
- Delivery of ingestions: HDFS
DevOps stack for in-stream processing
Deploying this platform on a dynamic cloud infrastructure so that it's available to its intended users is a nontrivial task. Our chosen technology stack contains these technologies:
- Cloud: AWS
- Deployment unit: Docker container
- Container management: Mesos + Marathon
- Bootstrapping Mesos + Marathon on bare cloud infrastructure: Ansible
- Application management and orchestration of Docker containers over Mesos + Marathon: Tonomi
In-stream processing high level architecture
In-stream processing DevOps stack architecture
Stream processing features
High throughput
Low latency
Fault tolerant
Supports several methods of insight delivery
Interoperable with any big data platform
Composed of 100% free, open source software
Complete In-Stream Processing blueprint
The technologies we use
How to build this
Grid Dynamics is here to help with architecture, design, implementation and operational support of In-Stream Processing platforms. Our services cover the full lifecycle for In-Stream Processing platforms, and include:
- Business needs, analysis and recommendation on the selection of a technology stack
- Recommendations on selection of a cloud provider, including cost estimation for the required infrastructure
- Design of a continuous integration and continuous deployment (CI/CD) pipeline
- Design for multi-datacenter deployment and disaster recovery
- Development of dashboards for visualization, monitoring and reporting of In-Stream Processing results
- Architectural supervision in instances of self-implementation on our blueprint
