Select
Stream processing Data governance ML platform Analytical data platform Cloud-native data platform Big data DevOps

Increasingly, business intelligence systems that were once based on historic data, offline modeling and traditional reporting are being replaced by algorithms that operate on real-time data about customers and the world at large. These algorithms can power new applications that take advantage of real-time business opportunities and automated decision-making. Achieving this business transformation and new use-cases requires new technology that can process and analyze data in real time. If data is not processed in real time, businesses might be operating based on out-of-date data, which can lead to poor decisions.

Over the years, we’ve helped many organizations jump-start their real-time projects. Quite a few of our solutions have grown into large-scale implementations, processing billions of events for applications ranging from fraud detection to real-time bidding marketplaces.

We’ve created a single reference architecture that details our complete end-to-end blueprint for an In-Stream Processing Service, based on lessons learned, best practices and proven configurations from our collective experience. It consists of 100% open source components, runs on any public cloud, and scales from developer sandboxes that can be spun-up at the click of a button, to always-on production configurations.

undefined

In-Stream Processing Blueprint

Our In Stream Processing blueprint is a preintegrated stack of these technologies and functions 

DevOps stack for In-Stream Processing

Deploying this platform on a dynamic cloud infrastructure so that it's available to its intended users is a nontrivial task. Our chosen technology stack contains these technologies:

  • Cloud: AWS
  • Deployment unit: Docker container
  • Container management: Mesos + Marathon
  • Bootstrapping Mesos + Marathon on bare cloud infrastructure: Ansible
  • Application management and orchestration of Docker containers over Mesos + Marathon: Tonomi

In-Stream Processing high level architecture

undefined

In-Stream Processing DevOps stack architecture

undefined

High throughput

High throughput

Our blueprint can process up to 100,000 events per second.
Low latency

Low latency

It takes under 60 seconds to get from event to insight.
Fault tolerant

Fault tolerant

The computational platform is highly available and dynamically scalable.
Supports several methods of insight delivery

Supports several methods of insight delivery

If there is a downstream system designed to handle a stream of insights, the results will be delivered via a message queue.
Interoperable with any big data platform

Interoperable with any big data platform

In-Stream Processing can be deployed as a stand-alone cloud service and integrated with Big Data platforms via APIs.
Composed of 100% free, open source software

Composed of 100% free, open source software

All blueprint components are open source projects under active development by a large community of contributors.
Complete In-Stream Processing blueprint
Streaming platform
In-Stream processing framework
In-memory data structure store
Database management system
Scalable data storage
Cloud platform
Deployment unit via containerization
Container orchestration platform
Provisioning and configuration management tool
Automated application deployment tool
Streaming platform
In-Stream processing framework
In-memory data structure store
Database management system
Scalable data storage
Cloud platform
Deployment unit via containerization
Container orchestration platform
Provisioning and configuration management tool
Automated application deployment tool

Schedule a free workshop with one of our senior engineers to learn more

This field is requiredPlease enter your name
This field is requiredPlease enter your email
This field is requiredPlease enter company name