In-Stream Processing 
Service Blueprint

What is In-Stream Processing?

In-Stream Processing is a powerful new technology that can scan mind-boggling volumes of data coming from sensors, credit card swipes, web clicks and find actionable insights nearly instanteneously. Companies across all industries are exploring new ways of processing information in real time, and in-stream processing is emerging as the leading framework to enable a wide range of real time applications.

What do organizations want from In-Stream Processing infrastructure?

We repeatedly get asked the same two questions by our customers;

1) What is the simplest, cheapest and quickest way to get our development team started with streaming algorithms?

2) How can they design and support a massively-scalable and highly-available production infrastructure?

Organizations need both a low barrier of entry to the project as well as the ability to rapidly scale seamlessly for massive production workloads as the system acquires more volume. In 2014 The future of open source report, the statistics also show that enterprise is trending towards open source and away from proprietary solutions it further goes to confirm that these same firms are looking to run solutions on the cloud of their choosing.

Over the years we've helped many organizations jump-start their real time projects. Many of our solutions have grown into large-scale implementations processing billions of events for applications ranging from fraud detection to real time bidding marketplaces.

What is in our In-Stream Processing Service Blueprint?

We've taken lessons learned, best practices and proven configurations from our experience and created a single reference architecture, and as is the spirit of open source, we have created a complete end-to-end Blueprint for In-Stream Processing Service. It's 100% open source, 100% cloud, 100% real time and scales from developer sandboxes that can be spun-up at a click of a button to always-on production configurations.

Customer Success Stories

Story 1

Customer: Leading VoIP services company

Application: Leading VoIP services company

Volumes: Leading VoIP services company

In production: Leading VoIP services company

What's cool:: Leading VoIP services company

Story 2

Customer: Leading VoIP services company

Application: Leading VoIP services company

Volumes: Leading VoIP services company

In production: Leading VoIP services company

What's cool:: Leading VoIP services company

Story 3

Customer: Leading VoIP services company

Application: Leading VoIP services company

Volumes: Leading VoIP services company

In production: Leading VoIP services company

What's cool:: Leading VoIP services company

Story 4

Customer: Leading VoIP services company

Application: Leading VoIP services company

Volumes: Leading VoIP services company

In production: Leading VoIP services company

What's cool:: Leading VoIP services company

Blueprint Goals

To provide engineering teams with pre-made, self-deployable cloud infrastructure in order to develop and test real in-stream processing applications while enabling operations teams to deploy, operate and grow enterprise-grade production infrastructure. Our design goals for the blueprint are as follows:

  • Pre-integrate event queueing, stream processing, data storage, insight delivery and result visualization into a single platform.
  • Support high throughput (up to 100,000 events/second), low-latency (under 60 seconds from event to insight) stream processing
  • Fault tolerant, highly available, dynamically scalable computational platform
  • Programmable in Spark Streaming API in Java or Scala
  • Support algorithms supported by Spark Streaming, including Spark SQL Streaming and machine learning
  • Store up to 30 days of raw data and isights
  • Support in-stream, batch and on-demand insight delivery
  • Composed of 100% free, open source software supported by an active company
  • Cloud-ready and portable across public and private clouds
  • Developer-friendly
  • Production-ready
  • Proven in mission-critical implementations
  • Interoperable with any big data platform
  • Extendable to support new use cases and unique requirements

Read the Blueprint

Post 4. In-Stream Processing Service Blueprint

Read more

Post 3. Overview of In-Stream Processing Solutions On the Market

Read more

Post 2. How In-Stream Processing Works

Read more

Post 1. What is In-Stream Processing?

Read more

Subscribe to Our Blog

Professional Services

Grid Dynamics is here to help with architecture, design, implementation or operational support of In-Stream Processing platforms.
Our services cover the full lifecycle of In-Stream Processing platforms, including:

  • Business needs analysis and recommendation on the technology stack selection,
  • Blueprint customization based on specific business needs,
  • Recommendations on selection of a cloud provider, including cost estimation for the required infrastructure,
  • Design of a continuous integration and continuous deployment (CICD) pipeline,
  • Design for multi-datacenter deployment and disaster recovery,
  • Development of dashboards for visualization, monitoring and reporting of In-Stream Processing results,
  • Recommendations for a testing strategy of In-Stream Processing applications (including test data management, workload modeling and test automation),
  • Complete implementation of the In-Stream Processing Service (including implementation of custom business logic, automated QA and deployement to the selected cloud),
  • Integration of the In-Stream Processing Service in customer infrastructure (including upstream, downstream systems as well as monitoring and alerting services); and
  • Architectural supervision in instances of self-implementation on our Blueprint.

Our professional services also extend to production support for the open source components used within the Blueprint, those being Apache Kafka, Apache Spark Streaming, Redis and Apache Cassandra.

Contact Us to Learn More
Contact Us To Learn More

Subscribe To Our Blog

Subscribe to Our Blog

Thank you! Your submission has been received!  

Oops! Something went wrong while submitting the form. Please try again.