Data Quality Platform

Data Quality

The client’s offering, a data observability platform, allows teams to discover, prioritise, and collaboratively resolve data issues on both batch- and streaming platforms through a single pane of glass.

Challenge

The client is specialized in state-of-the-art data quality monitoring in batch processing environments. Since they identified a number of their clients transitioning to streaming architectures they needed a solution to provide a comprehensive solution capable of addressing all of their clients’ needs.

Solution

The number of use cases which use some form of real-time data processing has increased drastically over the last decades. Data is generated at an unprecedented volume, velocity and variety. This brings up the concern of guaranteeing data quality of these generated streams. To provide a solution for this,  an automated streaming data quality monitoring system was created. This system generates real-time insights on the quality of data and visualizes it in a single pane of glass.

The Technology behind

  • Flink
  • Kafka
  • Scala

“Things work out best for those who make the best of how things work out.”

— John Wooden

Results

Data Streaming | An initial Alpha version was created that allowed the client to offer an open source data quality analysis on Streaming data sources  

Dashboard | Single pane of glass for both batch and streaming use cases.

Ask for a demo!

We’re your one-stop cloud-native partner

Big Data engineering. Data science. Data Analytics. Site Reliability Engineering. Consulting. And customized Open Source projects for companies of all sizes. Learn more about what Klarrio can do for you today.

Contact us

Other Projects

Just a few projects examples.