Real-time Stream Processing

As more and more data is being streamed, there is a growing need for the organizations to process the data instantly or near real time; this course will cover stream processing concepts and technologies that will allow participants to query, analyze, and process data as it flows.

The course aims to equip participants with knowledge and hands-on skills needed for a streaming data architecture and will also cover popular use cases driven by streaming data with the aim of helping participants learn to identify opportunities at work that could leverage stream processing.

Level
Level : 
Intermediate/Advanced
Lessons
Date
Starting on Nov 16
Lessons
Format : 
4 live workshops (2 hours each) with recordings

Courage Noko

Courage Noko has over 12 years experience in Data engineering. He started at the New York Times where he was a Hadoop administrator/developer maintaining Hadoop clusters as well as writing MapReduce pipelines in Java and Pig. I also designed and implemented the first real time recommendation engine for nytimes.com.

He has led a team at Spotify to build a scalable real-time streaming platform that allows engineers to write real time recommendation engines, receive over 100s of millions of live events per seconds from streaming devices and run real time performance diagnostics. Over time, he has worked on several streaming frameworks: Apache Beam/Dataflow, Spark, Flink. And Storm.

Lately, he has taken interest in real-time analytics, leading teams to build and deploy platforms that support OLAP databases such as Druid, ClickHouse and Pinot.

What you'll learn

Make Education Accessible

Module 1: Introduction to basic streaming concepts

Batch vs Streaming data - comparing the main differences between bounded and unbounded data processing.

Real world examples - a look at some real world examples of real-time stream processing.

Overview of streaming architecture - detailed look at the various components that form the core parts of a streaming infrastructure

Examples of streaming engines.

Make Education Accessible

Module 2: Real time data processing

Event processing - a flow of events in the streaming pipeline.

Stream window operations -a detailed look at how different execution engines handle aggregations.

Make Education Accessible

Module 3: Advanced concepts

Stream Joins - joining multiple sources to streaming events.

Unit tests for streaming pipelines.

Real world streaming project covering concepts in modules 1-3.

Make Education Accessible

Module 4: Operations

Monitoring streaming jobs.

Best practices.

Review of Real world streaming project from module 3.


Prerequisites:

This course is for anyone with knowledge of basic data processing which includes the following data transformation concepts:

  • Filtering
  • Map
  • GroupBy
  • Sum
  • Count
Here is a full list of data transforms.

This course is for:

Data Scientists, Data Engineers and ML Engineers

What you will be able to do after this course:

  • Understand the differences between streaming and batch processing.
  • Learn about different streaming platforms and their features.
  • Learn different aggregation types and operations.
  • Learn how to join streams with several sources.
  • Write streaming pipelines for each concept covered to help participants understand the concepts through hands-on exercises.
  • Write tests for streaming pipelines.
  • Work on a real world streaming project.

Frequently Asked Questions

Are all sessions live?

Yes, all sessions during the cohort will be live with the instructor. However, we will record each session and make them available for everyone in the cohort.

What is the time commitment?

Our courses typically have 2-4 modules, with each module lasting for approximately 2h per week which you can block out on your calendar during your work day. You also get some take home projects that you can complete at your own pace.

Do I earn a certificate for this course?

Of course! Once you’ve completed the course modules, you will get a certificate of completion that you can showcase to the world.

Can I expense this course?

Yes. Most of our cohort participants have expensed this course through their Learning & Development budget, similar to how you expense conferences. You can use this email template to request expense approval from your manager.

Do you offer discounts for students?

Yes. Email us at hello@learncrunch.com and we'll be happy to give you a discount.

I have more questions. Get in touch with us!

If you have more questions, email us at hello@learncrunch.com.

Book a call with us