How DataTalks.Club zoomcamps work - schedule, joining, modules, projects, peer review, certification.

Build data pipelines with Bruin and compete for prizes.

Build data pipelines with Bruin and compete for prizes.

Data Engineering Zoomcamp is a free 9-week course on building production-ready data pipelines. The next cohort starts in January 2026. Join the course here 👇🏼 - DataTalksClub/data-engineering-zoomcamp

Data Engineering Zoomcamp is a free 9-week course on building production-ready data pipelines. The next cohort starts in January 2026. Join the course here 👇🏼 - DataTalksClub/data-engineering-zoomcamp

I’m working on expanding the AI Shipping Labs website and wanted to migrate its current version from static GitHub Pages to AWS.

In this workshop, Alexey Grigorev breaks down the complexities of real-time data engineering, moving from basic Python-based Kafka consumers to enterprise-grade Apache Flink pipelines. This workshop, part of the Data Engineering Zoomcamp, provides a hands-on look at how to handle high-velocity data, out-of-order events, and stateful aggregations. You’ll learn about: - Kafka vs. Red Panda: Understand why Red Panda is a faster, simpler alternative to Kafka for developers. - Producer & Consumer: Learn to turn Python data into streamable bytes and read them back. - Database Integration: How to move data from a live stream into a permanent PostgreSQL database. - Flink Basics: Learn how Job Managers (the brain) and Task Managers (the muscle) run streaming jobs. - Handling Late Data: Use Watermarks to manage events that arrive late or out of order. - Time Windows: Group data into time blocks (like "every 5 minutes") to calculate real-time totals. Links: - Course: https://github.com/DataTalksClub/data-engineering-zoomcamp - Workshop: https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/07-streaming/workshop/README.md - DTC Courses: https://datatalks.club/blog/guide-to-free-online-courses-at-datatalks-club.html TIMECODES: 00:00 Workshop Introduction and Course Context 01:34 Finding the Workshop Materials and Course Links 03:01 Streaming Stack Overview: Kafka, Redpanda, and Flink 04:56 Credits, Workshop Changes, and Environment Setup Plan 06:32 Creating a Repository and Launching Codespaces 10:18 Python Project Setup with UV 13:09 Starting Redpanda and Defining Producer and Consumer Roles 15:37 Loading Taxi Data and Modeling Ride Events 23:48 Building a Kafka Producer and Sending JSON Events 27:16 Creating a Consumer and Cleaning Up Serialization Logic 40:02 Streaming Many Events and Writing Them to Postgres 48:41 Why Flink for Stream Processing 52:46 Flink Architecture and Docker Setup 01:03:24 First Flink Job: Kafka to Postgres Pass-Through 01:11:30 From Pass-Through to Real-Time Aggregations 01:14:50 Generating Real-Time and Late Events 01:18:58 Window Aggregations and Watermarks in Flink 01:28:57 Wrap-Up, Watermark Discussion, and Closing Connect with DataTalks.Club: - Join the community - https://datatalks.club/slack.html - Subscribe to our Google calendar to have all our events in your calendar - https://calendar.google.com/calendar/r?cid=ZjhxaWRqbnEwamhzY3A4ODA5azFlZ2hzNjBAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ - Check other upcoming events - https://lu.ma/dtc-events - GitHub: https://github.com/DataTalksClub - LinkedIn - https://www.linkedin.com/company/datatalks-club/ - Twitter - https://twitter.com/DataTalksClub - Website - https://datatalks.club/ Connect with Alexey - Twitter - https://twitter.com/Al_Grigor - Linkedin - https://www.linkedin.com/in/agrigorev/ Check our free online courses: - ML Engineering course - http://mlzoomcamp.com - Data Engineering course - https://github.com/DataTalksClub/data-engineering-zoomcamp - MLOps course - https://github.com/DataTalksClub/mlops-zoomcamp - LLM course - https://github.com/DataTalksClub/llm-zoomcamp - Open-source LLM course: https://github.com/DataTalksClub/open-source-llm-zoomcamp - AI Dev Tools course: https://github.com/DataTalksClub/ai-dev-tools-zoomcamp 👉🏼 Read about all our courses in one place - https://datatalks.club/blog/guide-to-free-online-courses-at-datatalks-club.html 👋🏼 Support/inquiries If you want to support our community, use this link - https://github.com/sponsors/alexeygrigorev If you’re a company, reach us at alexey@datatalks.club #apacheflink #kafka #redpanda #python #dataengineering #streaming #postgresql #docker #pyflink #realtimeanalytics #bigdata #softwareengineering #codespaces #datapipelines #streamprocessing #eventdriven #coding #techworkshop #learninginpublic #datatalksclub

In this video, Alexey Grigorev explains the critical final phase of the DataTalks.Club Zoomcamps: the Capstone Project. While homework builds specific skills, the project is the core requirement for graduation and certification across all courses, including MLOps, Machine Learning, and Data Engineering Zoomcamps. TIMECODES: 00:00 Introduction: Overview of the Zoomcamp Project Phase 03:12 Submission Requirements: GitHub Links and Commit IDs 06:18 Project Criteria: Applying Course Modules to Custom Datasets 09:20 Managing Your Identity on Certificates and Leaderboards 12:30 Peer Review Process: Evaluation and Reproduction 15:41 Academic Integrity: Policies on Plagiarism and Self-Plagiarism Connect with DataTalks.Club: - Join the community - https://datatalks.club/slack.html - Subscribe to our Google calendar to have all our events in your calendar - https://calendar.google.com/calendar/r?cid=ZjhxaWRqbnEwamhzY3A4ODA5azFlZ2hzNjBAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ - Check other upcoming events - https://lu.ma/dtc-events - GitHub: https://github.com/DataTalksClub - LinkedIn - https://www.linkedin.com/company/datatalks-club/ - Twitter - https://twitter.com/DataTalksClub - Website - https://datatalks.club/ Connect with Alexey - Twitter - https://twitter.com/Al_Grigor - Linkedin - https://www.linkedin.com/in/agrigorev/ Check our free online courses: - ML Engineering course - http://mlzoomcamp.com - Data Engineering course - https://github.com/DataTalksClub/data-engineering-zoomcamp - MLOps course - https://github.com/DataTalksClub/mlops-zoomcamp - LLM course - https://github.com/DataTalksClub/llm-zoomcamp - Open-source LLM course: https://github.com/DataTalksClub/open-source-llm-zoomcamp - AI Dev Tools course: https://github.com/DataTalksClub/ai-dev-tools-zoomcamp 👉🏼 Read about all our courses in one place - https://datatalks.club/blog/guide-to-free-online-courses-at-datatalks-club.html 👋🏼 Support/inquiries If you want to support our community, use this link - https://github.com/sponsors/alexeygrigorev If you’re a company, reach us at alexey@datatalks.club

Data Engineering Zoomcamp is a free 9-week course on building production-ready data pipelines. The next cohort starts in January 2026. Join the course here 👇🏼 - DataTalksClub/data-engineering-zoomcamp