TwitchPipeline

Why this project

TwitchPipeline connects my interests in data engineering and streaming. It explores how stream metadata can be collected, transformed, scored, and analyzed in a pipeline with separate responsibilities.

The project looks at category usage and contextual signals in Twitch streams, with a particular interest in less popular categories.

Architecture

The repository documents the following flow:

1. EC2 and PySpark collect Twitch API data and transform it into Parquet.
2. S3 stores transformed data.
3. Lambda calculates a configurable score, with MongoDB holding streams selected for deeper analysis.
4. Step Functions coordinates the analysis workflow, including contextual analysis with Gemini.
5. PostgreSQL on RDS stores the results.
6. API Gateway, SQS, and EventBridge support integrations and decoupled consumers, with Looker for visualization.

What I wanted to explore

• Batch data processing and storage formats.
• Cloud orchestration and separating pipeline stages.
• Combining rule-based scoring with contextual analysis.
• Working within a small infrastructure budget.

The implementation and architecture notes are available in the GitHub repository.