Cloud-Native Crypto ETL
Serverless cryptocurrency data pipeline using Cloud Run Jobs, BigQuery, and Terraform with automated cost optimization.
Role: Data Engineer•2024
~$2.36
Monthly Cost
Total BigQuery query spend
Serverless
Architecture
ETL → Transform → Validate → Visualize
Daily/6-hourly
Automation
Scheduled data ingestion
100% Code
Infrastructure
Terraform-managed resources
Technology Stack
PythonDockerCloud Run JobsCloud SchedulerBigQueryGCSSecret ManagerdbtTerraformGreat Expectations
Contents
Problem
Cryptocurrency market analysis requires real-time price data, technical indicators, and trend analysis across multiple assets. Manual data collection is unreliable, and existing solutions are expensive or lack the flexibility needed for custom analytics.
Architecture
Designed a serverless, cost-optimized data pipeline using Google Cloud Platform:
CoinGecko API → Cloud Run Jobs → GCS → BigQuery → dbt → Looker StudioResults & Impact
Cost Optimization
- Monthly spend: ~$2.36 for BigQuery queries
- Partition pruning: Eliminates unnecessary data scanning
- Clustering: Optimizes queries by asset symbol
- Serverless architecture: Zero idle costs