data-engineering-medallion-pipeline

Data Engineering Medallion Pipeline Skill

Skill by ara.so — Data Skills collection.

This skill enables AI agents to work with a complete data engineering pipeline implementing the Medallion Architecture (Bronze → Silver → Gold) using modern open-source tools: MinIO (S3-compatible storage), Airbyte (data ingestion), PostgreSQL (data warehouse), DBT (transformations), Apache Airflow (orchestration), and Grafana (monitoring).

What This Project Does

The data-engineering-medallion project provides a complete end-to-end data pipeline that:

1. Ingests raw data from MinIO object storage into PostgreSQL using Airbyte
2. Transforms data through three layers (Bronze/Silver/Gold) using DBT
3. Orchestrates the entire pipeline with Apache Airflow DAGs
4. Validates data quality with automated DBT tests
5. Monitors infrastructure health with Prometheus and Grafana
6. Visualizes business metrics in Power BI dashboards

Data Engineering Medallion Pipeline Skill

Skill by ara.so — Data Skills collection.

This skill enables AI agents to work with a complete data engineering pipeline implementing the Medallion Architecture (Bronze → Silver → Gold) using modern open-source tools: MinIO (S3-compatible storage), Airbyte (data ingestion), PostgreSQL (data warehouse), DBT (transformations), Apache Airflow (orchestration), and Grafana (monitoring).

What This Project Does

The data-engineering-medallion project provides a complete end-to-end data pipeline that:

1. Ingests raw data from MinIO object storage into PostgreSQL using Airbyte
2. Transforms data through three layers (Bronze/Silver/Gold) using DBT
3. Orchestrates the entire pipeline with Apache Airflow DAGs
4. Validates data quality with automated DBT tests
5. Monitors infrastructure health with Prometheus and Grafana
6. Visualizes business metrics in Power BI dashboards