Data Engineer - Azure

Role Overview

We are hiring a Data Engineer with strong hands-on Azure experience to build and run the pipelines behind our analytics products. You will ingest large public and partner datasets, land them in Azure Blob Storage, transform them in ClickHouse and PostgreSQL, and orchestrate it all with Azure Data Factory and Azure Batch. You will own pipelines end to end: ingestion, modelling, data quality, monitoring, and cost.

We build with AI coding tools every day, and we expect you to as well.

Key Responsibilities

• Build and maintain Python ETL pipelines (pandas, PyArrow, DuckDB) that ingest bulk files, APIs, and scraped web sources, including legacy and fixed-width formats.
• Design and operate Azure Data Factory pipelines: scheduling, dependencies between monthly and daily loads, parameterisation, retries, and failure alerts.
• Run containerised Python workloads on Azure Batch and Azure Container Registry, and manage pool sizing, quotas, and node cost.
• Organise raw and curated data in Azure Blob Storage / ADLS Gen2 with clear partitioning, versioned drops, and retention rules.
• Model data in ClickHouse (MergeTree engines, partitioning, ordering keys, materialised views) and PostgreSQL for application-facing data.
• Build staging, intermediate, and mart layers with dbt, including incremental and full-refresh strategies and audit tables that track rows added, updated, and deleted per run.
• Write data quality checks and reconciliations against source systems, and investigate when numbers drift.
• Secure pipelines with Managed Identity, Key Vault, and least-privilege RBAC; deploy through Azure DevOps CI/CD and Terraform.
• Monitor Azure spend and pipeline run times, and tune queries, storage, and compute to keep both down.
• Work with analysts, product, and app teams to turn business questions into reliable tables and views.

Required Skills & Qualifications

• 3–5 years building production data pipelines.
• Solid hands-on Azure experience (2+ years) with Data Factory, Blob Storage / ADLS Gen2, Azure Batch or Functions, Key Vault, Managed Identity, RBAC, networking basics, and cost management.
• Strong Python for data work: pandas or PyArrow, packaging, logging, error handling, and tests.
• Advanced SQL: window functions, CTEs, deduplication, slowly changing data, and query tuning.
• Hands-on PostgreSQL: schema design, indexing, and bulk loading.
• Data modelling for analytics: star schemas, facts and dimensions, grain, and keys.
• Git and CI/CD (Azure DevOps or GitHub Actions); Docker.
• Daily use of AI coding assistants (for example Claude Code, GitHub Copilot, or Cursor) to write, test, and refactor pipelines, with the judgement to review and verify what they produce.
• Clear written communication: you document pipelines and explain data issues to non-engineers.
• ClickHouse in production: table engines, partitions, mutations, and loading with clickhouse-connect.
• Snowflake, Databricks, or another cloud warehouse.
• dbt (Core or Cloud), including macros, tests, and incremental models.

Nice to Have

• Terraform or Bicep for Azure infrastructure as code.
• Web scraping and automation with Playwright for sources that have no API.
• SQL Server / Azure SQL and ODBC integrations.
• DuckDB for local and file-based processing; Parquet and columnar formats.
• Power BI or another BI tool on top of your models.
• Experience with oil and gas, energy, or government regulatory data (for example Texas RRC).
• Azure certifications such as DP-203 / DP-700 or AZ-104.