Senior Data Engineer

📋 Description

• Developing a unified data platform: Develop our new unified data platform on Databricks. You will be instrumental in establishing the Medallion Architecture (Bronze, Silver, Gold layers) using dlt for data modeling and transformations.
• Develop Data Pipelines: Create and manage resilient data pipelines for both batch and real-time processing from various sources in our Azure data ecosystem. This includes building a "hot path" for streaming data and orchestrating complex dependencies using Databricks Workflows.
• Enable Data Integration and Access: Implement and manage data replication processes from Databricks to Snowflake. You will also be responsible for developing a low-latency query endpoint to serve our production Karbon application.
• Champion Data Quality and Governance: Establish best practices for data quality, integrity, and observability. You will build automated quality checks, tests, and monitoring for all data assets and pipelines to ensure trust in our data.
• Implement Robust Security and Governance Practices: Design and enforce a comprehensive security model for the data platform. This includes management of PII and implementing a fine-grained Role-Based Access Control (RBAC) model through IaC
• Cross functional collaboration: Work within a cross-functional team of AI engineers, analysts, and developers to deliver impactful data products.

🎯 Requirements

• 5+ years of relevant work experience as a data engineer, with a proven track record of building and scaling data platforms
• Previous experience with Databricks
• Previous experience architecting ETL & ELT data migration patterns with strong proficiency in DLT
• Experience scaling data pipelines in a multi-cloud environment
• Strong proficiency in Python
• Strong proficiency in SQL and a deep understanding of relational DBMS
• DevOps experience, including CI/CD, and infrastructure-as-code (e.g., Terraform)

🎁 Benefits

• Gain global experience across the USA, Australia, New Zealand, UK, Canada and the Philippines
• 4 weeks annual leave plus 5 extra "Karbon Days" off a year
• Flexible working environment
• Be part of a fast-growing company with opportunities for internal promotion
• A collaborative, team-oriented culture that embraces diversity, invests in development, and provides consistent feedback
• Generous parental leave