Data Engineer (Infra/DevOps Focus)

We are looking for a highly skilled Azure Data Engineer with expert knowledge in cloud infrastructure and DevOps automation. This critical hybrid role will be responsible for designing, building, optimizing, and automating our entire end-to-end data platform within the Microsoft Azure ecosystem. The ideal candidate will ensure our data solutions are scalable, reliable, and deployed using modern Infrastructure as Code (IaC) and CI/CD practices.

Key Responsibilities

Data Platform Development & Engineering

Design & Implement ETL/ELT: Develop, optimize, and maintain scalable data pipelines using Python, SQL, and core Azure data services.

Azure Data Services Management: Architect and manage key Azure data components, including:

Data Lakes: Provisioning and structuring data within Azure Data Lake Storage (ADLS Gen2).

Data Processing: Implementing data transformation and analysis logic using Azure Data Factory (ADF), Azure Synapse Pipelines, and Azure Databricks (using Spark/PySpark).

Data Warehousing: Designing and optimizing the enterprise Data Warehouse in Azure Synapse Analytics (SQL Pool).

Data Modeling and Quality: Define and enforce data modeling standards and implement data quality checks within the pipelines.

Cloud Infrastructure & DevOps Automation

Infrastructure as Code (IaC): Design, manage, and provision all Azure data resources (ADLS, Synapse, ADF, Databricks Clusters) using Terraform or Azure Resource Manager (ARM) Templates/Bicep.

CI/CD Implementation: Build and maintain automated Continuous Integration/Continuous Deployment (CI/CD) pipelines for all code (data, infrastructure, and application) using Azure DevOps or GitHub Actions.

Containerization & Compute: Utilize Docker and manage deployment environments using Azure Kubernetes Service (AKS) or Azure Container Instances (ACI) when required for data applications.

Monitoring, Logging, & Security: Configure comprehensive monitoring and alerting using Azure Monitor and Log Analytics. Implement network security and access controls (RBAC) across the data platform.

Required Skills & Qualifications

Azure Cloud: Strong hands-on experience designing and deploying end-to-end data solutions specifically within the Azure ecosystem.

Programming: High proficiency in Python (including PySpark) and expert knowledge of SQL.

DevOps & IaC: Proven, production-level experience with Terraform (preferred) or ARM/Bicep for automating Azure infrastructure deployment.

CI/CD: Experience setting up CI/CD workflows using Azure DevOps Pipelines or GitHub Actions.

Data Tools: Deep working knowledge of Azure Data Factory, Azure Databricks, and Azure Synapse Analytics.

Orchestration: Experience with workflow orchestration tools like Azure Data Factory or Apache Airflow.

Preferred Qualifications

Azure certifications such as Azure Data Engineer Associate (DP-203) or Azure DevOps Engineer Expert (AZ-400).

Familiarity with Data Governance tools such as Azure Purview.

Experience with real-time data ingestion using Azure Event Hubs or Azure Stream Analytics.