Requirements

Expected technologies

Python

SQL

Terraform

PySpark

CI/CD

Azure Services

Our requirements

• Core Azure Expertise
• Advanced SQL, PySpark, and Python.
• Infrastructure-as-Code (Terraform, ARM/Bicep).
• Performance tuning (partitioning, indexing, query optimization).
• Azure Services:
• Data Factory (ADF), Synapse, Databricks, Cosmos DB.
• DevOps (Azure Pipelines, Repos), Monitor, and Security Center.
• GCP Proficiency (nice to have):
• BigQuery (partitioned tables, materialized views).
• Cloud Storage, Dataflow, Pub/Sub.
• IAM and VPC networking.
• Looker, Dataproc, Composer (Airflow).

Optional

• Certification:
• Azure: DP-203 (Data Engineer), AZ-400 (DevOps).
• GCP: Professional Data Engineer (preferred).
• Education:
• Bachelors/Masters in CS/IT or equivalent.

Your responsibilities

• Azure-Centric Data Engineering
• Design and deploy Azure data platforms:
• Data Lakes: Azure Data Lake Storage (ADLS Gen2) with Delta Lake optimizations.
• ETL/ELT: Azure Data Factory, Synapse Analytics, and Databricks workflows.
• Data Warehousing: Synapse Dedicated Pools, Azure SQL DB.
• Optimize Spark workloads (e.g. Azure Databricks) for performance tuning (partitioning, caching).
• Implement real-time pipelines: Azure Event Hubs, Stream Analytics, and IoT Hub integrations.
• Ensure data governance:
• Enforce data quality with Azure Purview (lineage tracking, sensitivity labeling).
• Implement RBAC and Azure Active Directory (AAD) integration for secure access.
• Build real-time analytics pipelines:
• Process streaming data via Azure Event Hubs/Stream Analytics with windowing and watermarking strategies.
• Operational Excellence
• Automate CI/CD pipelines.
• Lead disaster recovery (DR)
• Manage data lifecycle