Requirements

Expected technologies

Delta Lake

Unity Catalog

Python

SQL

Medallion Architecture

Azure Data Lake Storage

Azure Event Hubs

Azure Data Factory

SQL Database

Synapse

Fabric

Optional technologies

Hadoop

Hive

Kafka

Flink

Our requirements

• Minimum 3 years' experience working with Databricks platform (Delta Lake, Workflows/Jobs, DLT, Unity Catalog)
• Strong experience with programming in Python
• Solid understanding of SQL and relational databases
• Knowledge of Data Warehouse, Business Intelligence and ETL/ELT data processing
• Familiarity with Medallion Architecture
• Very good knowledge of English (particular emphasis on written English)
• Proactive approach to tasks, problem-solving attitude and critical thinking skills
• Flexibility, independence and responsibility for assigned tasks
• A constant desire to improve your skills and learn new technologies
• Knowledge of Azure cloud components for data storage and processing: Azure Data Lake Storage, Azure Event Hubs, Azure Data Factory, SQL Database, Synapse, Fabric

Optional

• Experience with other big data technologies such as Hadoop, Hive, Kafka, and Flink would be an asset

Your responsibilities

• Design, build, and optimize scalable data pipelines in Databricks platform using SQL/Python/Spark
• Developing existing projects in the Microsoft Azure environment to gain valuable insights for the business
• Ensuring that data is modelled and processed according to the architecture and both functional and non-functional requirements
• Planning and implementing processing pipelines for structured and unstructured data (e.g. video and images)
• Working to automate and optimize internal processes in Azure
• Collaborating with cross-functional and international teams both internally and externally