Requirements

Expected technologies

Python

PySpark

Databricks

Our requirements

• expert-level knowledge of Python, with 2+ years of production PySpark experience,
• 1+ years of hands-on architecture experience on Databricks, with deep understanding of Unity Catalog and Hive metastore,
• 3+ years of technical leadership experience, driving engineering strategy and team delivery,
• proven experience leading enterprise-scale data platform migrations,
• strong background in distributed system design handling petabyte-scale data,
• solid understanding of data governance, compliance, and audit requirements,
• experience designing robust APIs and integration patterns for enterprise environments,
• passion for defensive programming and building scalable, reliable software,
• excellent written and verbal communication skills, able to lead cross-functional collaboration,
• strong problem-solving skills, strategic thinking, and decision-making under ambiguity,
• good verbal and written communication skills in English (min. B2),
• availability to work with US East Coast time zone.

Optional

• experience as a Staff or Principal Engineer at a major technology company,
• contributions to open-source data platforms,
• experience working directly with large enterprise customers in a sales engineering context,
• degree in Computer Science, Engineering, or a related technical field (or equivalent practical experience).

Your responsibilities

• defining and evolving the technical roadmap for UCX migration workflows and platform architecture,
• driving key architectural decisions and ensuring consistency across storage, compute, and governance layers,
• leading the design and implementation of complex migration features and data workflows,
• optimizing Spark workloads and ensuring performance, reliability, and scalability of the platform,
• mentoring engineers, setting coding standards, and promoting best engineering practices,
• researching and prototyping new migration capabilities and Unity Catalog integrations.