Responsibilities:

• Design, implement, and maintain large-scale distributed data processing systems using Hadoop, Spark, and related technologies
• Develop scalable and efficient data pipelines using Scala, Spark, Hive, and SQL
• Work with Google Cloud Platform (GCP) services for data ingestion, transformation, storage, and orchestration
• Migrate and process data using GCP tools such as BigQuery, Dataflow, Dataproc, Cloud Storage, Pub/Sub, and Composer (Airflow).
• Collaborate with architects and development teams to define technically sound and scalable solution designs.
• Debug and troubleshoot data processing and code-related issues; communicate findings with the development team
• Create automated data workflows using tools like Airflow and Jenkins as part of CI/CD pipelines.
• Ensure proper version control, testing (unit/integration), and deployment practices in accordance with DevOps methodologies.
• Communicate effectively with business stakeholders and contribute to a collaborative team environment
• Work in Agile methodologies (Scrum/Kanban) and participate in planning, stand-ups, and retrospectives

Requirements:

• Strong experience with the Hadoop ecosystem: Hadoop, HDFS, Hive
• Proficiency in Apache Spark and Scala
• Advanced SQL skills
• Experience with Google Cloud Platform (GCP), especially:
• BigQuery
• Cloud Dataflow
• Cloud Dataproc
• Cloud Storage
• Pub/Sub
• Cloud Composer & Airflow
• Familiarity with CI/CD tools such as Jenkins and Git/GitHub
• Experience with Airflow or similar orchestration tools
• Understanding of big data architecture and data modeling techniques (both relational and non-relational)
• Experience in designing and deploying scalable data solutions
• Strong debugging and code review skills
• Exposure to Enterprise Data Warehousing concepts
• Experience with DevOps practices and tools (Ansible, JIRA)
• Knowledge of Agile frameworks (Scrum, Kanban)