Key Responsibilities

• Design, develop, and maintain robust data pipelines using PySpark in a scalable Big Data environment.
• Contribute to system architecture and design discussions.
• Participate in Agile ceremonies, including sprint planning, reviews, and retrospectives.
• Collaborate with Business Analysts to ensure accurate interpretation and implementation of requirements.
• Implement tools and processes that ensure performance, scalability, availability, and accuracy.
• Promote best practices through code reviews, mentoring, and knowledge sharing within the team.
• Provide production support, including troubleshooting and resolving data pipeline issues.
• Maintain clear documentation and participate in planning and status meetings.

Must-Have Qualifications and Skills

• 5+ years of experience in software design and PySpark/Scala development.
• Strong experience in developing and optimizing data pipelines using Hadoop components (Hive, Spark, Spark SQL).
• Hands-on experience with scheduling tools like Apache Airflow.
• Proficiency with Big Data technologies: Apache Spark, YARN, Hive, Python, SQL, RESTful APIs, MapReduce.
• Solid understanding of working in Unix/Linux environments.
• Experience with version control and CI/CD tools: Git/GitHub, Ansible, Jenkins.
• Familiarity with Agile project management tools such as JIRA.
• Strong understanding of both relational and non-relational data modeling techniques.
• Proven debugging and code analysis skills with the ability to clearly communicate findings to the development team.