Requirements

Expected technologies

SQL

Python

Java

AWS

Git

Kafka

Scala

Our requirements

• Experience with most of the following technologies (Apache Hadoop, Scala, Apache Spark, Spark streaming, YARN, Kafka, Hive, Python, ETL frameworks, Map Reduce, SQL, RESTful services).
• Sound knowledge on working Unix/Linux Platform.
• Hands-on experience building data pipelines using Hadoop components - Hive, Spark, Spark SQL.
• Experience with industry standard version control tools (Git, GitHub), automated deployment tools (Ansible & Jenkins) and requirement management in JIRA.
• Understanding of big data modelling techniques using relational and non-relational techniques.
• Experience on Debugging the Code issues and then publishing the highlighted differences to the development team/Architects.
• Experience with: time-series/analytics dB’s such as Elastic search, scheduling tools such as Airflow, Control-M, understanding of Cloud design patterns, developing Hive QL, UDF’s for analysing semi structured/structured datasets.
• Exposure to DevOps & Agile Project methodology such as Scrum and Kanban.

Your responsibilities

• Spark development and design using Scala 2.10+ or Java development or Python development and design using Java 1.8+, automated testing of new and existing components in an Agile, DevOps and dynamic environment.
• Promote development standards, code reviews, mentoring, knowledge sharing.
• Production support & troubleshooting.
• Implement the tools and processes, handling performance, scale, availability, accuracy and monitoring
• Liaison with BAs to ensure that requirements are correctly interpreted and implemented.
• Participate in regular planning and status meetings. Input to the development process – through the involvement in Sprint reviews and retrospectives. Input into system architecture and design.
• Peer code reviews.