Need Local or Nearby candidate and only H1B Candidate who is comfortable to share Passport No.
Role: Python Developer with Airflow
Location: Reston, VA (Day One Onsite)
Duration: Fulltime/Long Term
Experience Level:
- Overall Professional Experience: 15+ years
- Apache Airflow Specific Experience: 3+ years
- Autosys Specific Experience: 3+ years
- AWS MWAA Specific Experience: Proven track record of successful migrations
Job Summary:
- We are seeking a highly accomplished and strategically minded hands-on Lead Engineer to drive the modernization of our customer’s batch processing system. This pivotal role requires over 15 years of progressive experience in application development, integration of batch processes, with a specialized focus on Apache Airflow (and a deep understanding of legacy scheduling tools like Autosys). A critical component of this role will be leading the migration of existing Autosys workflows and jobs to AWS Managed Workflows for Apache Airflow (MWAA).
- Based in Virginia, this individual will play a key role in designing, implementing, and migrating batch processes from Autosys to Airflow.
Responsibilities:
- MWAA Migration & Strategy Leadership:
- Lead the end-to-end migration of existing Autosys workflows and jobs to AWS MWAA, including assessment, planning, re-platforming, testing, and validation support.
- Develop comprehensive migration strategies, roadmaps, and execution plans, minimizing disruption to ongoing operations.
- Design and implement robust, scalable, and secure batch pipelines within MWAA, translating Autosys concepts and logic into efficient Airflow DAGs.
- Serve as the primary technical expert for the Autosys to MWAA migration, providing guidance and troubleshooting support.
Airflow & MWAA Expertise:
- Architect, develop, migrate highly scalable, reliable, and efficient batch pipelines using Python and Airflow DAGs within MWAA.
- Manage and optimize MWAA environments, including infrastructure setup, configuration, monitoring, and scaling.
- Implement best practices for Airflow DAG development, testing, deployment, and version control (e.g., Git, CI/CD pipelines).
- Troubleshoot and resolve complex issues related to Airflow DAGs, infrastructure, and performance within MWAA.
Autosys Legacy System Understanding:
- Possess a strong understanding of Autosys concepts, job types (e.g., Command, File Watcher, Box jobs), dependencies, calendars, and monitoring.
- Perform in-depth analysis of existing Autosys workflows to identify migration complexities and opportunities for optimization.
- Collaborate with existing Autosys administrators and application teams to ensure accurate and complete migration of functionalities.
Team Leadership & Mentorship:
- Provide technical leadership, guidance, and mentorship to a team of migration engineers, fostering a collaborative and high-performing environment focused on migration and cloud adoption.
- Conduct code reviews, provide constructive feedback, and ensure adherence to coding standards and best practices.
- Contribute to the professional development of team members through training, knowledge sharing, and coaching on Airflow, MWAA, and migration best practices.
Architecture & Operational Excellence:
- Collaborate with SMEs, product owners, and other stakeholders to define data pipeline requirements and translate them into robust Airflow solutions.
- Contribute to the overall batch process architecture strategy, identifying opportunities for optimization, automation, and innovation post-migration.
- Implement robust monitoring, alerting, and logging solutions for Airflow DAGs and MWAA environments.
- Ensure compliance with security best practices for process handling and access within Airflow and MWAA.
Expertise in AWS:
- Hands on knowledge and expertise in S3, SQS, SNS, EKS, ECS Fargate, Step Functions
- Evaluate, and Implement architecture options for implementing end to end batch processes on AWS
Qualifications:
- Bachelor’s or master’s degree in computer science, or a related quantitative field.
- 10+ years of overall professional experience in Java/ JEE and RDBMS databases, and software development.
- 5+ years of hands-on experience in Springboot and Spring batch
- 5+ years of hands-on experience specifically with Apache Airflow, including complex DAG development, custom operators/hooks, and plugin creation.
- 3+ years of hands-on experience with Autosys, demonstrating a solid understanding of its features, job scheduling, and administration.
- Proven track record of successfully leading and executing migrations from legacy scheduling tools (specifically Autosys) to cloud-native orchestration platforms like AWS MWAA.
- Demonstrable expert-level experience with AWS Managed Workflows for Apache Airflow (MWAA), including environment setup, configuration, scaling, security, and troubleshooting.
- 5+ years of hands-on experience in Python programming for Airflow DAG development.
- Extensive experience with AWS cloud services, including but not limited to CloudWatch, and IAM.
- Proven experience designing and implementing highly scalable and fault-tolerant batch pipelines.
- Experience with relational and NoSQL databases (e.g., PostgreSQL, Oracle, DynamoDB, MongoDB).
- Familiarity with CI/CD practices and tools (e.g., Jenkins, GitLab CI, AWS CodePipeline).
- Strong problem-solving skills and the ability to diagnose and resolve complex technical issues, especially during migration.
- Excellent communication, interpersonal, and leadership skills, with the ability to effectively collaborate with cross-functional teams and mentor junior engineers.
Preferred Qualifications:
- AWS Certifications (e.g., AWS Certified Solutions Architect – Associate/Professional).
- Experience with containerization technologies (Docker, Kubernetes).
- Experience in the Financial services sector, particularly in environments with complex legacy systems.
Regards
Sagar Bhardwaj
Sr. Technical Recruiter
300 Alexander Park |Suite #200|Princeton , NJ 08540
Office: +1 7324521006 Ext238
Email: sagar.bhardwaj@diverselynx.com| URL: http://www.diverselynx.com