Lead PySpark Developer c2c hiring Owings Mills, MD

Lead PySpark Developer

Location: Owings Mills, MD

Duration: 6 months

Job description:

• 10+ years of experience in big data and distributed computing.

• Very Strong hands-on experience with PySpark, Apache Spark, and Python.

• Strong Hands-on experience with SQL and NoSQL databases (DB2, PostgreSQL, Snowflake, etc.).

• Proficiency in data modeling and ETL workflows.

• Proficiency with workflow schedulers like Airflow.

• Hands on experience with AWS cloud-based data platforms.

• Experience in DevOps, CI/CD pipelines, and containerization (Docker, Kubernetes) is a plus.

• Strong problem-solving skills and ability to lead a team • DBT, AWS Astronomer

• Lead the design, development, and deployment of PySpark-based big data solutions.

• Architect and optimize ETL pipelines for structured and unstructured data.

• Collaborate with Client, data engineers, data scientists, and business teams to understand requirements and provide scalable solutions.

• Optimize Spark performance through partitioning, caching, and tuning.

• Implement best practices in data engineering (CI/CD, version control, unit testing).

•Work with cloud platforms like AWS.

• Ensure data security, governance, and compliance.

• Mentor junior developers and review code for best practices and efficiency.

“Tekgence is an equal opportunity employer. Applicants must be authorized to work in the U.S. U.S. citizens and Green Card holders are strongly encouraged to apply.”