Urgent Requirement || Data Engineer || PLEASANTON, CA

Note: Only Azure Data Engineer

Only H1B and Passport no

Must have Azure Data Lake Storage

Title: Azure Data Engineer

Location: PLEASANTON, CA

Hiring: Contract

Job Description

We are seeking a talented and motivated Data Engineer with expertise in Spark SQL, Databricks, Azure Data Factory (ADF), SQL, IICS, Unix, PySpark, Python, and Azure Data Lake Storage (ADLS) to join our dynamic team.

In this role, you will be responsible for designing, developing, and maintaining scalable data pipelines, optimizing data workflows, and ensuring efficient data processing.

You will collaborate closely with data scientists, analysts, and other stakeholders to ensure the timely delivery of data solutions.

Key Responsibilities: Design, develop, and maintain ETL/ELT pipelines using Azure Data Factory (ADF), Databricks, PySpark, and Python.

Build and optimize Spark SQL queries and workflows to process large volumes of data.

Implement and manage data pipelines and data integration solutions in cloud environments, specifically on Azure.

Leverage PySpark and Python to develop scalable, distributed data processing solutions and build custom data transformations.

Work with IICS (Informatica Intelligent Cloud Services) for data integration, transformation, and management.

Develop and optimize data workflows to support analytics and reporting needs.

Write complex SQL queries to extract, manipulate, and load data from various sources into data warehouses.

Utilize Unix/Linux shell scripting for automating tasks, managing file systems, and processing data.

Work with Azure Data Lake Storage (ADLS) to store, manage, and access large-scale data sets in a cloud environment.

Collaborate with data scientists, analysts, and other stakeholders to ensure the data infrastructure meets business requirements.

Ensure data quality, consistency, and security through best practices and governance frameworks.

Troubleshoot and resolve data processing issues, ensuring the scalability and reliability of data pipelines. Monitor and optimize the performance of data solutions for efficiency and cost-effectiveness.