Urgent position of Senior Azure Data Engineer (13 years must)

Urgent position of Senior Azure Data Engineer with Databricks Expertise

Location -NJ/NYC (Local Only)

Experience – 12-15 Years

Job Summary:

We are seeking highly skilled Azure Data Engineer with strong expertise in Databricks to join our data team. The ideal candidate will design, implement and optimize large-scale data pipeline, ensuring scalability, reliability and performance. This role involves working closely with multiple teams and business stakeholders to deliver cutting-edge data solutions.

Key Responsibilities:

1. Data Pipeline Development:

· Build and maintain scalable ETL/ELT pipelines using Databricks.

· Leverage PySpark/Spark and SQL to transform and process large datasets.

· Integrate data from multiple sources including Azure Blob Storage, ADLS and other relational/non-relational systems.

2. Collaboration & Analysis:

· Work Closely with multiple teams to prepare data for dashboard and BI Tools.

· Collaborate with cross-functional teams to understand business requirements and deliver tailored data solutions.

3. Performance & Optimization:

· Optimize Databricks workloads for cost efficiency and performance.

· Monitor and troubleshoot data pipelines to ensure reliability and accuracy.

4. Governance & Security:

· Implement and manage data security, access controls and governance standards using Unity Catalog.

· Ensure compliance with organizational and regulatory data policies.

5. Deployment:

· Leverage Databricks Asset Bundles for seamless deployment of Databricks jobs, notebooks and configurations across environments.

· Manage version control for Databricks artifacts and collaborate with team to maintain development best practices.

Technical Skills:

· Strong expertise in Databricks (Delta Lake, Unity Catalog, Lakehouse Architecture, Table Triggers, Delta Live Pipelines, Databricks Runtime etc.)

· Proficiency in Azure Cloud Services.

· Solid Understanding of Spark and PySpark for big data processing.

· Experience in relational databases.

· Knowledge on Databricks Asset Bundles and GitLab.

Preferred Experience:

· Familiarity with Databricks Runtimes and advanced configurations.

· Knowledge of streaming frameworks like Spark Streaming.

· Experience in developing real-time data solutions.

Certifications:

· Azure Data Engineer Associate or Databricks certified Data Engineer Associate certification. (Optional)

Please share to Clark@vkoresolutions.com