Senior Data Lead - Azure Data Factory (ADF), Databricks

Job Summary

JD:
Need experienced Senior Data Engineer (6+ years) to design and optimize scalable data solutions on Microsoft Azure. You will work with Snowflake for data warehousing and Databricks for big data processing within a Lakehouse architecture using medallion principles (Bronze, Silver, Gold).


Key Responsibilities:
JD:
Need experienced Senior Data Engineer (6+ years) to design and optimize scalable data solutions on Microsoft Azure. You will work with Snowflake for data warehousing and Databricks for big data processing within a Lakehouse architecture using medallion principles (Bronze, Silver, Gold).


Key Responsibilities:
Data Pipelines: Build and optimize ETL/ELT pipelines using Azure Data Factory, Databricks (PySpark, SQL), and ingest data from diverse sources into ADLS and Snowflake.
Databricks Development: Create and tune Spark jobs, leverage Delta Lake, Autoloader, and advanced optimization techniques.
Data Warehousing: Design and manage Snowflake schemas, queries, and performance tuning.
Data Quality & Governance: Implement security, lineage, validation frameworks, and compliance policies.
Programming: Develop in Python, PySpark, Scala, and advanced SQL for data processing and automation.
Collaboration: Partner with data architects and mentor junior engineers.
DevOps: Maintain CI/CD pipelines using Azure DevOps
Databricks Development: Create and tune Spark jobs, leverage Delta Lake, Autoloader, and advanced optimization techniques.
Data Warehousing: Design and manage Snowflake schemas, queries, and performance tuning.
Data Quality & Governance: Implement security, lineage, validation frameworks, and compliance policies.
Programming: Develop in Python, PySpark, Scala, and advanced SQL for data processing and automation.
Collaboration: Partner with data architects and mentor junior engineers.
DevOps: Maintain CI/CD pipelines using Azure DevOp

Key Responsibilities

1. Lead and manage technical teams, providing guidance and support in the implementation of snowflake, azure data factory (adf), and data bricks projects.
2. Design, develop, and maintain data pipelines using snowflake, azure data factory (adf), and data bricks to support business requirements.
3. Work closely with stakeholders to gather requirements, identify opportunities for data analytics, and propose data driven solutions.
4. Monitor and troubleshoot data pipelines, ensuring data quality, reliability, and performance.
5. Stay updated with the latest trends and technologies in data management and analytics, incorporating best practices into projects.
6. Collaborate with cross functional teams to integrate data solutions with existing systems and applications.
7. Provide technical expertise and mentorship to team members, promoting a culture of learning and innovation.

Skill Requirements

JD:
Need experienced Senior Data Engineer (6+ years) to design and optimize scalable data solutions on Microsoft Azure. You will work with Snowflake for data warehousing and Databricks for big data processing within a Lakehouse architecture using medallion principles (Bronze, Silver, Gold).


Key Responsibilities:
Data Pipelines: Build and optimize ETL/ELT pipelines using Azure Data Factory, Databricks (PySpark, SQL), and ingest data from diverse sources into ADLS and Snowflake.
Databricks Development: Create and tune Spark jobs, leverage Delta Lake, Autoloader, and advanced optimization techniques.
Data Warehousing: Design and manage Snowflake schemas, queries, and performance tuning.
Data Quality & Governance: Implement security, lineage, validation frameworks, and compliance policies.
Programming: Develop in Python, PySpark, Scala, and advanced SQL for data processing and automation.
Collaboration: Partner with data architects and mentor junior engineers.
DevOps: Maintain CI/CD pipelines using Azure DevOps

Other Requirements