DataBricks - Data Engineering

Job Description

Required Skills and Experience

• Hands-on experience developing data engineering solutions on Databricks.
• Strong proficiency in Python, PySpark, and SQL, including complex transformations and query optimization.
• Working knowledge of Apache Spark execution, partitioning, joins, caching, and performance troubleshooting.
• Experience with Delta Lake, including merge operations, schema management, and transactional data processing.
• Experience building ETL/ELT pipelines, implementing incremental loads, and applying data warehousing and dimensional modeling concepts.
• Experience with at least one cloud platform—Azure, AWS, or Google Cloud—and its storage and identity services.
• Working knowledge of Databricks jobs, notebooks, compute, Unity Catalog, and access management.
• Proficiency with Git, code reviews, automated testing, and deployment practices.
• Strong analytical, troubleshooting, documentation, and stakeholder communication skills.

Key Responsibilities

• Design, build, and maintain batch and streaming data pipelines using Databricks, PySpark, Python and SQL.
• Ingest and integrate data from databases, files, APIs, and cloud storage into a lakehouse architecture.
• Develop bronze, silver, and gold data layers using Delta Lake and implement incremental processing, deduplication, and change data capture where required.
• Optimize Spark workloads, SQL queries, compute configurations, and storage layouts for performance and cost efficiency.
• Schedule and orchestrate data workflows; implement monitoring, alerting, error handling, and recovery mechanisms.
• Implement data validation, reconciliation, automated testing, and data quality controls.
• Apply data governance and access controls using Unity Catalog and organizational security standards.
• Collaborate with architects, analysts, and business stakeholders to translate requirements into technical solutions.
• Maintain version-controlled code, support CI/CD processes, and document pipelines, data models, and operational procedures.
• Troubleshoot production issues and contribute to root-cause analysis and preventive improvements.

͏

Required Skills and Experience

• Hands-on experience developing data engineering solutions on Databricks.
• Strong proficiency in Python, PySpark, and SQL, including complex transformations and query optimization.
• Working knowledge of Apache Spark execution, partitioning, joins, caching, and performance troubleshooting.
• Experience with Delta Lake, including merge operations, schema management, and transactional data processing.
• Experience building ETL/ELT pipelines, implementing incremental loads, and applying data warehousing and dimensional modeling concepts.
• Experience with at least one cloud platform—Azure, AWS, or Google Cloud—and its storage and identity services.
• Working knowledge of Databricks jobs, notebooks, compute, Unity Catalog, and access management.
• Proficiency with Git, code reviews, automated testing, and deployment practices.
• Strong analytical, troubleshooting, documentation, and stakeholder communication skills.

Key Responsibilities

• Design, build, and maintain batch and streaming data pipelines using Databricks, PySpark, Python and SQL.
• Ingest and integrate data from databases, files, APIs, and cloud storage into a lakehouse architecture.
• Develop bronze, silver, and gold data layers using Delta Lake and implement incremental processing, deduplication, and change data capture where required.
• Optimize Spark workloads, SQL queries, compute configurations, and storage layouts for performance and cost efficiency.
• Schedule and orchestrate data workflows; implement monitoring, alerting, error handling, and recovery mechanisms.
• Implement data validation, reconciliation, automated testing, and data quality controls.
• Apply data governance and access controls using Unity Catalog and organizational security standards.
• Collaborate with architects, analysts, and business stakeholders to translate requirements into technical solutions.
• Maintain version-controlled code, support CI/CD processes, and document pipelines, data models, and operational procedures.
• Troubleshoot production issues and contribute to root-cause analysis and preventive improvements.

͏

Required Skills and Experience

• Hands-on experience developing data engineering solutions on Databricks.
• Strong proficiency in Python, PySpark, and SQL, including complex transformations and query optimization.
• Working knowledge of Apache Spark execution, partitioning, joins, caching, and performance troubleshooting.
• Experience with Delta Lake, including merge operations, schema management, and transactional data processing.
• Experience building ETL/ELT pipelines, implementing incremental loads, and applying data warehousing and dimensional modeling concepts.
• Experience with at least one cloud platform—Azure, AWS, or Google Cloud—and its storage and identity services.
• Working knowledge of Databricks jobs, notebooks, compute, Unity Catalog, and access management.
• Proficiency with Git, code reviews, automated testing, and deployment practices.
• Strong analytical, troubleshooting, documentation, and stakeholder communication skills.

Key Responsibilities

• Design, build, and maintain batch and streaming data pipelines using Databricks, PySpark, Python and SQL.
• Ingest and integrate data from databases, files, APIs, and cloud storage into a lakehouse architecture.
• Develop bronze, silver, and gold data layers using Delta Lake and implement incremental processing, deduplication, and change data capture where required.
• Optimize Spark workloads, SQL queries, compute configurations, and storage layouts for performance and cost efficiency.
• Schedule and orchestrate data workflows; implement monitoring, alerting, error handling, and recovery mechanisms.
• Implement data validation, reconciliation, automated testing, and data quality controls.
• Apply data governance and access controls using Unity Catalog and organizational security standards.
• Collaborate with architects, analysts, and business stakeholders to translate requirements into technical solutions.
• Maintain version-controlled code, support CI/CD processes, and document pipelines, data models, and operational procedures.
• Troubleshoot production issues and contribute to root-cause analysis and preventive improvements.

͏

Required Skills and Experience

• Hands-on experience developing data engineering solutions on Databricks.
• Strong proficiency in Python, PySpark, and SQL, including complex transformations and query optimization.
• Working knowledge of Apache Spark execution, partitioning, joins, caching, and performance troubleshooting.
• Experience with Delta Lake, including merge operations, schema management, and transactional data processing.
• Experience building ETL/ELT pipelines, implementing incremental loads, and applying data warehousing and dimensional modeling concepts.
• Experience with at least one cloud platform—Azure, AWS, or Google Cloud—and its storage and identity services.
• Working knowledge of Databricks jobs, notebooks, compute, Unity Catalog, and access management.
• Proficiency with Git, code reviews, automated testing, and deployment practices.
• Strong analytical, troubleshooting, documentation, and stakeholder communication skills.

Key Responsibilities

• Design, build, and maintain batch and streaming data pipelines using Databricks, PySpark, Python and SQL.
• Ingest and integrate data from databases, files, APIs, and cloud storage into a lakehouse architecture.
• Develop bronze, silver, and gold data layers using Delta Lake and implement incremental processing, deduplication, and change data capture where required.
• Optimize Spark workloads, SQL queries, compute configurations, and storage layouts for performance and cost efficiency.
• Schedule and orchestrate data workflows; implement monitoring, alerting, error handling, and recovery mechanisms.
• Implement data validation, reconciliation, automated testing, and data quality controls.
• Apply data governance and access controls using Unity Catalog and organizational security standards.
• Collaborate with architects, analysts, and business stakeholders to translate requirements into technical solutions.
• Maintain version-controlled code, support CI/CD processes, and document pipelines, data models, and operational procedures.
• Troubleshoot production issues and contribute to root-cause analysis and preventive improvements.

͏

Required Skills and Experience

• Hands-on experience developing data engineering solutions on Databricks.
• Strong proficiency in Python, PySpark, and SQL, including complex transformations and query optimization.
• Working knowledge of Apache Spark execution, partitioning, joins, caching, and performance troubleshooting.
• Experience with Delta Lake, including merge operations, schema management, and transactional data processing.
• Experience building ETL/ELT pipelines, implementing incremental loads, and applying data warehousing and dimensional modeling concepts.
• Experience with at least one cloud platform—Azure, AWS, or Google Cloud—and its storage and identity services.
• Working knowledge of Databricks jobs, notebooks, compute, Unity Catalog, and access management.
• Proficiency with Git, code reviews, automated testing, and deployment practices.
• Strong analytical, troubleshooting, documentation, and stakeholder communication skills.

Key Responsibilities

• Design, build, and maintain batch and streaming data pipelines using Databricks, PySpark, Python and SQL.
• Ingest and integrate data from databases, files, APIs, and cloud storage into a lakehouse architecture.
• Develop bronze, silver, and gold data layers using Delta Lake and implement incremental processing, deduplication, and change data capture where required.
• Optimize Spark workloads, SQL queries, compute configurations, and storage layouts for performance and cost efficiency.
• Schedule and orchestrate data workflows; implement monitoring, alerting, error handling, and recovery mechanisms.
• Implement data validation, reconciliation, automated testing, and data quality controls.
• Apply data governance and access controls using Unity Catalog and organizational security standards.
• Collaborate with architects, analysts, and business stakeholders to translate requirements into technical solutions.
• Maintain version-controlled code, support CI/CD processes, and document pipelines, data models, and operational procedures.
• Troubleshoot production issues and contribute to root-cause analysis and preventive improvements.