Data Engineer

Responsibilities

Role demands a highly skilled Data Engineer to design, build, and optimize scalable data pipelines and data platforms. The ideal candidate will have strong expertise in data modeling, cloud-based data architectures, and modern data engineering tools across Azure, Snowflake, and Databricks environments.

Key Responsibilities

Data Engineering & Pipeline Development

• Design, develop, and maintain robust ETL/ELT pipelines using Databricks, PySpark, and Azure Data Factory (ADF).
• Build scalable and efficient data ingestion frameworks for structured and unstructured data.
• Optimize pipeline performance through performance tuning and orchestration best practices.

Data Modeling & Management

• Develop and maintain data models using modern tools (DBT preferred).
• Implement Master Data Management (MDM) solutions to ensure data consistency and integrity.
• Design scalable and efficient Snowflake schemas (star/snowflake schema, dimensional modeling).

Database & Query Optimization

• Write and optimize advanced SQL queries across Snowflake, Azure SQL, and Synapse.
• Develop and manage stored procedures and database objects.
• Ensure efficient data retrieval through indexing, partitioning, and query optimization.

Cloud & Platform Integration

• Work with Azure data services including:

o Azure Data Factory (ADF)

o Azure Data Lake Storage (ADLS)

o Azure Synapse Analytics

o Azure SQL Database

• Integrate and maintain Snowflake with Azure ecosystem.

Python Development

• Develop data transformation and automation scripts using Python libraries:

o pandas

o pyodbc

o SQLAlchemy

• Build reusable components for data processing and validation.

Data Quality, Validation & Monitoring

• Implement data validation rules, quality checks, and anomaly detection frameworks.
• Perform root cause analysis for data inconsistencies.
• Develop dashboards or tools for data quality monitoring.

Collaboration & DevOps

• Use GitHub for version control, branching strategies, and code reviews.
• Manage workload scheduling and dependency management for pipelines.
• Collaborate with cross-functional teams including data analysts, data scientists, and business stakeholders.

Required Skills & Qualifications

• Bachelor’s or Master’s degree in Computer Science, Information Systems, or related field.
• Strong experience in data engineering and data platform development.

Technical requirements

Technical Skills

• Expertise in DBT (preferred) for data modeling.
• Strong SQL skills with hands-on experience in:

o Snowflake

o Azure SQL

o Stored procedures

• Proficiency in Python for data engineering workflows.
• Hands-on experience with:

o Databricks & PySpark

o Azure Data Services (ADF, ADLS, Synapse)

• Strong knowledge of Snowflake architecture and schema design.
• Experience with data validation, quality frameworks, and analysis tools.
• Familiarity with GitHub and CI/CD practices.

Additional responsibilities

Preferred Qualifications

• Experience implementing data governance and MDM solutions.
• Knowledge of performance tuning in distributed processing systems.
• Familiarity with workflow orchestration tools.
• Experience in agile environments (SCRUM/Kanban).

Preferred skills

• Technology › Data Engineering › Databricks
• Technology › Cloud Integration › Azure Data Factory (ADF)
• Technology › Big Data - Data Processing › PySpark
• Technology › Oracle › PL/SQL
• Technology › Data on Cloud-DataStore › Snowflake
• Technology › Cloud Platform › Databases on Azure › Azure SQL Database
• Technology › Cloud Platform › Azure Analytics Services › Azure Data Lake

About the role

• Role: Senior Consultant
• Experience: 5 - 10 years
• Education: Master Of Engineering, Master Of Technology, Bachelor of Engineering, Bachelor Of Technology