Key Responsibilities:
Required Skills & Qualifications:
Strong Development Area:
Data Engineering Foundations
Batch vs streaming; lakehouse concepts; medallion (Bronze/Silver/Gold); file formats (Parquet/Delta/CSV/JSON/Avro); partitioning & clustering; schema evolution; data governance basics; DevOps/CI-CD for data.
SQL
Joins, subqueries, CTEs, window functions, set operations; aggregation & rollups; MERGE/UPSERT; analytic functions; performance (indexes, partition pruning, statistics); data validation scenarios (dedupe, top-N, SCD keys).
Apache Spark (Core)
Spark architecture (driver/executors), DAG, stages/tasks; RDD vs DataFrame/Dataset; wide vs narrow transformations; shuffle mechanics; caching/persistence; partitioning; broadcast joins; skew handling; checkpointing; job tuning.
PySpark
DataFrame API, Spark SQL; UDF vs pandas UDF; windowing; incremental loads; structured streaming (triggers, watermarks); handling semi-structured data; optimizing with predicates, pushdown, join strategies; error handling; unit testing (pytest + chispa).
Databricks Platform
Workspace basics; clusters (Single Node/All-Purpose/Jobs), cluster policies; DBR/LTS; notebooks & Repos; Jobs & Workflows; Delta Lake & Delta Live Tables; Unity Catalog (catalog/schema/table, permissions, lineage); MLflow basics; secret scopes; DBFS; REST APIs; Databricks Connect.
Delta Lake / Lakehouse Patterns
ACID transactions; schema enforcement/evolution; time travel; OPTIMIZE/ZORDER; VACUUM; CDC patterns (MERGE INTO, change data feed); streaming vs batch Delta; expectations/constraints; table maintenance strategies.
Orchestration & Scheduling
Databricks Workflows; Azure Data Factory/Synapse pipelines; triggers; parameter passing; fail/retry; alerts; integration with AutoSys/Control-M/Jenkins/GitHub Actions; event-driven patterns.
Python (Core for Data)
Core syntax; typing & data structures; file I/O; logging; virtual environments; packaging; testing (pytest); common data libs (pandas, pyarrow); error handling; performance considerations (vectorization, generators).
Data Quality & Testing
Great Expectations/dbx expectations or custom checks; unit/integration tests; reconciliation (row/amount); anomaly detection; contract testing for schemas; data observability (metrics, SLAs, freshness).
Security & Governance
Unity Catalog permissions, row/column-level security; secrets management (Key Vault/Secret scopes); PII handling; audit logs; token management; compliance basics.
Cost & Performance Optimization
Cluster sizing, autoscaling; DBU awareness; spot/preemptible instances; storage formats; caching; efficient joins; job scheduling; monitoring with metrics & Ganglia/Spark UI; cost tagging and chargeback.
Analytical Skills & Problem Solving
Break down data problems; root-cause incidents; propose alternatives; estimate complexity; communicate clearly with stakeholders.
Stake Holder Management
Interaction with Client Stakeholders, Communication.
Source: HCLTech careers — Read the original posting and apply
This role is listed by the employer on its own careers site. Kaam Ki Khoj does not process applications for it.
Make a free profile in a minute — no CV needed — and we score every job against it.
Make my profileFrom the employer's own careers site. Kaam Ki Khoj reads this job from careers.hcltech.com. You apply there, on the employer's own website.
Applying is free. A genuine employer never asks you to pay: no registration fee, deposit, training fee or kit charge, and never your OTP or bank PIN. If anyone asks for money for this job, it is a scam.
How to spot a fake job · Check an offer free
Upload your CV once — we read it, build your profile and put you in front of every employer hiring on Kaam Ki Khoj. No forms, no fees.