Job Description:

Job Description: Data Engineer – Databricks & Microsoft Fabric

Location : Chennai, India
Experience : 6 to 8 years

Role Summary
We are looking for an experienced Data Engineer with strong hands-on expertise in Databricks, Microsoft Fabric, PySpark, SQL, and cloud-based data engineering. The candidate will be responsible for designing, developing, and optimizing scalable data pipelines, lakehouse solutions, and analytics-ready data models.
The role requires strong experience in building end-to-end data engineering solutions, working with structured and semi-structured data, implementing data quality controls, and supporting enterprise reporting and analytics platforms.
Key Responsibilities
Data Engineering and Pipeline Development
•    Design, build, and maintain scalable data pipelines using Databricks, PySpark, Spark SQL, and Microsoft Fabric.
•    Develop batch and incremental data ingestion pipelines from multiple source systems.
•    Build and optimize Bronze, Silver, and Gold layer data models using lakehouse architecture.
•    Implement ELT/ETL workflows for data transformation, enrichment, validation, and publishing.
•    Work with structured, semi-structured, and unstructured data formats such as CSV, Parquet, JSON, Delta, and XML.
Databricks Development
•    Develop notebooks, jobs, workflows, and reusable components in Azure Databricks.
•    Implement Delta Lake features such as schema evolution, merge/upsert, time travel, and optimized storage.
•    Optimize Spark jobs for performance, scalability, and cost efficiency.
•    Implement partitioning, caching, indexing, and cluster optimization strategies.
•    Troubleshoot job failures, performance bottlenecks, and data quality issues.
Microsoft Fabric Development
•    Build data solutions using Microsoft Fabric Lakehouse, Warehouse, Data Factory, Pipelines, Notebooks, and OneLake.
•    Develop and manage data pipelines in Fabric for ingestion, transformation, and orchestration.
•    Work with Fabric SQL endpoints, semantic models, and Power BI integration.
•    Support migration or modernization of existing data platforms into Microsoft Fabric.
•    Implement reusable data engineering patterns and framework-based development in Fabric.
Data Quality, Governance, and Security
•    Implement data validation, reconciliation, exception handling, and audit controls.
•    Define and apply data quality rules including null checks, duplicate checks, referential checks, and cross-field validations.
•    Maintain data lineage, metadata, source-to-target mapping, and technical documentation.
•    Ensure data pipelines comply with enterprise security, access control, and governance standards.
•    Support integration with data governance tools such as Microsoft Purview, where applicable.
DevOps and Production Support
•    Implement CI/CD practices for notebooks, pipelines, SQL scripts, and configuration files.
•    Use Git-based version control and deployment processes across environments.
•    Monitor production jobs and resolve incidents within agreed timelines.
•    Prepare runbooks, deployment guides, operational support documents, and handover materials.
•    Collaborate with architects, business analysts, data analysts, and reporting teams to deliver reliable data solutions.
Required Skills
Technical Skills
•    Strong hands-on experience in Azure Databricks.
•    Strong experience in Microsoft Fabric components such as Lakehouse, Warehouse, Data Factory, Pipelines, Notebooks, and OneLake.
•    Proficiency in PySpark, Spark SQL, Python, and SQL.
•    Strong knowledge of Delta Lake, lakehouse architecture, and medallion architecture.
•    Experience with cloud storage and data platforms, preferably Azure Data Lake Storage, Azure SQL, Synapse, or Fabric OneLake.
•    Experience in data ingestion from databases, APIs, files, SFTP, cloud storage, and streaming sources.
•    Good understanding of data modeling, dimensional modeling, and analytics-ready data structures.
•    Experience with performance tuning of Spark jobs and SQL queries.
•    Experience in job scheduling, monitoring, logging, and error handling.
•    Knowledge of CI/CD, Git, Azure DevOps, and deployment automation.
Preferred Skills
•    Experience with Power BI and semantic model integration.
•    Experience in migrating workloads from legacy ETL tools, Synapse, ADF, or Databricks to Microsoft Fabric.
•    Knowledge of Microsoft Purview for data cataloging, lineage, and governance.
•    Experience in building reusable data engineering frameworks.
•    Exposure to real-time or near-real-time data processing.
•    Azure certifications or Databricks certifications are preferred.
Roles and Responsibilities Summary
•    Build and maintain scalable data pipelines using Databricks and Microsoft Fabric.
•    Develop lakehouse solutions using Bronze, Silver, and Gold architecture.
•    Perform data transformation, validation, reconciliation, and publishing.
•    Optimize Spark workloads and SQL queries for performance.
•    Implement data quality, audit, monitoring, and exception handling frameworks.
•    Support deployment, production monitoring, incident resolution, and documentation.
•    Collaborate with cross-functional teams to deliver enterprise data and analytics solutions.
Required Experience
•    6 to 8 years of overall experience in data engineering, ETL/ELT, or data platform development.
•    At least 3+ years of hands-on experience in Databricks / PySpark.
•    At least 1+ year of hands-on experience or strong working knowledge of Microsoft Fabric.
•    Experience working in enterprise-scale data platforms and analytics projects.
•    Experience in Agile delivery models and production support environments.
Educational Qualification
Bachelor’s degree in computer science, Information Technology, Engineering, Data Analytics, or a related discipline.
Good to Have Certifications
•    Microsoft Certified: Fabric Analytics Engineer Associate
•    Microsoft Certified: Azure Data Engineer Associate
•    Databricks Certified Data Engineer Associate / Professional
•    Microsoft Certified: Azure Fundamentals
Key Deliverables
•    Production-ready data pipelines and notebooks.
•    Optimized Databricks and Fabric workloads.
•    Bronze, Silver, and Gold layer data models.
•    Data quality and reconciliation reports.
•    Source-to-target mapping and technical design documents.
•    Deployment guides, runbooks, and support documentation.

Recruiter Name: Srinija Adapa

Recruiter Email ID: Srinija.Adapa@bs.nttdata.com

Get empowered by NTT DATA Business Solutions!

We transform. SAP® solutions into Value

NTT DATA Business Solutions is a fast-growing international IT company and one of the world’s leading SAP partners. We are a full service provider delivering everything from business consulting to implementation of SAP solutions, including hosting services and support.