Sedin Technologies - Senior Data Engineer - Databricks in Chennai, भारत

Position Overview:Datakulture, Sedin's Data Analytics & Business Intelligence division, is looking for a highly experienced Senior Data Engineer with 8+ years of hands-on expertise, with a strong focus on Databricks, to join our Chennai team in an in-office capacity. In this role, you will architect and build enterprise-scale data platforms, Lakehouse environments, and complex data pipelines that power analytics and AI workloads for global https://jobeax.com/link/OBSYxKmspgkYWOcn Responsibilities:- Architect, design, and build large-scale data pipelines and Lakehouse solutions on Databricks using PySpark, Spark SQL, and Delta Lake, following Medallion Architecture principles.- Lead the design and implementation of batch and streaming data pipelines using Spark Structured Streaming, Apache Kafka, AWS Kinesis, or Azure Event Hubs.- Develop and manage Databricks Workflows, Unity Catalog, Delta Live Tables (DLT), and cluster configurations for optimal performance and governance.- Engineer robust data ingestion frameworks from diverse sources including RDBMS, REST APIs, flat files, NoSQL stores, and event streams.- Design and optimize data models for analytical workloads; implement advanced performance tuning including Z-ordering, partitioning, compaction, and caching strategies.- Build and maintain cloud data infrastructure on AWS and/or Azure alongside Databricks.- Implement CI/CD pipelines for data engineering workflows; enforce code review standards and Git-based development practices.- Define and enforce data quality frameworks, data observability standards, and data governance policies.- Collaborate with data scientists, ML engineers, and BI developers to deliver feature stores, analytical datasets, and ML-ready data pipelines.- Lead technical discussions, architecture reviews, and stakeholder engagements; mentor and guide junior and mid-level data engineers.- Drive adoption of Infrastructure as Code (IaC) using Terraform or CloudFormation; manage cost optimization and security best https://jobeax.com/link/wJIgo45kO3tINlTo Qualifications & Skills:- B.E. / https://jobeax.com/link/2DENLGgQM7whDhaE / https://jobeax.com/link/s1WvHFV9ITSwCkyS / M.Sc. in Computer Science, Information Technology, Data Engineering, or a related discipline.- 8+ years of proven experience in data engineering, with at least 3 - 4 years of deep, hands-on Databricks expertise in production environments.- Databricks (Expert): PySpark, Spark SQL, Delta Lake, Unity Catalog, Delta Live Tables, Databricks Workflows, cluster tuning, and Databricks SQL.- Cloud Platforms: Strong proficiency in AWS (S3, Glue, Athena, EMR, Redshift, Kinesis, Lambda, Step Functions) and/or Azure (ADLS Gen2, ADF, Synapse Analytics, Azure SQL, Event Hubs).- Programming: Advanced Python (pandas, PySpark) and SQL (complex queries, window functions, performance optimization).- Strong communication and leadership skills; demonstrated ability to drive technical decision-making, engage with senior stakeholders, and mentor engineering teams.- Data Architecture: Deep understanding of Medallion Architecture, Lakehouse design, Data Vault, star/snowflake schema, and streaming data patterns.- DevOps & IaC: Proficiency in CI/CD (GitHub Actions, Azure DevOps, Jenkins), Terraform or CloudFormation, and Git-based development workflows.- Data Governance: Experience implementing data quality frameworks, data observability, and Unity Catalog-based https://jobeax.com/link/jNu8rE2vHzGnMJ64 Qualifications:- Databricks Certified Data Engineer Professional or Associate; AWS Certified Data Engineer or Azure Data Engineer Associate.- Experience delivering data platforms across BFSI, retail, healthcare, manufacturing, or other data-intensive industries.- Exposure to dbt, Apache Iceberg, Apache Flink, Airflow/Prefect, or ML platform tooling (MLflow, Feature Store). (ref:hirist.tech)