Back to list

Vacancy description

Senior Data Engineer in Chennai, भारत

Merrill Communications

  • Full-timeStandard weekly hours

Chennai, भारत

Job Description: Responsibilities:

  • Develop & Optimize Data Pipelines
  • Build, test, and maintain ETL/ELT data pipelines using Azure Databricks & Apache Spark (PySpark) .
  • Optimize performance and cost-efficiency of Spark jobs.
  • Ensure data quality through validation, monitoring, and alerting mechanisms.
  • Understand cluster types, configuration, and use-case for serverless
  • Implement Unity Catalog for Data Governance
  • Design and enforce access control policies using Unity Catalog.
  • Manage data lineage, auditing, and metadata governance .
  • Enable secure data sharing across teams and external stakeholders.
  • Integrate with Cloud Data Platforms
  • Work with Azure Data Lake Storage / Azure Blob Storage/ Azure Event Hub to integrate Databricks with cloud-based data lakes, data warehouses, and event streams .
  • Implement Delta Lake for scalable, ACID-compliant storage.
  • Automate & Orchestrate Workflows
  • Develop CI/CD pipelines for data workflows using Azure Databricks Workflows or Azure Data Factory .
  • Monitor and troubleshoot failures in job execution and cluster performance .
  • Collaborate with Stakeholders
  • Work with Data Analysts, Scientists, and Business Teams to understand requirements.
  • Translate business needs into scalable data engineering solutions .
  • API expertise
  • Ability to pull data from a wide variety of APIs using different strategies and methods
Required Skills & Experience:
  • Azure Databricks & Apache Spark (PySpark) – Strong experience in building distributed data pipelines .
  • Python – Proficiency in writing optimized and maintainable Python code for data engineering.
  • Unity Catalog – Hands-on experience implementing data governance, access controls, and lineage tracking .
  • SQL – Strong knowledge of SQL for data transformations and optimizations.
  • Delta Lake – Understanding of time travel, schema evolution, and performance tuning .
  • Workflow Orchestration – Experience with Azure Databricks Jobs or Azure Data Factory .
  • CI/CD & Infrastructure as Code (IaC) – Familiarity with Databricks CLI, Databricks DABs, and DevOps principles .
  • Security & Compliance – Knowledge of IAM, role-based access control (RBAC), and encryption .
Preferred Qualifications:
  • Experience with MLflow for model tracking & deployment in Databricks.
  • Familiarity with streaming technologies (Kafka, Delta Live Tables, Azure Event Hub, Azure Event Grid).
  • Hands-on experience with dbt (Data Build Tool) for modular ETL development.
  • Certification in Databricks, Azure is a plus.
  • Experience with Azure Databricks Lakehouse connectors for SalesForce and SQL Server
  • Experience with Azure Synapse Link for Dynamics, dataverse
  • Familiarity with other data pipeline strategies, like Azure Functions, Fabric, ADF, etc
Soft Skills:
  • Strong problem-solving and debugging skills.
  • Ability to work independently and in teams .
  • Excellent communication and documentation skills.

Similar vacancies

More like this role

  • Full-time
... migrating data pipelines from on-prem or other cloud platforms Understanding of data governance, security, and access control on AWS Experience with CI/CD for data pipelines Nice to have AWS certification (Data Analytics Specialty or Data Engineer Associate) Experience with streaming data (Kinesis, MSK/Kafka) Familiarity ...

Bangalore, भारत

GE HealthCare seeks a Senior Data Engineer to design and implement data engineering solutions, ETL pipelines, and analytics infrastructure. The role involves data profiling, metadata management, and automation using cloud and low-code tools to support data science outcomes. Responsibilities - Design and build automations ...

Bangalore, भारत

  • On-site
  • Full-time
... your world, creating more fulfilling work and projects for our people and clients. For more information, visit [ Job Description We’re looking for a skilled Senior Data Engineer with strong experience in distributed systems, Python/Scala, and modern data engineering tools to help design and implement an end-to-end data ...

Hyderabad, भारत

  • Full-time
Job Summary Valorem Reply is looking for a Data Engineer with experience building and contributing to the design of database systems, both normalized transactional systems and dimensional reporting systems. Strong experience with SQL Server as a database engine, as well as Microsoft and Databricks technology experience ...

Ernakulam, भारत

  • Remote
  • Hybrid
  • Full-time
  • Contract
... perfect your skills in a field filled with exciting challenges and opportunities. About the role As part of the Big Data B2B program, OBS set up a shared Big Data platform and a Data Lake for Use cases exploration and industrialization. We are looking for Senior Data Engineer 5-8 years of experience in building Data Pipelines ...

Gurgaon, भारत

  • On-site
  • Contract
... is a sophisticated technology ecosystem that processes billions of payment events with speed, resilience, and precision. As a Senior Data Engineer within the Data Collection & Engineering (DC&E) organisation, you will play a critical role in designing, building, and scaling Mastercard's next-generation data platforms and ...

Hyderabad, भारत

  • Hybrid
... innovation, and large-scale data engineering? If so, we have an exciting opportunity for you to make a meaningful impact on a global scale. About the Role As a Senior Data Engineer , you will be at the heart of our ambition to become a truly data-driven organisation. You will design and deliver best-in-class data engineering ...

Bangalore, भारत

Leadership data science role at Commvault in Bengaluru. Needs 8+ years and a https://jobeax.com/link/wOcGjnRrrRFUZzB4, https://jobeax.com/link/2DENLGgQM7whDhaE or B.E.. What you'll do - Platform Architecture & Engineering - Design and implement secure, scalable platform services for Databricks, Azure Data Lake, Microsoft ...

Bangalore, भारत

  • Full-time
Mid-level data science role at KreditBee in Bengaluru. Needs 2–4 years and a https://jobeax.com/link/2DENLGgQM7whDhaE or B.E.. What you'll do - ⁠ ⁠Develop various RestFul APIs & Integrate with various data sources - ⁠ ⁠Develop tools & techniques related to ETL - ⁠ ⁠Exposure in AWS or Other Cloud Platform is advantageous ...

Bangalore, भारत

  • Full-time
  • Contract
Senior Data Engineer POSITION PURPOSE: - The Data & Analytics organization enables Bayer’s Corporate Functions—including Finance, Supply Chain & Logistics, HR, Procurement, Legal, and Sustainability—to become more insight and data-driven. Our mission is to deliver a robust data foundation and FAIR—Findable, Accessible, ...

Bangalore, भारत