Back to list

Vacancy description

Senior Data Engineer in Chennai, भारत

Merrill Communications

  • Full-timeStandard weekly hours

Chennai, भारत

Job Description: Responsibilities:

  • Develop & Optimize Data Pipelines
  • Build, test, and maintain ETL/ELT data pipelines using Azure Databricks & Apache Spark (PySpark) .
  • Optimize performance and cost-efficiency of Spark jobs.
  • Ensure data quality through validation, monitoring, and alerting mechanisms.
  • Understand cluster types, configuration, and use-case for serverless
  • Implement Unity Catalog for Data Governance
  • Design and enforce access control policies using Unity Catalog.
  • Manage data lineage, auditing, and metadata governance .
  • Enable secure data sharing across teams and external stakeholders.
  • Integrate with Cloud Data Platforms
  • Work with Azure Data Lake Storage / Azure Blob Storage/ Azure Event Hub to integrate Databricks with cloud-based data lakes, data warehouses, and event streams .
  • Implement Delta Lake for scalable, ACID-compliant storage.
  • Automate & Orchestrate Workflows
  • Develop CI/CD pipelines for data workflows using Azure Databricks Workflows or Azure Data Factory .
  • Monitor and troubleshoot failures in job execution and cluster performance .
  • Collaborate with Stakeholders
  • Work with Data Analysts, Scientists, and Business Teams to understand requirements.
  • Translate business needs into scalable data engineering solutions .
  • API expertise
  • Ability to pull data from a wide variety of APIs using different strategies and methods
Required Skills & Experience:
  • Azure Databricks & Apache Spark (PySpark) – Strong experience in building distributed data pipelines .
  • Python – Proficiency in writing optimized and maintainable Python code for data engineering.
  • Unity Catalog – Hands-on experience implementing data governance, access controls, and lineage tracking .
  • SQL – Strong knowledge of SQL for data transformations and optimizations.
  • Delta Lake – Understanding of time travel, schema evolution, and performance tuning .
  • Workflow Orchestration – Experience with Azure Databricks Jobs or Azure Data Factory .
  • CI/CD & Infrastructure as Code (IaC) – Familiarity with Databricks CLI, Databricks DABs, and DevOps principles .
  • Security & Compliance – Knowledge of IAM, role-based access control (RBAC), and encryption .
Preferred Qualifications:
  • Experience with MLflow for model tracking & deployment in Databricks.
  • Familiarity with streaming technologies (Kafka, Delta Live Tables, Azure Event Hub, Azure Event Grid).
  • Hands-on experience with dbt (Data Build Tool) for modular ETL development.
  • Certification in Databricks, Azure is a plus.
  • Experience with Azure Databricks Lakehouse connectors for SalesForce and SQL Server
  • Experience with Azure Synapse Link for Dynamics, dataverse
  • Familiarity with other data pipeline strategies, like Azure Functions, Fabric, ADF, etc
Soft Skills:
  • Strong problem-solving and debugging skills.
  • Ability to work independently and in teams .
  • Excellent communication and documentation skills.

Similar vacancies

More like this role

  • Full-time
... migrating data pipelines from on-prem or other cloud platforms Understanding of data governance, security, and access control on AWS Experience with CI/CD for data pipelines Nice to have AWS certification (Data Analytics Specialty or Data Engineer Associate) Experience with streaming data (Kinesis, MSK/Kafka) Familiarity ...

Bangalore, भारत

GE HealthCare seeks a Senior Data Engineer to design and implement data engineering solutions, ETL pipelines, and analytics infrastructure. The role involves data profiling, metadata management, and automation using cloud and low-code tools to support data science outcomes. Responsibilities - Design and build automations ...

Bangalore, भारत

  • On-site
  • Full-time
... your world, creating more fulfilling work and projects for our people and clients. For more information, visit [ Job Description We’re looking for a skilled Senior Data Engineer with strong experience in distributed systems, Python/Scala, and modern data engineering tools to help design and implement an end-to-end data ...

Hyderabad, भारत

  • Full-time
Job Summary Valorem Reply is looking for a Data Engineer with experience building and contributing to the design of database systems, both normalized transactional systems and dimensional reporting systems. Strong experience with SQL Server as a database engine, as well as Microsoft and Databricks technology experience ...

Ernakulam, भारत

  • Remote
  • Hybrid
  • Full-time
  • Contract
... perfect your skills in a field filled with exciting challenges and opportunities. About the role As part of the Big Data B2B program, OBS set up a shared Big Data platform and a Data Lake for Use cases exploration and industrialization. We are looking for Senior Data Engineer 5-8 years of experience in building Data Pipelines ...

Gurgaon, भारत

  • On-site
  • Contract
... is a sophisticated technology ecosystem that processes billions of payment events with speed, resilience, and precision. As a Senior Data Engineer within the Data Collection & Engineering (DC&E) organisation, you will play a critical role in designing, building, and scaling Mastercard's next-generation data platforms and ...

Hyderabad, भारत

  • Hybrid
... innovation, and large-scale data engineering? If so, we have an exciting opportunity for you to make a meaningful impact on a global scale. About the Role As a Senior Data Engineer , you will be at the heart of our ambition to become a truly data-driven organisation. You will design and deliver best-in-class data engineering ...

Bangalore, भारत

  • Full-time
Mid-level data science role at KreditBee in Bengaluru. Needs 2–4 years and a https://jobeax.com/link/2DENLGgQM7whDhaE or B.E.. What you'll do - ⁠ ⁠Develop various RestFul APIs & Integrate with various data sources - ⁠ ⁠Develop tools & techniques related to ETL - ⁠ ⁠Exposure in AWS or Other Cloud Platform is advantageous ...

Bangalore, भारत

  • Full-time
  • Contract
Senior Data Engineer POSITION PURPOSE: - The Data & Analytics organization enables Bayer’s Corporate Functions—including Finance, Supply Chain & Logistics, HR, Procurement, Legal, and Sustainability—to become more insight and data-driven. Our mission is to deliver a robust data foundation and FAIR—Findable, Accessible, ...

Bangalore, भारत

  • Full-time
  • IT
... networks combine to deliver a unique set of products and services that help people, businesses and governments realize their greatest potential. Title and Summary Senior Data Engineer Overview We are the global technology company behind the world’s fastest payments processing network. We are a vehicle for commerce, a connection ...

Pune, भारत