Back to list

Vacancy description

Lead Software Engineer - Data Engineer in Chennai, भारत

Caterpillar

  • On-siteWork from the office
  • Full-timeStandard weekly hours

Chennai, भारत

Responsibilities

  • Lead and mentor a team of data engineers and platform developers.
  • Architect scalable and secure AWS-based data platforms.
  • Design and develop batch and near-real-time ingestion pipelines for structured and unstructured data.
  • Build and optimize Snowflake data platforms, including data modeling, governance, metadata management, performance, and cost optimization.
  • Use AWS services to implement end-to-end data pipelines.
  • Establish engineering best practices, code quality, testing, version control, CI/CD standards, and reusable ingestion and transformation frameworks.
  • Drive Agile delivery, automation, continuous improvement, scalability, resiliency, and operational readiness.
  • Partner with product managers, subject matter experts, data science and AI teams, and business stakeholders to deliver scalable data solutions.
  • Operationalize ML models and analytics workflows and support real-time analytics, edge AI, and AI-enabled data engineering use cases.

Requirements

  • 10+ years of experience in data engineering or data platform roles.
  • Strong experience with the AWS data ecosystem, including S3, Glue, Lambda, EMR, and Redshift.
  • Deep expertise in Snowflake architecture, optimization, and data modeling.
  • Strong programming skills in Python and SQL.
  • Extensive experience with data ingestion pipelines and ETL/ELT frameworks.
  • Experience with real-time streaming technologies such as Kafka and Spark Streaming.
  • Experience with CI/CD and engineering tools including GitHub, Jenkins, AWS CloudFormation, and Git.
  • Understanding of distributed systems, scalable architectures, software engineering principles, testing, and design patterns.
  • Experience working with Agile teams and leading engineering teams at scale.
  • Bachelor’s or master’s degree in Computer Science or Engineering.
  • Preferred experience with graph databases such as Neo4j and Neptune; vector databases such as Milvus and OpenSearch; and the NVIDIA ecosystem, including RAPIDS, cuDF, cuML, and cuGraph.
  • Preferred experience integrating AI/ML pipelines or GenAI workflows.

Benefits

  • The position requires working onsite five days per week.
  • Relocation is available.
  • The posting is open from November 4, 2026 through November 12, 2026.
Loading similar vacancies…