Vacancy description
Lead Software Engineer - Data Engineer in Chennai, भारत
Caterpillar
- On-siteWork from the office
- Full-timeStandard weekly hours
Chennai, भारत
Responsibilities
- Lead and mentor a team of data engineers and platform developers.
- Architect scalable and secure AWS-based data platforms.
- Design and develop batch and near-real-time ingestion pipelines for structured and unstructured data.
- Build and optimize Snowflake data platforms, including data modeling, governance, metadata management, performance, and cost optimization.
- Use AWS services to implement end-to-end data pipelines.
- Establish engineering best practices, code quality, testing, version control, CI/CD standards, and reusable ingestion and transformation frameworks.
- Drive Agile delivery, automation, continuous improvement, scalability, resiliency, and operational readiness.
- Partner with product managers, subject matter experts, data science and AI teams, and business stakeholders to deliver scalable data solutions.
- Operationalize ML models and analytics workflows and support real-time analytics, edge AI, and AI-enabled data engineering use cases.
Requirements
- 10+ years of experience in data engineering or data platform roles.
- Strong experience with the AWS data ecosystem, including S3, Glue, Lambda, EMR, and Redshift.
- Deep expertise in Snowflake architecture, optimization, and data modeling.
- Strong programming skills in Python and SQL.
- Extensive experience with data ingestion pipelines and ETL/ELT frameworks.
- Experience with real-time streaming technologies such as Kafka and Spark Streaming.
- Experience with CI/CD and engineering tools including GitHub, Jenkins, AWS CloudFormation, and Git.
- Understanding of distributed systems, scalable architectures, software engineering principles, testing, and design patterns.
- Experience working with Agile teams and leading engineering teams at scale.
- Bachelor’s or master’s degree in Computer Science or Engineering.
- Preferred experience with graph databases such as Neo4j and Neptune; vector databases such as Milvus and OpenSearch; and the NVIDIA ecosystem, including RAPIDS, cuDF, cuML, and cuGraph.
- Preferred experience integrating AI/ML pipelines or GenAI workflows.
Benefits
- The position requires working onsite five days per week.
- Relocation is available.
- The posting is open from November 4, 2026 through November 12, 2026.
Loading similar vacancies…