Senior / Lead Data Engineer – Service Layer / QTC Integration Platform

Role Details

Role: Senior / Lead Data Engineer
Primary Skills: Go / Golang, Python, AWS, Kafka / MSK, Microservices, Distributed Systems, Data Engineering
Platform Exposure: AWS Serverless, Kafka, Snowflake, PostgreSQL, ClickHouse, Event-Driven Architecture
Domain Exposure: Quote-to-Cash, CRM, ERP, Customer Lifecycle, Enterprise Data preferred
Experience: 8+ years
Location/ Mode: Bengaluru/Hybrid
Budget : 27 LPA
Contract Duration : 12 months

Role Overview

We are looking for a strong, hands-on, and highly autonomous Senior / Lead Data Engineer to design, build, and operate the Service Layer platform powering enterprise Quote-to-Cash (QTC) , customer lifecycle, finance, and enterprise application integrations.

This role requires deep expertise in distributed systems, event-driven architecture, microservices, data engineering, cloud-native development, and platform reliability . The candidate will also provide technical leadership to a team of engineers, driving solution design, engineering standards, delivery execution, and operational support across multiple integration initiatives.

The platform is built using AWS, Kafka / AWS MSK, Go, Python, canonical data models, asynchronous messaging, and cloud-native architecture patterns , processing high-volume, business-critical events with strong focus on reliability, security, scalability, observability, and operational readiness.

Key Responsibilities

Technical Delivery & Engineering Leadership

Own end-to-end technical delivery of key initiatives, ensuring solutions are delivered with quality, scalability, security, and operational readiness.
Lead and mentor a team of engineers through task planning, technical guidance, code reviews, design reviews, and delivery management.
Drive engineering best practices across software development, CI/CD, observability, resiliency, security, and cloud-native architecture.
Manage multiple concurrent initiatives while balancing business priorities, technical debt, and platform evolution.

Architecture & Platform Design

Drive architecture and design decisions for event-driven integrations , canonical data models, orchestration frameworks, and platform capabilities.
Define and enforce engineering standards, secure-by-design principles, cloud best practices, data governance, and non-functional requirements.
Design and build scalable Service Layer foundation capabilities including messaging, orchestration, observability, reconciliation, data migration, operational tooling, and canonical data management.
Ensure platform design supports resilience, scalability, fault tolerance, performance, and supportability.

Event-Driven Integration & Microservices

Design, develop, and operate real-time event-driven integration services using Go / Golang and Python .
Build and enhance the Service Layer platform to orchestrate Quote-to-Cash workflows across enterprise applications.
Design and implement scalable Kafka / AWS MSK-based event processing with support for:
- Retries
- Idempotency
- Sequencing
- Message replay
- Fault recovery
- Dead-letter queues
- Eventual consistency
Build secure and scalable RESTful APIs , integration services, and enterprise system interfaces.

Data Engineering, Migration & Reconciliation

Design and implement large-scale ETL, migration, reconciliation, and data-quality solutions using:
- AWS Glue
- Step Functions
- Snowflake
- PostgreSQL
- ClickHouse
Build and enforce canonical data models , data contracts, transformation layers, and validation frameworks.
Support enterprise integrations by standardizing communication across multiple business systems.
Ensure data accuracy, traceability, consistency, and operational reliability across the platform.

AWS Cloud-Native Engineering

Design and build cloud-native applications on AWS using:
- AWS Lambda
- API Gateway
- S3
- RDS / Aurora PostgreSQL
- DynamoDB
- SQS
- SNS
- EventBridge
- IAM
- CloudWatch
- VPC
Build high-availability, business-critical applications with strong focus on performance, security, and operational supportability.

Observability, Monitoring & Operations

Design and implement observability, monitoring, alerting, and operational dashboards for end-to-end workflow visibility.
Work with tools such as:
- CloudWatch
- Grafana
- Datadog
- Prometheus
- OpenTelemetry
Support production operations including incident management, root-cause analysis, troubleshooting, and performance optimization.
Drive metrics-driven improvements for reliability, scalability, and platform performance.

Collaboration & Stakeholder Management

Partner with Product Managers, Architects, Enterprise Data teams, Engineering, DevOps, and business stakeholders.
Define technical roadmaps, execution plans, dependencies, and delivery milestones.
Communicate technical risks, trade-offs, design decisions, and mitigation plans clearly to stakeholders.
Collaborate across multiple teams to ensure integration initiatives are delivered efficiently.

Must-Have Skills

Leadership & Delivery

Proven experience leading and mentoring engineering teams.
Strong experience in task planning, technical guidance, code reviews, and delivery management.
Ability to drive architecture decisions for large-scale distributed systems and enterprise integration platforms.
Experience managing multiple parallel technical initiatives.

Programming & Backend Engineering

Strong hands-on experience in:
- Go / Golang
- Python
Experience building scalable backend services, event-driven integrations, orchestration workflows, and data processing solutions.
Strong understanding of REST API development and secure API design.

AWS Cloud & Serverless

Strong experience designing and building cloud-native applications on AWS.
Hands-on experience with:
- Lambda
- API Gateway
- S3
- RDS / Aurora PostgreSQL
- DynamoDB
- SQS / SNS
- EventBridge
- IAM
- CloudWatch
- VPC
Strong understanding of AWS serverless architecture patterns.

Kafka / Event-Driven Systems

Strong experience designing and operating event-driven and asynchronous systems using:
- Kafka
- AWS MSK
- RabbitMQ or equivalent messaging technologies
Strong understanding of:
- Fault tolerance
- Idempotency
- Retry patterns
- Dead-letter queues
- Message replay
- Eventual consistency
- Sequencing
- Event orchestration

Databases & Data Engineering

Strong experience with:
- PostgreSQL
- ClickHouse
- Relational databases
- DynamoDB / NoSQL databases
Strong knowledge of:
- Data modelling
- Query optimization
- Performance tuning
- Data validation
- Reconciliation frameworks
Experience with AWS Glue, Step Functions, Snowflake, and large-scale data workflows.

CI/CD, Security & Observability

Experience with CI/CD pipelines, automated testing, code quality, and secure delivery using tools such as:
- GitLab CI
- Jenkins
- SonarQube
- Trivy
- Semgrep
Strong understanding of cloud security and secure-by-design principles including:
- IAM
- OAuth2
- JWT authentication
- Encryption at rest and in transit
- Secrets Manager
- Vulnerability management
- Audit logging
- Compliance controls

Required Qualifications

Bachelor’s degree in Computer Science, Engineering, Information Technology , or related discipline.
8+ years of software engineering experience.
At least 4+ years focused on cloud-native, data engineering, integration, or distributed platforms.
4+ years building cloud-native solutions on AWS.
4+ years working with Go / Golang, Python, AWS services, and Kafka in production environments.
Proven experience designing, building, and operating highly available enterprise integration and event-driven platforms.
Strong experience in production support, incident management, root-cause analysis, and performance optimization.

Nice to Have

AWS certifications such as:
- AWS Certified Data Engineer – Associate
- AWS Certified Solutions Architect – Associate
- AWS Certified Developer – Associate
- Equivalent AWS certifications
Experience in:
- Quote-to-Cash
- CRM
- ERP
- Customer Lifecycle Management
- Enterprise Data domains
Exposure to large-scale finance, billing, or enterprise integration transformation programmes.
Experience building operational dashboards and platform-level reporting for engineering and business teams.

Senior / Lead Data Engineer - Service Layer / QTC Integration Platform

Start with your email