Senior Data Engineer - AWS, DevOps in Bengaluru, India

Role Purpose This role is responsible for driving enterprise-wide data engineering strategy. It ensures the design, implementation, and continuous improvement of data pipelines, scalable and reusable frameworks, and implement data governance and data quality engines. The position enables trusted, governed data across systems, supporting analytics, compliance, and business operations at ELGi.

Data Engineering & Architecture
Design scalable, production-grade ETL/ELT pipelines across Azure, AWS, and GCP.
Develop modular, reusable data engineering frameworks for ingestion, transformation, and orchestration.
Build Spark, SQL, and Python workflows in Databricks.
Optimize compute, storage, and pipeline performance for cost efficiency.
Data Quality, Governance & Security
Build custom data quality (DQ) frameworks with rule-based validation.
Apply row/column-level security and data masking policies.
Manage secure secrets using Azure Key Vault / AWS Secrets Manager.
Pipeline Reliability & DevOps
Set up monitoring, logging, and alerting frameworks for pipeline health.
Drive test-driven data engineering (unit, integration, regression tests).
Use Azure DevOps and GitHub for CI/CD automation and version control.
Collaborate with Data Engineering, Data Architects, Analytics teams, and vendors.
Partner with ERP, SCM, Logistics, CRM, IoT, and Quality business teams.
Guide architectural decisions for data platform modernization.
Drive innovation in ingestion, transformation, governance, and cloud engineering.

Required Skills and Experience

• 8+ years of experience in data engineering.
• Strong programming skills in Python and SQL.
• Multi-cloud experience: Azure, AWS, and/or GCP.
• Experience with Azure Data Factory for orchestration.
• Building reusable ETL/ELT frameworks.
• Custom DQ frameworks and DQ dashboard implementation.
• Experience with Azure DevOps and GitHub for CI/CD.
• Exposure to ERP, SCM, Logistics, CRM, IoT, and Quality systems