Optum - Senior Data Engineer - Databricks & AWS in Pune, भारत

Senior Data Engineer - Databricks & AWSJob Description :We are looking for a Senior Data Engineer with 5+ years of experience in designing, developing, and operating scalable data platforms and production-grade data https://jobeax.com/link/hEiNRx4W5ZDbjxHN ideal candidate will have strong hands-on expertise in Python, SQL, PySpark, Databricks, AWS, and modern data lakehouse technologies, along with exposure to healthcare data standards and privacy https://jobeax.com/link/1grKfAa2qTbtq07R role involves building resilient data ingestion and processing pipelines, optimizing distributed workloads, implementing data quality and observability controls, and contributing to production-grade data engineering https://jobeax.com/link/TtkK7BHPbancvnFq Responsibilities :- Design, develop, and maintain scalable data pipelines and data processing solutions using Python, SQL, Apache Spark/PySpark, and Databricks.- Build and operate production-grade pipelines using Databricks Auto Loader, Delta Lake, Workflows, Unity Catalog, Notebooks/Jobs, and SQL Warehouses.- Optimize Spark and Databricks workloads, including cluster configuration, job performance, query optimization, and distributed processing.- Develop cloud-native data solutions using AWS services, including S3, IAM, VPC, ECS/EKS or Lambda, Step Functions, EventBridge, CloudWatch, Secrets Manager, and KMS.- Work with Apache Iceberg, including tables, catalogs, snapshots, schema and partition evolution, compaction, and interoperability with different query engines.- Design resilient ingestion frameworks for structured and semi-structured data, with appropriate validation, data quality checks, error handling, monitoring, and replay mechanisms.- Implement engineering best practices around Git, pull requests, CI/CD, Docker, Infrastructure as Code, automated testing, observability, and incident response.- Collaborate with cross-functional teams to understand data requirements and translate them into scalable technical solutions.- Ensure appropriate security, access controls, data governance, and privacy measures are incorporated into data engineering workflows.- Contribute to the responsible adoption of AI-assisted coding tools, critically reviewing, testing, securing, and productionizing AI-generated https://jobeax.com/link/KA5Ec3zvvM3f6ASe Skills & Experience :- 5+ years of overall experience in Data Engineering.- Advanced proficiency in Python and SQL.- Strong hands-on experience with Apache Spark/PySpark, distributed processing, and performance tuning.- Production experience with Databricks, including :1. Auto Loader2. Delta Lake3. Databricks Workflows4. Unity Catalog5. Notebooks and Jobs6. SQL Warehouses7. Cluster and job optimization- Strong hands-on experience with AWS, particularly :1. Amazon S32. IAM3. VPC4. ECS/EKS or Lambda5. Step Functions6. EventBridge7. CloudWatch8. Secrets Manager9. KMS- Practical knowledge of Apache Iceberg, including catalogs, snapshots, schema/partition evolution, compaction, and query-engine interoperability.- Experience with Git, CI/CD, Docker, Terraform/CloudFormation, automated testing, observability, and production incident management.- Experience working with semi-structured data and building reliable ingestion pipelines.- Strong understanding of data quality, validation, error handling, monitoring, and replay/reprocessing https://jobeax.com/link/QOQDeleZgQ2WJZ94 Data Experience :- Experience with healthcare data and standards is highly desirable, including :1. FHIR R4 and/or HL7 v22. OMOP Common Data Model (OMOP CDM)3. Clinical terminologies and healthcare data standards4. Handling of PHI (Protected Health Information)5. HIPAA-aligned engineering and security controls6. Data privacy and de-identification conceptsAI & Engineering Practices :- Demonstrated responsible use of AI coding assistants/tools in software and data engineering workflows.- Ability to critically review AI-generated code for correctness, security, performance, maintainability, and production readiness.- Strong focus on testing, documentation, code quality, and engineering best https://jobeax.com/link/5F3kJSDyGonXkMze :- UG : Any GraduateEmployment Type :- Full Time, PermanentDepartment :- Data Science & Analytics (ref:hirist.tech)