Job Title: Data Engineer Experience Level: 4 - 6 years Location: Hyderabad / Pune
Qualification: BTech/MTech/MCA Mode of Work: Hybrid
We are seeking an experienced AWS Data Engineer with deep expertise in Banking, Financial Services, and Insurance systems. The candidate must be familiar at least in one Data ware housing concepts such as ODS/Marts and must have done at least one end-to-end data processing pipeline designing. He/she should be capable of handling the ETL design and implementation solely and would work towards creating a positive and innovation friendly environment. The incumbent would be responsible for creating new capabilities.
Responsibilities
Develop, maintain, and optimize data processing applications using Python and PySpark.
Build scalable ETL/ELT pipelines to process large volumes of structured and unstructured data.
Perform data extraction, cleansing, transformation, aggregation, and validation.
Work with Apache Spark for distributed data processing and large-scale data transformations.
Write complex SQL queries and optimize database operations.
Develop reusable Python modules, functions, and libraries.
Optimize Spark jobs by analyzing execution plans, partitioning, caching, and performance bottlenecks.
Work with file formats such as CSV, JSON, Parquet.
Integrate data from multiple sources, including databases, APIs, and cloud storage.
Implement error handling, logging, monitoring, and data quality checks.
Collaborate with data engineers, data architects, analysts, and business stakeholders.
Follow coding standards, version control practices, and Agile development methodologies.
Requirements
Requirements
Strong programming knowledge in Python, including functions, OOP, exception handling, and data structures.
Hands-on experience with PySpark DataFrames, Spark SQL, transformations, and actions.
Good understanding of Apache Spark architecture and distributed computing.
Strong SQL skills, including joins, sub queries, CTEs, and window functions.
Experience developing ETL/ELT pipelines and handling large datasets.
Knowledge of data processing formats such as Parquet, JSON, and CSV.
Experience with Git and software development best practices.
Understanding of debugging, performance tuning, and troubleshooting.
Preferred Skills
Experience with cloud platforms such as AWS, Azure, or Google Cloud.
Familiarity with Databricks, Azure Data Factory, AWS Glue, or similar data engineering tools.
Knowledge of data warehouses and lake house architectures.
Experience with Airflow or other workflow orchestration tools.
Familiarity with CI/CD pipelines, Docker, and automated testing.
Knowledge of Delta Lake, Kafka, or streaming data processing.
Benefits
Comprehensive Medical Coverage:
Health insurance of INR 5.0 Lakhs for you and your family (up to 6 members), ensuring complete peace of mind.
Robust Protection Plans:
Group Personal Accident Insurance and Group Term Life Insurance to safeguard you and your loved ones.
Retirement Benefits:
PF and Gratuity provided as per standard government regulations.
Flexible Work Options:
Enjoy hybrid work arrangements & flexible working hours
Generous Leave Policy:
21 days of annual leave, in addition to 10 company-declared holidays.
Employee Well-being Spaces:
Access to a dedicated break-out area with round-the-clock refreshments for relaxation and rejuvenation.
... SUMMARY : We are looking for an experienced AWS Data Engineer with hands-on experience in building and maintaining data pipelines, data platforms, and cloud-based data solutions on https://jobeax.com/link/bGlnCchIlVM0MSjQ candidate should have strong experience in Python, SQL, AWS data services, ETL/ELT pipelines, and data warehousing, ...
Role : Data Engineer Location : ChennaiExperience : 5+ yearsInterview : FridayMust-Have Skills : - Python- PySpark- AWS- SQL- Azure Data Factory (ADF)- dbtKey Responsibilities : - Develop and maintain data pipelines and ETL/ELT processes.- Build data solutions using Python and PySpark.- Work with AWS cloud services for ...
... with hands-on experience in one or more of the following technology areas : Snowflake, AWS, Microsoft Azure, and SQL. The ideal candidate should have strong data engineering fundamentals, experience in building and maintaining data pipelines, ETL/ELT processes, data integration, data transformation, and data warehouse ...
... Description Be part of a dynamic team where your distinctive skills will contribute to a winning culture and team. As a Data Engineer III - Databricks, Pyspark, Python, AWS at JPMorgan Chase within the Commercial & Investment Bank, you'll serve as a seasoned member of an agile team to design and deliver trusted data collection, storage, ...
Role : AWS Cloud Engineer for Automotive Data Science, AI & GenAI ExpertRole and Responsibilities :- Design, develop, and implement deployment pipelines for Data Science, ML, AI, and GenAI solutions on AWS cloud.- Build and maintain CI/CD and CT pipelines using GitHub Actions, Airflow, or similar orchestration tools.- Support ...
... Engineer to design, develop, and manage scalable cloud-native data engineering solutions on AWS. The role involves building automated ETL/ELT pipelines, real-time data processing frameworks, and governed data platforms. Key Responsibilities: - Build and maintain scalable data pipelines using AWS, Python, SQL, and Databricks. ...
We are seeking a Data Engineer to join our team. This role involves building and maintaining data pipelines on AWS and on-premises platforms. Key responsibilities - Design and implement data pipelines using AWS services such as S3, Glue, Lambda, and EMR. - Develop ETL workflows in DBT and Pyspark. - Set up and manage Airflow ...
Job Summary:We are looking for a highly skilled DevOps Engineer with experience building, deploying, and managing AWS cloud and data platform services including AWS Glue, Step Functions, Lambda, MSK, and EKS. The role requires expertise in automation, CI/CD, Kubernetes, GitOps, security, observability, and cloud cost https://jobeax.com/link/9mzDTlf0VOLnPxOw ...
Role Overview : We are looking for an experienced AWS DevOps Engineer with strong hands-on expertise in Terraform, AWS infrastructure provisioning, GitLab CI/CD, and DevSecOps practices. The ideal candidate must have practical experience designing and provisioning AWS infrastructure using Terraform, developing reusable ...