Job Title: Data Engineer Experience Level: 4 - 6 years Location: Hyderabad / Pune
Qualification: BTech/MTech/MCA Mode of Work: Hybrid
We are seeking an experienced AWS Data Engineer with deep expertise in Banking, Financial Services, and Insurance systems. The candidate must be familiar at least in one Data ware housing concepts such as ODS/Marts and must have done at least one end-to-end data processing pipeline designing. He/she should be capable of handling the ETL design and implementation solely and would work towards creating a positive and innovation friendly environment. The incumbent would be responsible for creating new capabilities.
Responsibilities
Develop, maintain, and optimize data processing applications using Python and PySpark.
Build scalable ETL/ELT pipelines to process large volumes of structured and unstructured data.
Perform data extraction, cleansing, transformation, aggregation, and validation.
Work with Apache Spark for distributed data processing and large-scale data transformations.
Write complex SQL queries and optimize database operations.
Develop reusable Python modules, functions, and libraries.
Optimize Spark jobs by analyzing execution plans, partitioning, caching, and performance bottlenecks.
Work with file formats such as CSV, JSON, Parquet.
Integrate data from multiple sources, including databases, APIs, and cloud storage.
Implement error handling, logging, monitoring, and data quality checks.
Collaborate with data engineers, data architects, analysts, and business stakeholders.
Follow coding standards, version control practices, and Agile development methodologies.
Requirements
Requirements
Strong programming knowledge in Python, including functions, OOP, exception handling, and data structures.
Hands-on experience with PySpark DataFrames, Spark SQL, transformations, and actions.
Good understanding of Apache Spark architecture and distributed computing.
Strong SQL skills, including joins, sub queries, CTEs, and window functions.
Experience developing ETL/ELT pipelines and handling large datasets.
Knowledge of data processing formats such as Parquet, JSON, and CSV.
Experience with Git and software development best practices.
Understanding of debugging, performance tuning, and troubleshooting.
Preferred Skills
Experience with cloud platforms such as AWS, Azure, or Google Cloud.
Familiarity with Databricks, Azure Data Factory, AWS Glue, or similar data engineering tools.
Knowledge of data warehouses and lake house architectures.
Experience with Airflow or other workflow orchestration tools.
Familiarity with CI/CD pipelines, Docker, and automated testing.
Knowledge of Delta Lake, Kafka, or streaming data processing.
Benefits
Comprehensive Medical Coverage:
Health insurance of INR 5.0 Lakhs for you and your family (up to 6 members), ensuring complete peace of mind.
Robust Protection Plans:
Group Personal Accident Insurance and Group Term Life Insurance to safeguard you and your loved ones.
Retirement Benefits:
PF and Gratuity provided as per standard government regulations.
Flexible Work Options:
Enjoy hybrid work arrangements & flexible working hours
Generous Leave Policy:
21 days of annual leave, in addition to 10 company-declared holidays.
Employee Well-being Spaces:
Access to a dedicated break-out area with round-the-clock refreshments for relaxation and rejuvenation.
... client products and external vendors.- Work with modern data technologies such as Hadoop, Spark, DBT, Dagster/Airflow, Atlan, Trino, Databricks, Snowflake, and AWS.- Identify, design, and implement automation of manual processes, optimizing data delivery and reducing Cloud costs.- Implement systems to monitor Data Quality, ...
... processes, optimising data delivery and re-designing infrastructure for greater scalability.- Collaborate with stakeholders to ensure seamless integration of data with internal data marts, enhancing advanced reporting.- Setup and maintain data ingestion, streaming, scheduling, and job monitoring automation using AWS services.- ...
... looking for a Data Engineer with excellent SQL skills and advanced Python and PySpark along with other required skillsets.- 5 - 7 years of experience as a Data Engineer with extensive working experience using PySpark, Advanced SQL, Snowflake, complex understanding of SQL and Skills, Spark, Snowflake, Glue and Airflow.- AWS ...
Job Description : We are hiring for a highly skilled and experienced AWS-Scala Data Engineer to join our dynamic https://jobeax.com/link/GAQw2xZfHsdWGuLm ideal candidate will have a strong background in cloud-based data engineering with at least 4 years of hands-on experience in https://jobeax.com/link/FjUoEPg34ABX4bRx ...
... decision-making. Role AI Data Platform Engineer - AWS Experience Guide 8-11 years Primary Skill Area AWS Data Platforms, Glue, EMR, SageMaker, S3, Redshift, Event-Driven Data & Agentic Operations The opportunity Build and operate AWS-native Data & AI platforms with strong data engineering and platform engineering ownership. The role ...
We have vacancies for Aws - Bigdata Specialist https://jobeax.com/link/LSgW7nHRcJryaGUr are looking for a highly skilled Senior AWS Data Engineer to join our growing data engineering team. The ideal candidate will have deep hands-on experience in AWS cloud services, big data processing frameworks, and advanced SQL programming. ...
... https://jobeax.com/link/ieR2UQ5cXUcxBtsX Requirements :- Must have at least 4+ years of hands-on data engineering experience with the recent 2+ years in AWS cloud data warehouses and AWS cloud services.- Must have advanced SQL knowledge and hands-on experience with relational databases and query authoring, plus a cloud data warehouse like AWS ...
... maintain scalable data pipelines and distributed systems. The ideal candidate should have hands-on experience with ETL/ELT pipelines, workflow orchestration, AWS cloud services, data visualization, distributed architectures, and modern data engineering https://jobeax.com/link/TtkK7BHPbancvnFq Responsibilities:- Design, ...
... Services (AWS) technologies, such as Kinesis, S3, Redshift, Athena, Lambda.- Experience with following aspect of data engineering :1. Building and optimizing big data data pipelines, architectures and data sets.2. Manipulating, processing and extracting value from large, disconnected datasets.3. Development of data pipeline ...
... exploratory data analysis (EDA) and create machine learning models as required.- Implement data quality checks, validation rules, and monitoring processes.- Automate data workflows and ensure timely availability of data.- Work with cloud platforms such as Azure, AWS, or GCP for data engineering https://jobeax.com/link/mAexMzztOeDvmTB5 ...