Back to list

Vacancy description

Atyeti - Lead Big Data Engineer - Hadoop/Hive/Spark in भारत

Atyeti IT Services Private Limited

भारत

Key Responsibilities : - Design and execute Hadoop to Databricks migration strategies for large-scale data platforms.- Migrate existing HDFS, Hive, Spark, and MapReduce workloads to Databricks.- Data Engineering - Hadoop to Databricks migration experience.- Strong hands-on experience in migrating and optimizing Spark code.- Analyze existing Hadoop/Spark applications and identify opportunities for modernization and optimization.- Strong hands-on experience in migrating, refactoring, and optimizing Spark code for Databricks.- Develop scalable ETL/ELT pipelines using Spark, SQL, and Databricks.- Optimize Spark jobs by improving partitioning, caching, joins, shuffling, serialization, and memory utilization.- Work with Databricks Delta Lake / Delta tables for reliable and scalable data processing.- Implement Delta Lake features such as ACID transactions, schema evolution, time travel, and optimized data layouts.- Convert legacy Hive SQL / Spark SQL workloads to Databricks SQL and optimize queries.- Migrate data from HDFS/Hive to cloud-based storage and Databricks.- Experience with Azure Data Lake Storage (ADLS), AWS S3, or equivalent cloud storage.- Develop and maintain robust data pipelines using Databricks Workflows and job orchestration tools.- Implement data quality, validation, reconciliation, and error-handling frameworks during migration.- Perform source-to-target mapping, data profiling, transformation analysis, and migration validation.- Troubleshoot performance issues across Spark clusters, notebooks, jobs, and data pipelines.- Work closely with architects, business analysts, data scientists, and application teams to understand data requirements.- Implement CI/CD practices for Databricks notebooks, jobs, and code using Git and DevOps tools.- Follow enterprise security, governance, audit, and compliance standards applicable to banking environments.- Ensure migrated workloads meet performance, scalability, availability, and data-quality requirements. (ref:hirist.tech)

Apply to vacancy

Similar vacancies

More like this role

Capco

Lead Big Data Engineer

... Big Data repositories and enforce standards and principles in the Big Data space. The candidate will be part of an exciting, fast paced environment developing Data Engineering solutions in the data and analytics domain. Role • Develop high quality, secure and scalable data pipelines using spark, Scala/ python on Hadoop or ...

Pune, भारत

Atyeti IT Services Private Limited

Atyeti - Senior/Lead Snowflake Support Engineer - Data Platform

Role Overview : We are looking for a skilled Lead/Senior Snowflake Support Engineer to support and optimize enterprise cloud data platforms, with a strong focus on Snowflake, SQL development, scripting, data management, and data https://jobeax.com/link/rkHQT7VbjPvKYHAb Experience : 2 - 8 yearsKey Responsibilities : - Provide ...

Pune, भारत

Wipro

Big Data Engineer

  • Full-time
... interactive visualizations etc) which can offer ways to improve a business thus affecting business decisions. Must Have: - Strong design and coding skills using Big Data technologies such as Hadoop Spark(Pyspark) Hive and Airflow. - Experience with at least one cloud platform or service such as Azure or AWS. - Data warehousing ...

Bangalore, भारत

Citi

PySpark Big Data Engineer

  • Full-time
Responsibilities - Develop and enhance application systems and programs using application development procedures, programming concepts, and script tools. - Analyze and interpret code to identify system enhancements, vulnerabilities, and security issues.

Pune, भारत

Vserv Digital Services Pvt. Ltd.

Vserv - Python Team Lead - Big Data

Role Overview:We are looking for an experienced Python Team Lead with strong expertise in Big Data, statistical data analysis, AWS cloud services, and Agentic https://jobeax.com/link/1zoyWH1AVQN1pQPx Responsibilities:- Lead, mentor, and guide a team of Python developers/engineers.- Own technical architecture, design, development, ...

Bombay, भारत

Moonfrog Labs

Moonfrog Labs - Lead Data Analyst - Python/Big Data

  • Data Analytics
... they don't: numbers you'll present, causal claims, anything you can't verify. You know the difference between a fast answer and a correct one, and you check.- Big data and pipelines: Redshift, Athena, BigQuery, Hive or Spark, plus orchestration.- Real statistical foundations: experiment design, causal inference basics, ...

Bangalore, भारत

Atyeti IT Services Private Limited

Atyeti - Vulnerability Management Engineer

... findings.- Generate reports and extract relevant data from vulnerability management tools for further analysis.- Work with tool administrators and security engineering teams to address reporting, integration, and data quality issues.- Stay updated on new platform capabilities, vulnerability detection methods, and exposure ...

Bangalore, भारत

Atyeti IT Services Private Limited

Atyeti - Senior Detection Engineer

... https://jobeax.com/link/kobDDMkB48d6CyMn Qualifications :Required Qualifications & Experience :- 7+ years of experience in cybersecurity with a focus on detection engineering, SOC operations, or incident response.- 3+ years of hands-on experience in :i. Detection engineering / SIEM content developmentii. SOAR platforms (CrowdStrike ...

Pune, भारत

Softility

Softility - Big Data Lead - Hadoop/Apache Spark

... https://jobeax.com/link/0kSopwW8F6ag7jjD Technical & Professional Skills : - Experience : 8 to 10 years in software engineering and big data development, including experience leading client-facing delivery teams.- Big Data Stack : Strong hands-on experience with Hadoop (HDFS, MapReduce, YARN), Apache Spark (Core, SQL, Streaming), Apache ...

Hyderabad, भारत

Runmap Infotech Private Ltd.

Senior Big Data Engineer - AWS Platform

... Apache Spark | PySpark / https://jobeax.com/link/rvQtoDz2MEe5S7Tx are looking for an experienced Senior Big Data Engineer to design, develop, and optimize scalable big data solutions in an AWS-based https://jobeax.com/link/EG3PG5FSTAqKZnRA Responsibilities : - Design, develop, and maintain scalable big data solutions and data ...

Bangalore, भारत

Wingify

Wingify - Senior Big Data Engineer - ClickHouse

About the Role:We are looking for a Big Data Engineer to design, build, and operate large-scale data pipelines and analytical infrastructure that transform high-volume raw data into reliable, query-ready datasets for analytics, reporting, and data-driven https://jobeax.com/link/HouIpt6AtbK1KWOU data platform ingests and ...

Delhi, भारत