Data Engineer - Scala/Spark in Bangalore, India - Jobeax
Vacancy description
Data Engineer - Scala/Spark in Bangalore, भारत
Natalie consultants
Bangalore, भारत
Core Responsibilities : - Design and implement large-scale distributed data platforms and data lakes.- Manage data ingestion from diverse sources and file formats.- Perform Big Data orchestration using Airflow, Spark on Kubernetes, Yarn, and Oozie.- Optimize SQL-Hive and Spark queries for performance.- Ensure data validation and quality.- Support 24x7 operations as per the https://jobeax.com/link/esbmFVFD6SFap7xl Requirements : - Advanced Scala (Functional Programming, Case classes, Complex Data Structures & Algorithms).- Big Data Unit, System, Integration & Regression Testing.- DevOps : Jenkins, Maven/sbt, Git, GitHub Actions.- Agile processes & tools : Jira & https://jobeax.com/link/pGAKHe9kGNkPSpVI Advantage : - Data Streaming : Kafka and Spark Streaming.- Data Lake & Medallion Architecture.- Shell Scripting & Automation (Ansible).- Cloud Native Storage (Ceph or S3) and Cloud Data Engineering (Azure, AWS & GCP).- SQL & NoSQL DB : Hive, Cassandra, Dremio, Presto.- Infrastructure : Kubernetes, Docker and related container technologies. (ref:hirist.tech)
... design modern equivalents in Hadoop and Spark environments. Define long-term migration strategy, data partitioning, and retention policies aligned with Visa’s data governance standards. - Data Architecture & Engineering: Implement scalable, distributed data pipelines using Spark, Kafka, Airflow, and Delta Lake. Build robust ...
... 7 years as a Data Engineer or similar role.- Hands-on experience with Databricks, Delta Lake, Spark, and Scala.- Proven ability to design, build, and operate Data Lakes or Data Warehouses.- Proficiency with Data Orchestration tools (Airflow, Dagster, Prefect).- Familiarity with Change Data Capture tools (Canal, Debezium, ...
... customers.- 2+ years of experience supporting Apache Spark pipelines for data engineering, analytics. - Proficient in running Spark jobs from various clients, Spark resource provisioning, Spark parameter configuration and tuning, using Spark History Server and Spark UI, analyzing Spark event logs and other Spark logs.- Solid ...
About the role We are seeking an experienced Data Engineer to design, build, and maintain scalable data pipelines that support our analytics and machine‑learning initiatives. This on‑site position requires deep expertise in Python, Spark, and AWS to deliver reliable, high‑performance data solutions. Key responsibilities ...
... Collaborate with engineering teams to establish best practices in data governance, security, and performance tuning, ensuring the long-term stability of our data https://jobeax.com/link/1dkmzToQOwmnlB8O Skillset :- Demonstrated expertise in building high-performance data pipelines using Python and Apache Spark, with a ...
... statistics, machine learning algorithms, and data mining techniques.- Experience with data preprocessing, feature engineering, and model validation.- Skilled in data visualization tools and libraries (Tableau, Power BI, Matplotlib, Seaborn).- Familiarity with big data platforms (Spark, Hadoop) and SQL databases.- Knowledge ...
Role Overview :We are looking for an experienced Big Data Engineer with strong hands-on expertise in Apache Spark and Scala to design, develop, and optimize large-scale data processing solutions. The role will involve building scalable data pipelines, processing high-volume datasets, optimizing distributed workloads, and ...
... Engineer with Kafka Position: Data Engineer with Kafka Experience: 5+ Years Work Mode: Hybrid (Capco Office) Job Summary We are looking for an experienced Senior Data Engineer with strong expertise in Apache Kafka, Apache Spark, Scala/PySpark, SQL, and modern data engineering technologies . The ideal candidate should have hands-on ...
... Develop data models for reporting, analytics, machine learning and fraud investigations.- Design scalable cloud data solutions using AWS and/or Azure.- Ensure data quality, governance, reliability, monitoring and compliance.- Collaborate with Data Scientists, Fraud Analysts and Engineering teams to deliver data-driven solutions.- ...
... Clear verbal and written communication with the flexibility to work night shifts and overlap with US https://jobeax.com/link/DnRO02YSTsV4WkV8 Certifications : - Databricks Certified Apache Spark Developer / Data Engineer Professional- AWS Certified Data Engineer or GCP Professional Data Engineer (ref:hirist.tech) [{"name": ...