Core Responsibilities : - Design and implement large-scale distributed data platforms and data lakes.- Manage data ingestion from diverse sources and file formats.- Perform Big Data orchestration using Airflow, Spark on Kubernetes, Yarn, and Oozie.- Optimize SQL-Hive and Spark queries for performance.- Ensure data validation and quality.- Support 24x7 operations as per the https://jobeax.com/link/esbmFVFD6SFap7xl Requirements : - Advanced Scala (Functional Programming, Case classes, Complex Data Structures & Algorithms).- Big Data Unit, System, Integration & Regression Testing.- DevOps : Jenkins, Maven/sbt, Git, GitHub Actions.- Agile processes & tools : Jira & https://jobeax.com/link/pGAKHe9kGNkPSpVI Advantage : - Data Streaming : Kafka and Spark Streaming.- Data Lake & Medallion Architecture.- Shell Scripting & Automation (Ansible).- Cloud Native Storage (Ceph or S3) and Cloud Data Engineering (Azure, AWS & GCP).- SQL & NoSQL DB : Hive, Cassandra, Dremio, Presto.- Infrastructure : Kubernetes, Docker and related container technologies. (ref:hirist.tech)
... Big Data Engineer – Scala We are looking for an experienced Big Data Engineer with strong expertise in Scala and Apache Spark to design, develop, and optimize scalable data-processing solutions. The ideal candidate should have hands-on experience with distributed data platforms, large-scale ETL/data pipelines, and big data ...
... | Apache Spark | PySpark / https://jobeax.com/link/rvQtoDz2MEe5S7Tx are looking for an experienced Senior Big Data Engineer to design, develop, and optimize scalable big data solutions in an AWS-based https://jobeax.com/link/EG3PG5FSTAqKZnRA Responsibilities : - Design, develop, and maintain scalable big data solutions ...
... software engineers, and business stakeholders to understand data requirements and design efficient data models.- Develop, implement, and maintain robust and scalable data pipelines, ETL processes, and data integration solutions. - Extract, transform, and load data from various sources, ensuring data quality, integrity, ...
... Engineer with Kafka Position: Data Engineer with Kafka Experience: 5+ Years Work Mode: Hybrid (Capco Office) Job Summary We are looking for an experienced Senior Data Engineer with strong expertise in Apache Kafka, Apache Spark, Scala/PySpark, SQL, and modern data engineering technologies . The ideal candidate should have hands-on ...
... Senior Data Engineer (Kafka) Experience: 5+ Years Location: Pune (Preferred) Employment Type: Full-Time Job Summary We are looking for an experienced Senior Data Engineer with strong expertise in Apache Kafka, Apache Spark, Scala/PySpark, SQL, and modern data engineering technologies . The ideal candidate should have hands-on ...
... issues.- Collaborate with data architects, analysts, and engineering teams to support data platform requirements.- Maintain technical documentation and follow data engineering best https://jobeax.com/link/wJIgo45kO3tINlTo Skills :- Apache Spark, PySpark, Hadoop, SQL, Python/Scala/Java, ETL, Big Data ProcessingSupported Skills ...
... issues.- Collaborate with data architects, analysts, and engineering teams to support data platform requirements.- Maintain technical documentation and follow data engineering best https://jobeax.com/link/wJIgo45kO3tINlTo Skills :- Apache Spark, PySpark, Hadoop, SQL, Python/Scala/Java, ETL, Big Data ProcessingSupported Skills ...
... issues.- Collaborate with data architects, analysts, and engineering teams to support data platform requirements.- Maintain technical documentation and follow data engineering best https://jobeax.com/link/wJIgo45kO3tINlTo Skills :- Apache Spark, PySpark, Hadoop, SQL, Python/Scala/Java, ETL, Big Data ProcessingSupported Skills ...
... seeking a skilled Senior Data Engineer with expertise in Databricks to join our dynamic data team. The ideal candidate will design build and maintain scalable data pipelines and architectures to support our organizations data-driven initiatives and leverage Databricks to process large-scale datasets optimize data workflows ...
... using Scala Databricks and Airflow Improve observability alerting and maintain existing spark jobs that power key business metrics. Tech stack includes Airflow Spark Databricks Delta Lake Snowflake Scala Python Collaborate with product managers architects and other engineers to drive the success of the Core Data platform ...
... experience; 3+ years hands-on data engineering - Expert SQL, relational databases, data modeling, and modern data warehousing - Programming proficiency in Python, Scala, or Java for production data solutions - Hands-on cloud platform experience, preferably AWS, GCP, or Azure - Experience with Apache Spark, Hadoop/Hive, Kafka, ...