Data Engineer - Scala/Spark in Bangalore, India - Jobeax
Vacancy description
Data Engineer - Scala/Spark in Bangalore, भारत
Natalie consultants
Bangalore, भारत
Core Responsibilities : - Design and implement large-scale distributed data platforms and data lakes.- Manage data ingestion from diverse sources and file formats.- Perform Big Data orchestration using Airflow, Spark on Kubernetes, Yarn, and Oozie.- Optimize SQL-Hive and Spark queries for performance.- Ensure data validation and quality.- Support 24x7 operations as per the https://jobeax.com/link/esbmFVFD6SFap7xl Requirements : - Advanced Scala (Functional Programming, Case classes, Complex Data Structures & Algorithms).- Big Data Unit, System, Integration & Regression Testing.- DevOps : Jenkins, Maven/sbt, Git, GitHub Actions.- Agile processes & tools : Jira & https://jobeax.com/link/pGAKHe9kGNkPSpVI Advantage : - Data Streaming : Kafka and Spark Streaming.- Data Lake & Medallion Architecture.- Shell Scripting & Automation (Ansible).- Cloud Native Storage (Ceph or S3) and Cloud Data Engineering (Azure, AWS & GCP).- SQL & NoSQL DB : Hive, Cassandra, Dremio, Presto.- Infrastructure : Kubernetes, Docker and related container technologies. (ref:hirist.tech)
... pipelines for large-scale data processing.- Build and optimize distributed data applications using Apache Spark and Scala.- Develop reliable, high-performance data pipelines for batch and streaming workloads.- Work with large datasets to ensure data quality, consistency, and performance.- Collaborate with engineering, product, ...
... and high-performance applications. #LI-BK7 Your future duties and responsibilities Core Data Engineering • ETL/ELT Architecture: Design and implement complex data pipelines from disparate sources into our centralized data lake/warehouse. • Big Data Processing: Leverage Apache Spark (via Dataproc) and Python/Scala to handle ...
Data Engineer with ScalaJob Summary :We are seeking a highly skilled and motivated Data Engineer with 6+ years of experience in designing, developing, and maintaining scalable data platforms and https://jobeax.com/link/hEiNRx4W5ZDbjxHN ideal candidate will possess strong expertise in Apache Spark, Scala, Python, SQL, and ...
... : - Strong knowledge of data engineering principles and practices.- Proficiency in programming languages (Python, Java, Scala) and SQL.- Experience with big data platforms and tools (Spark, Hadoop, or cloud data services).- Knowledge of database design and data modeling.- Understanding of ETL and data integration concepts.- ...
... experience in https://jobeax.com/link/FjUoEPg34ABX4bRx will be responsible for designing, building, and maintaining scalable data pipelines and solutions using modern data engineering tools and https://jobeax.com/link/OfTNakiwtq51n9YB Responsibilities : - Design and implement robust, scalable, and high-performance data pipelines ...
... issues in https://jobeax.com/link/doscRB1DTT6ZneaY :- 5+ years of experience in Data Engineering or a related role.- Strong proficiency in SQL and Python, with Scala/Java experience also considered.- Hands-on experience with AWS cloud platforms.- Experience with big data technologies such as Spark, Kafka, or dbt.- Strong ...
... NoSQL databases, such as HBase, Cassandra, MongoDB- Knowledge of ETL techniques and frameworks- Performance tuning of Spark Jobs- Experience with native Cloud data services AWS databricks OR Azure databricks- Ability to lead a team efficiently- Experience with designing and implementing Big data solutions- Practitioner of ...
... https://jobeax.com/link/Kon99UBVdBLT1RIz Qualifications :- B.E/MTech in Computer Science.- 4+ years of experience in software development.- Experience in building scalable products, preferably with Big Data.- Good coding skills in Python and experience in Spark.- Experience in data warehouses or relational databases is mandatory.- ...
... https://jobeax.com/link/T08X3WP3qGUc4E8N Candidate Profile :- 5+ years of experience in Big Data Engineering, Data Engineering, or a closely related role.- Strong hands-on experience with Apache Spark / PySpark and distributed data processing.- Strong programming skills in Python, Scala, or Java.- Strong understanding of data lake, data warehouse, dimensional ...