Data Engineer - Scala/Spark in Bangalore, India - Jobeax
Vacancy description
Data Engineer - Scala/Spark in Bangalore, भारत
Natalie consultants
Bangalore, भारत
Core Responsibilities : - Design and implement large-scale distributed data platforms and data lakes.- Manage data ingestion from diverse sources and file formats.- Perform Big Data orchestration using Airflow, Spark on Kubernetes, Yarn, and Oozie.- Optimize SQL-Hive and Spark queries for performance.- Ensure data validation and quality.- Support 24x7 operations as per the https://jobeax.com/link/esbmFVFD6SFap7xl Requirements : - Advanced Scala (Functional Programming, Case classes, Complex Data Structures & Algorithms).- Big Data Unit, System, Integration & Regression Testing.- DevOps : Jenkins, Maven/sbt, Git, GitHub Actions.- Agile processes & tools : Jira & https://jobeax.com/link/pGAKHe9kGNkPSpVI Advantage : - Data Streaming : Kafka and Spark Streaming.- Data Lake & Medallion Architecture.- Shell Scripting & Automation (Ansible).- Cloud Native Storage (Ceph or S3) and Cloud Data Engineering (Azure, AWS & GCP).- SQL & NoSQL DB : Hive, Cassandra, Dremio, Presto.- Infrastructure : Kubernetes, Docker and related container technologies. (ref:hirist.tech)
... pipelines for large-scale data processing.- Build and optimize distributed data applications using Apache Spark and Scala.- Develop reliable, high-performance data pipelines for batch and streaming workloads.- Work with large datasets to ensure data quality, consistency, and performance.- Collaborate with engineering, product, ...
Data Engineer with ScalaJob Summary :We are seeking a highly skilled and motivated Data Engineer with 6+ years of experience in designing, developing, and maintaining scalable data platforms and https://jobeax.com/link/hEiNRx4W5ZDbjxHN ideal candidate will possess strong expertise in Apache Spark, Scala, Python, SQL, and ...
... : - Strong knowledge of data engineering principles and practices.- Proficiency in programming languages (Python, Java, Scala) and SQL.- Experience with big data platforms and tools (Spark, Hadoop, or cloud data services).- Knowledge of database design and data modeling.- Understanding of ETL and data integration concepts.- ...
... issues in https://jobeax.com/link/doscRB1DTT6ZneaY :- 5+ years of experience in Data Engineering or a related role.- Strong proficiency in SQL and Python, with Scala/Java experience also considered.- Hands-on experience with AWS cloud platforms.- Experience with big data technologies such as Spark, Kafka, or dbt.- Strong ...
... experience in https://jobeax.com/link/FjUoEPg34ABX4bRx will be responsible for designing, building, and maintaining scalable data pipelines and solutions using modern data engineering tools and https://jobeax.com/link/OfTNakiwtq51n9YB Responsibilities : - Design and implement robust, scalable, and high-performance data pipelines ...
... customers.- 2+ years of experience supporting Apache Spark pipelines for data engineering, analytics. - Proficient in running Spark jobs from various clients, Spark resource provisioning, Spark parameter configuration and tuning, using Spark History Server and Spark UI, analyzing Spark event logs and other Spark logs.- Solid ...
... NoSQL databases, such as HBase, Cassandra, MongoDB- Knowledge of ETL techniques and frameworks- Performance tuning of Spark Jobs- Experience with native Cloud data services AWS databricks OR Azure databricks- Ability to lead a team efficiently- Experience with designing and implementing Big data solutions- Practitioner of ...
... statistics, machine learning algorithms, and data mining techniques.- Experience with data preprocessing, feature engineering, and model validation.- Skilled in data visualization tools and libraries (Tableau, Power BI, Matplotlib, Seaborn).- Familiarity with big data platforms (Spark, Hadoop) and SQL databases.- Knowledge ...
... https://jobeax.com/link/T08X3WP3qGUc4E8N Candidate Profile :- 5+ years of experience in Big Data Engineering, Data Engineering, or a closely related role.- Strong hands-on experience with Apache Spark / PySpark and distributed data processing.- Strong programming skills in Python, Scala, or Java.- Strong understanding of data lake, data warehouse, dimensional ...