582 spark jobs in India.

• Hybrid

... actionable insights that drive business https://jobeax.com/link/KgVLdM5cThorPSD1 Responsibilities : - Design and implement scalable data pipelines using Scala and Spark to ingest and transform large-scale datasets, ensuring high availability and performance for downstream analytics.- Optimize complex SparkSQL queries and Hadoop ...

• Remote

... understanding of relational and NoSQL databases, including optimization and management techniques.- Experience with distributed data processing frameworks (e.g., Hadoop, Spark, Flink).- Strong foundation in data modeling, ETL processes, and data warehousing principles.- Excellent written and verbal communication skills, with the ability ...

Consultant Data Engineer Databricks | Azure | PySpark | Spark SQL - Bengaluru / Hyderabad / GurugramLooking for an experienced Data Engineer with strong expertise in Azure Databricks, PySpark, Spark SQL, Azure Data Factory (ADF), ADLS, ETL, and SQL. You will design scalable data pipelines, build modern Lakehouse solutions, ...

Consultant Data Engineer Databricks | Azure | PySpark | Spark SQL - Bengaluru / Hyderabad / GurugramLooking for an experienced Data Engineer with strong expertise in Azure Databricks, PySpark, Spark SQL, Azure Data Factory (ADF), ADLS, ETL, and SQL. You will design scalable data pipelines, build modern Lakehouse solutions, ...

• Contract

... management - Kafka Streams DSL and Processor API; stateful and stateless topologies - Kafka Connect — source and sink connectors, lifecycle management - Apache Spark — Structured Streaming, DataFrames, Spark SQL at scale - Java 11+ and Spring Boot 3.x — REST, security, reactive patterns - Deep debugging across multi-threaded, ...

• On-site

... like JSON, Parquet, CSV, and ORC, utilizing batch and streaming ingestion. Cloud Data Migration Leadership : Lead cloud migration projects, developing scalable Spark pipelines. Implement Bronze, Silver, and gold tables for scalable data systems. Optimize Spark code to ensure efficient cloud migration. Data Modeling : Develop ...

• On-site

... candidate will have a solid foundation in coding fundamentals and engineering principles, with hands-on experience in big data technologies such as Scala and Spark. The role involves working with cloud or on-premises data infrastructure and contributing to the design and development of robust, scalable data pipelines and ...

... Delta Lake, Databricks Notification Alerts, Genie Spaces, and performance optimization.- Azure Data Factory (ADF) for ETL/ELT orchestration.- Advanced Python, PySpark, and SQL skills with strong understanding of Spark performance tuning.- Azure Ecosystem : Azure Data Lake Storage (ADLS Gen2), Azure Synapse Analytics, Azure ...

• Full-time

... a related field. - Strong hands-on programming experience with Python for data processing, automation, and pipeline development. - Strong expertise in Apache Spark , particularly PySpark and/or Spark SQL. - Deep working knowledge of the AWS data ecosystem , including S3, Glue, Redshift, Athena, EMR, Kinesis, Lambda, and ...

• Remote
• Hybrid

... Continuously learn mindset and apply new Hadoop ecosystem tools and data technologies. Required Skills and Experience - Proficiency in Hadoop ecosystems such as Spark, HDFS, Hive, Iceberg, Spark SQL. - Extensive experience with Apache Kafka, Apache Flink, and other relevant streaming technologies. - Proven ability to design ...

... Engineer, you'll be part of a team of smart, highly skilled technologists who are passionate about learning and supporting cutting-edge technologies such as Spark, Scala, Pyspark, Databricks, Airflow, SQL, Docker, Kubernetes, and other Data engineering tools. These technologies are deployed using DevOps pipelines leveraging ...

... in cloud-based data platform development - Expertise in building Azure-based data pipelines, including: - Azure Data Factory / Synapse - DataBricks / Synapse Spark Pool - Cosmos DB - Azure Data Lake Storage (ADLS) - Dedicated SQL Pool / Azure SQL - Azure Logic Apps - Hands-on experience with data transformation and cleansing ...

... customers. The Data Pipeline team is tasked with creating, deploying, and managing this platform, utilizing leading technologies like Databricks, Delta Lake, Spark, PySpark, Scala, and the AWS suite. We are actively seeking skilled Data Engineers to join our team and contribute to scaling our platform across the organization. ...

... focuses on real-time streaming, large-scale data processing, cloud platforms, and analytics engineering . The ideal candidate has strong expertise in Apache Spark, Kafka, Snowflake, and AWS/Azure . Build and maintain scalable batch and real-time data pipelines for fraud analytics. Develop and optimize Spark-based data ...

• Full-time

... data engineering team to build, maintain, and optimize scalable data pipelines for large-scale data processing. Develop and implement ETL/ELT processes using PySpark, Spark, and other relevant tools to move and transform data from various sources. Assist in designing and deploying solutions in major cloud platforms such as ...

... Experience in API Gateways, Apache Kafka, Cassandra or other NoSQL DB Nice to Have: Awareness about CI/CD, test automation methodologies Awareness on python, spark as programming languages Knowledge on Kubernetes platform Observability platform Splunk, Kibana, AppDynamics or Dynatrace 6+yrs WFO Immediate to 15days NP Bangalore/ ...

... warehouses.- Optimize database performance, SQL queries, and query execution plans.- Develop and maintain solutions using SQL and advanced PL/SQL.- Use Python or PySpark for data processing and pipeline development.- Ensure data quality, governance, security, and compliance standards.- Monitor and troubleshoot data pipeline and ...

... documentation Essential Technical Skills - Data Engineering: Strong foundation in data engineering principles, ETL/ELT processes, and data pipeline design patterns - PySpark: Proven hands-on experience developing data pipelines using PySpark, including DataFrames API, Spark SQL, and performance optimization - Databricks Platform: ...

... Architecture & Modeling : Enterprise Data Modeling, Dimensional Modeling, Data Vault, Semantic Layer, MDM, KPI & Metric Modeling.- Data Platform Engineering : Apache Spark (PySpark), Python, Advanced SQL, ETL/ELT, Batch & Streaming Processing, Pipeline Design, Performance Tuning.- Data Integration : Apache Kafka, AWS DMS, REST ...

• Hybrid

... pipelines. - Hands-on experience with Cloud Composer / Apache Airflow . - Experience with Dataform and/or dbt . - Experience with Dataflow / Apache Beam or Dataproc / Spark / PySpark . - Strong understanding of data modelling and large-scale datasets. - Experience with incremental and idempotent processing. - Understanding of data ...