78 spark jobs in Pune, India.

... - Build and optimize ETL/ELT pipelines for large volumes of structured and unstructured data. - Work with technologies such as Hadoop, HDFS, Hive, Kafka, and Spark SQL . - Perform Spark job performance tuning and optimization . - Develop reusable and maintainable data-processing frameworks. - Troubleshoot production issues ...

... experience in data engineering or data development.- Mandatory hands-on experience with the Palantir Foundry platform.- Strong programming skills in Python and PySpark.- Proficiency in SQL, preferably Spark SQL, including query development and optimization.- Experience building and maintaining scalable data pipelines and data ...

... methodology JavaScript Jenkins Kimball Linux - Redhat Negotiation Netezza NewSQL Oracle Exadata Performance Tuning Perl Platform Update Management Project Management PySpark Python R RDD Optimization SantOs SaS Scala Spark Shell Script Snowflake SPARK SPARK Code Optimization SQL Stakeholder Management Sun Solaris Synapse Talend Teradata ...

... event streaming solutions , ensuring reliable data ingestion, processing, and integration. - Develop production-grade data processing applications using Apache Spark and Scala/PySpark . - Optimize Spark workloads through effective partitioning, caching, skew handling, join optimization, and performance tuning. - Write advanced ...

Responsibilities - Build and implement end-to-end data warehouse, data lake, and lake house projects in big data environments. - Design complex data pipelines using Spark with Scala, Python, or Java on Hadoop or object storage. - Work with Oracle and Netezza databases and apply strong SQL skills. - Participate in Agile/Scrum activities ...

... data from heterogeneous sources into Data Lake and Data Warehouse environments.- Optimize ETL workflows for performance, scalability, and cost efficiency using Spark SQL and PySpark.- Design and implement data solutions following Medallion Architecture (Bronze, Silver, Gold layers).2. Data Pipeline Management : - Hands-on ...

... Make our release cycle boring. Recurring deliveries should not depend on people watching them. - Treat compute cost as a first-class engineering metric across Spark, orchestration, and the serving tier. What we are looking for - 15+ years building production grade data engineering systems, with a minimum of 5 years in a ...

... PubMatic is seeking Data Analytics-focused Senior / Software Engineers with expertise in building and optimizing AI agents, including strong skills in Hadoop, Spark, Scala, Kafka, Spark Streaming, and cloud-based solutions, with proficiency in programming languages such as Scala and Python. You will be responsible for developing ...

... data integration and cloud deployment Mandatory Skills - Python and/or R Programming - SQL & NoSQL Databases - Cloud Platforms ( AWS / Azure / GCP ) - Apache Spark ( Databricks ) - ETL & ELT Development - Data Warehousing & Data Lakes ( Snowflake, Amazon Redshift, etc. ) - GitHub / GitLab - Jenkins - XML, JSON & REST APIs ...

... license is a plus. Salary Probation: Duration: Salary during probation: After probation: Annual CTC: ₹ 2,00,000 - 3,00,000 /year Number of openings 1 About Small Spark Concepts Technologies Private Limited Small Spark Concepts makes SCHORL washable, reusable performance air filters for cars, motorcycles and trucks at its own ...

... experience developing scalable backend services using Java, REST APIs, JDBC, and relational databases. - Strong experience building distributed data pipelines using Spark, Kafka, Hadoop, Snowflake, SQL, and AWS. - Experience designing and optimizing large-scale analytical data platforms for Mobile AdTech using Java, Spark, Kafka, ...

... is preferred, we are happy to consider experience with AWS, GCP, or other cloud platforms. - Technical Stack: - 4+ years of hands-on experience with Python, Pyspark, or SparkSQL is key. Experience with Scala is a plus. - 4+ years of experience with Big Data pipelines or DAG Tools (such as Airflow, Dbt, Data Factory, or similar). ...

... performance. Architecture & Modern Platforms - Architect end-to-end solutions leveraging: - Databricks - Snowflake - Azure Data Factory - AWS Glue - Apache Airflow - Spark AI-Driven ETL Modernization - Drive adoption of AI-enabled development practices. - Lead ETL migration projects using Generative AI-assisted code conversion ...

... paced environment developing Data Engineering solutions in the data and analytics domain. Role • Develop high quality, secure and scalable data pipelines using spark, Scala/ python on Hadoop or object storage. • Leverage new technologies and approaches to innovate with increasingly large data sets. • Drive automation and ...

... scalable cloud data platform design and architecture. - Deep expertise in Databricks for enterprise-scale data engineering workloads. - Strong proficiency in PySpark for distributed data processing and optimization. - Experience with Delta Lake Spark SQL DataFrames and Spark performance tuning. - Strong understanding of distributed ...

... advancements in AI and data engineering to continuously improve https://jobeax.com/link/clLzYnMppY3KNZfx and Tools Required :- Proficiency in Databricks and Apache Spark.- Strong background in RAG, Agentic Ops, machine learning algorithms and frameworks.- Experience with cloud platforms such as AWS, Azure, or Google Cloud.- Knowledge ...

... management - Problem-solving and troubleshooting skills. Technical Skills - Back-end & Front-end : Django, Angular/React, HTML, CSS - Cloud Technologies : AWS, GCP, Azure - Big Data : Hadoop, Spark (knowledge is a plus) - Containerized Deployment : Dockers/Kubernetes (a plus) - Other : Understanding of GoLang is a plus []

... technical direction aligned with business priorities, target architecture and divisional engineering standards. - Lead delivery using Java, Spring Boot, Apache Spark, PySpark, Kubernetes, OpenShift and Google Cloud Platform services such as Cloud Run, Dataproc, GKE and BigQuery. - Develop reusable frameworks, common components ...

... embeddings, tool calling, structured outputs, and AI evaluation frameworks.- Proficiency in one or more programming and analytical technologies such as Python, R, Spark, Scala, and SQL.- Strong understanding of supervised and unsupervised machine learning techniques, including neural networks, XGBoost, support vector machines, ...

... Python, proficiency in Python, SQL, and at least one additional language (e.g. Java or Scala) for data engineering tasks - Hands-on experience utilizing Apache Spark for large-scale data processing, including developing and optimizing data pipelines, performing real-time and batch analytics, and leveraging Spark’s libraries ...