68 spark jobs in Pune, India.

... in cloud-based data platform development - Expertise in building Azure-based data pipelines, including: - Azure Data Factory / Synapse - DataBricks / Synapse Spark Pool - Cosmos DB - Azure Data Lake Storage (ADLS) - Dedicated SQL Pool / Azure SQL - Azure Logic Apps - Hands-on experience with data transformation and cleansing ...

... warehouses.- Optimize database performance, SQL queries, and query execution plans.- Develop and maintain solutions using SQL and advanced PL/SQL.- Use Python or PySpark for data processing and pipeline development.- Ensure data quality, governance, security, and compliance standards.- Monitor and troubleshoot data pipeline and ...

... documentation Essential Technical Skills - Data Engineering: Strong foundation in data engineering principles, ETL/ELT processes, and data pipeline design patterns - PySpark: Proven hands-on experience developing data pipelines using PySpark, including DataFrames API, Spark SQL, and performance optimization - Databricks Platform: ...

• Hybrid

... pipelines, and support enterprise-scale data processing workloads. Key Responsibilities Administer and support Cloudera CDP/CDH platforms, including HDFS, Hive, Spark, YARN, Hue, and CDE. Develop, deploy, and optimize PySpark and Python-based data processing solutions. Build and maintain CI/CD pipelines using Jenkins and GitHub/Bitbucket. ...

... fast paced environment developing Data Engineering solutions in the data and analytics domain. Develop high quality, secure and scalable data pipelines using spark, Scala/ python on Hadoop or object storage. Leverage new technologies and approaches to innovate with increasingly large data sets. Drive automation and efficiency ...

• Remote

... services or data intensive environments. - Strong experience building and orchestrating data pipelines with Apache Airflow. • Solid working knowledge of Apache Spark for large-scale data processing. - Strong Python and SQL skills, with a focus on clean, maintainable, production-quality code. Experience designing data models ...

• Full-time

... design for analytics and reporting Performance optimization and scalability Preferred Experience Databricks Delta Lake experience Azure Synapse Analytics Python for data engineering Spark SQL optimization Real-time data streaming Data governance and metadata management Agile/Scrum development model CI/CD pipeline experience

... (AWS / Azure / GCP).- Familiarity with MLOps tooling and practices (e.g., model versioning, CI/CD for ML, monitoring).- Experience with big data technologies (Spark, Databricks, etc.).- Bachelor's / Master's / PhD in Computer Science, Statistics, Mathematics, or a related quantitative https://jobeax.com/link/sEOksQVqU173JbV8 ...

... 8+ years of experience in data engineering and architecture, with a proven track record of leading large-scale data initiatives.- Deep expertise in Python, PySpark.- Strong hands-on experience with Databricks (Spark, Delta Lake, Workflows).- Strong experience with AWS (S3, IAM, Textract, Bedrock or equivalent).- Experience ...

... Exposure to GenAI / AI integration with data platforms.- Experience in domain analytics (Insurance, BFSI, Healthcare).- Familiarity with modern data stack tools (dbt, Python frameworks, Spark).Education : - Bachelor's/Master's in Computer Science, Data Engineering, Information Systems, or related field. (ref:hirist.tech)

... Amazon Bedrock, Bedrock Agents, AgentCore, Guardrails, RAG, Q, Prompt engineering and prompt management, multi-agent orchestration.- Data Engineering : Lakehouse, Spark, Iceberg, Delta, streaming, governance, semantic modeling.- Cloud Engineering : Terraform, CDK, CloudFormation, Kubernetes, Docker, CI/CD, https://jobeax.com/link/UYVApRZobwWH3WSv ...

• Remote
• Contract

... geometry simplification - to control runtime and cost.- Big-data and pipeline fluency: Advanced SQL plus distributed processing for large spatial workloads (Spark or Dask), and building reliable, repeatable data pipelines.- Productionizing models: Experience turning models into deployable, real-time APIs in collaboration ...

• Full-time

... on-prem systems to AWS, leveraging native AWS transformation technologies Key Responsibilities Implement end-to-end ETL pipelines using AWS native services like Spark, Step Function, EventBridge, Glue (PySpark, SQL), and Lambda for data extraction, transformation, and loading. Use pre-created utility & for seamless migration, ...

... solutions. - Machine Learning Integration: Collaborating with teams to integrate machine learning models into data pipelines for enhanced data processing. - Utilising Spark & Kafka: Implementing and working with Spark and Kafka for real-time data processing and analytics. What we are looking for? - Proficiency in Python, SQL, and ...

... business workflows and integrate AI solutions with enterprise platforms, APIs, and event streams.- Conduct operational and historical data analysis using SQL, Spark, and Databricks to perform root-cause investigations and drive resolution outcomes.- Partner directly with Operations teams and SMEs to understand business problems, ...

... large datasets, distributed computing, and modern data architectures. Key Responsibilities Design, develop, and maintain big data pipelines using Hadoop and Spark Build scalable data processing frameworks using Scala and Spark Develop and optimize Hive queries and SQL scripts for large datasets Work with distributed storage ...

• On-site
• Contract

... Sales and Marketing teams to identify new opportunities from incoming and existing leads. Demonstrate strong selling, communication, and influencing abilities to spark interest and build rapport with prospects. What Makes You a Great Fit - If you are motivated by cold calling, outbound engagement, and driving new opportunities ...

... Data Lake (ADL) Professionals in the following areas : Role - Azure Data Engineer Azure Data Lake, Data Factory, Pipelines, Synapse (mandatory) SQL, Python, PySpark / Spark-SQL Web tech, relational DBs, data warehousing, ETL design Microsoft Certified Data Engineer preferred User stories, use cases, requirements, design, ...

... should also possess a good understanding of Oracle HCM Cloud Core HR and related HCM data https://jobeax.com/link/uSD6RSIBgrRuP4Cr in Data Hub configuration, Spark, and Scala will be an added https://jobeax.com/link/kRdC5QOGt2ekpDi6 Skills : Oracle Integration Cloud (OIC) : - OIC Setup & Configuration- Integration Development- ...

• Hybrid

... Architect end-to-end data solutions spanning ingestion, transformation, storage, and presentation layers. Develop and optimize ETL/ELT pipelines using Databricks, Spark, and cloud-native technologies. Design and implement Lakehouse architectures and data models following Medallion principles. Collaborate with product, engineering, ...