Vacancy description

Yoda Technologies - Databricks Consultant - Python/PySpark in India, भारत

Yoda Technologies Pty Ltd

India, भारत

Yoda Tech is looking for an experienced Databricks Consultant to design, develop, and optimize scalable data engineering solutions using the Databricks https://jobeax.com/link/f5FQc9SxrVvnQKkz ideal candidate will have strong expertise in data engineering, Apache Spark, Python, SQL, and cloud-based data https://jobeax.com/link/TkEUcdyymD55DMj7 candidate will work with business stakeholders, data architects, and engineering teams to build reliable data pipelines, improve data processing performance, and deliver high-quality data https://jobeax.com/link/XnflZD2dGSZYgXFG Responsibilities :- Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.- Build and optimize ETL/ELT workflows for processing structured and semi-structured data.- Develop data transformation logic using Python, PySpark, and SQL.- Implement data ingestion, cleansing, transformation, and integration processes.- Work with the Lakehouse architecture and Delta Lake for reliable data storage and processing.- Optimize Spark jobs, cluster configurations, and data workflows to improve performance and cost efficiency.- Develop and manage batch and, where required, streaming data pipelines.- Implement data quality checks, error handling, logging, and monitoring mechanisms.- Work with cloud-based data services and integrate Databricks with existing data platforms.- Follow data security, access control, governance, and compliance best practices.- Collaborate with architects, analysts, and cross-functional teams to understand requirements and deliver solutions.- Troubleshoot production issues and ensure the reliability and scalability of data pipelines.- Participate in technical discussions, code reviews, solution design, and https://jobeax.com/link/V48pK40B5heE2tJG Skills & Qualifications :- 8 - 11 years of experience in data engineering, data development, or related roles.- Strong hands-on experience with Databricks and its data engineering capabilities.- Proficiency in Python, PySpark, and SQL.- Strong understanding of Apache Spark, distributed data processing, and performance optimization.- Experience designing and implementing ETL/ELT pipelines.- Knowledge of Delta Lake, lakehouse architecture, and data lake concepts.- Experience working with cloud-based data platforms such as AWS, Microsoft Azure, or Google Cloud.- Understanding of data modelling, data integration, and data quality management.- Experience with version control, deployment processes, and data engineering best practices.- Strong analytical, troubleshooting, and problem-solving skills.- Excellent communication and collaboration https://jobeax.com/link/Cexs453KRiH2S1rJ Skills :- Experience with Databricks Workflows and Delta Live Tables/Lakeflow capabilities.- Familiarity with Azure Data Factory, AWS Glue, or comparable data integration tools.- Experience with CI/CD practices for data engineering solutions.- Knowledge of data governance, cataloguing, and access management.- Exposure to streaming technologies such as Kafka or Structured Streaming.- Experience handling large-scale enterprise data platforms and complex data https://jobeax.com/link/rprPKHGpmoPSuNrH Candidate Profile :- The ideal candidate will be an experienced data engineering professional with strong hands-on Databricks expertise, capable of designing scalable data pipelines, optimizing distributed processing, and delivering reliable cloud-based data solutions. (ref:hirist.tech)

Similar vacancies

More like this role

  • Remote
  • Cybersecurity
Yoda Tech is seeking an experienced Network Security Engineer to lead network security operations, manage security incidents, and maintain secure network infrastructure in a 24 x 7 operational https://jobeax.com/link/2rce0yCTCWx0P6oA ideal candidate will have strong hands-on experience with enterprise firewalls, VPN technologies, ...

India, भारत

  • Remote
Role Overview : We are looking for an experienced OT Security SME to lead and support cybersecurity initiatives across Operational Technology (OT) and Industrial Control System (ICS) environments. - Ability to work with cross-functional technical and engineering teams. tech) []

भारत

$30 USD

... experience in Python programming & package https://jobeax.com/link/4XetkpOyOT04KE0o Responsibilities : - Designing and building scalable data pipelines using Databricks with PySpark, supporting batch and near-real-time ingestion, transformation, and processing.- Optimize Spark jobs and manage large-scale data processing using ...

Bangalore, भारत

  • IT
  • QA
... complex joins and CTEs for deep data validation.- ETL & DWH : Deep understanding of ETL and Data Warehouse https://jobeax.com/link/foMcZv4qpsOyQhfA to have :- Databricks Experience : Hands-on experience with Databricks Notebooks and cluster management.- Programming : Strong skills in Python, specifically PySpark and Pandas ...

Pune, भारत

  • IT
  • QA
... complex joins and CTEs for deep data validation.- ETL & DWH : Deep understanding of ETL and Data Warehouse https://jobeax.com/link/foMcZv4qpsOyQhfA to have :- Databricks Experience : Hands-on experience with Databricks Notebooks and cluster management.- Programming : Strong skills in Python, specifically PySpark and Pandas ...

Indhur, भारत

  • Remote
  • IT
  • Data Science
  • DevOps
  • Consulting
... Ability to understand and communicate technical concepts; proficient with Microsoft Project, Visio and Office https://jobeax.com/link/Nwz1gXVGCH0IUEbM Skills : - Python (pandas, numpy, pyspark)- SQL (MS SQL, OracleDB, Terradata)- Google Cloud Platform- CI/CD tools- Google Vertex AI- Big Data (Spark, pig, hive, scoop, kafka etc.)- ...

Hyderabad, भारत

  • On-site
  • Contract
... https://jobeax.com/link/elpmNuUlUrtRmNOJ : Bangalore, Hyderabad, Chennai, Pune, Noida, Gurugram, Kolkata, https://jobeax.com/link/NAObBBXTjXTvpQpJ Description : Oracle SCM Functional Consultant (Fusion Proc and MFG).As an Oracle Proc Maint MFG, CST (SCM) Functional specialist, you will be working both at offshore and onshore client engagements which ...

Bombay, भारत

  • On-site
  • Contract
... https://jobeax.com/link/elpmNuUlUrtRmNOJ : Bangalore, Hyderabad, Chennai, Pune, Noida, Gurugram, Kolkata, https://jobeax.com/link/NAObBBXTjXTvpQpJ Description : Oracle SCM Functional Consultant (Fusion Proc and MFG).As an Oracle Proc Maint MFG, CST (SCM) Functional specialist, you will be working both at offshore and onshore client engagements which ...

Calcutta, भारत

  • On-site
  • Contract
... https://jobeax.com/link/elpmNuUlUrtRmNOJ : Bangalore, Hyderabad, Chennai, Pune, Noida, Gurugram, Kolkata, https://jobeax.com/link/NAObBBXTjXTvpQpJ Description : Oracle SCM Functional Consultant (Fusion Proc and MFG).As an Oracle Proc Maint MFG, CST (SCM) Functional specialist, you will be working both at offshore and onshore client engagements which ...

Hyderabad, भारत

About the job : We are seeking an experienced SAP SD Consultant with 6+ years of experience to join our team at Zensar Technologies Limited. The ideal candidate will have a deep understanding of SAP ECC and S/4 HANA systems and cross-functional https://jobeax.com/link/LXQAwj0FdzyCDg9K Description : As an ESaaS - SAP - Functional ...

Bombay, भारत