Lead the design, architecture, and optimization of enterprise-scale data pipelines using Azure Databricks for ACDP projects, driving innovation in DATA & AI initiatives.
Leverage 7-8 years of expertise to mentor teams, ensure scalability, and deliver mission-critical data solutions.
Key Responsibilities:
Architect and implement end-to-end data pipelines with Azure Databricks (notebooks, jobs, clusters) for ingestion, ETL/ELT, and complex transformations, incorporating Medallion Architecture (Bronze, Silver, Gold) using Delta Lake for ACID compliance, versioning, schema evolution, and data quality.
Provide hands-on leadership in Databricks Spark (PySpark, Scala, SQL), Delta Live Tables, Structured Streaming, and Unity Catalog for advanced governance, security, and multi-workspace management.
Integrate with Azure ecosystem including ADLS Gen2, Azure SQL Database, Serverless/Dedicated SQL Pools; orchestrate via Databricks workflows and hybrid ADF triggers for high-volume ACDP workloads.
Design and deploy CI/CD pipelines using Azure DevOps, Bicep for infrastructure-as code, and advanced automation for Databricks workspaces, emphasizing zero downtime deployments and cost optimization.
Champion data quality, validation, cleansing with Databricks expectations/constraints, and compliance via RBAC, encryption, and Purview; mentor on best practices for Photon engine, auto-scaling, spot instances, and performance tuning
Collaborate with senior analysts, data scientists, and executives to translate complex business requirements into reusable Databricks components, fostering AI/ML integration.
Drive monitoring, troubleshooting, and optimization for petabyte-scale processing, including unstructured data handling, data modeling, and streaming from Kafka/Event Hubs
Requirements
Required Qualifications:
7-8+ years in data engineering, with deep Azure Databricks expertise for big data processing in high-demand environments
Advanced proficiency in Python/PySpark, SQL, Scala; strong data architecture and distributed systems experience
Proven track record in cloud-native solutions (AZ-900+ preferred), CI/CD, and mentoring junior engineers
Excellent communication for stakeholder alignment and documentation
Preferred: Power BI, Power Automate, Azure ML, enterprise governance tools.
Data Engineer (AWS & Databricks)Experience : 3 - 5 YearsRole Summary :We are looking for a Data Engineer with 3 - 5 years of experience specializing in Databricks and AWS to design, build, and optimize scalable data pipelines for analytical and BI https://jobeax.com/link/br6coHVFBehnqWEZ ideal candidate brings hands-on ...
About the Role : We are looking for an experienced Azure Data Engineer with 6 - 15 years of hands-on experience in designing, developing, and implementing scalable data engineering solutions on the Microsoft Azure platform. The ideal candidate should have strong expertise in Azure Databricks, Azure Data Factory, and PySpark, ...
... looking for an experienced Sr. QA Data Engineer to join our Data Engineering team and take ownership of the quality, accuracy, consistency, and reliability of our data pipelines and data-driven systems. The ideal candidate will have strong hands-on experience in Data Testing, ETL/ELT Testing, Database Testing, SQL, Python, Apache ...
... Perform exploratory data analysis, feature engineering, and support data quality assessment Requirements - Bachelor's or Master's degree in Computer Science, Engineering, or related field - 5–8 years of IT or Information Systems experience - Hands-on experience with big data platforms such as Databricks, PySpark, and SparkSQL ...
Thermo Fisher Scientific seeks a Senior Data Platform Engineer with 10+ years of analytics platform experience and 5+ years of hands-on Databricks expertise. This role involves designing, building, and operating secure, scalable data platforms that deliver reliable data, AI, and analytics solutions across the organization ...
Bangalore, भारत
View vacancy
HOLLISTER GLOBAL BUSINESS SERVICES INDIA PRIVATE L
... Description : Sr. Data Developer ITotal Experience : 5-9 yearsWe are seeking an experienced Data Developer with expertise in Microsoft Fabric, Azure Synapse Analytics, Databricks, and strong SQL development https://jobeax.com/link/H2BkKpqLFNhzT57K ideal candidate will work on end-to-end data solutions supporting analytics initiatives ...
... data quality issues. Work with technical teams to ensure appropriate data governance and security. Required Skills & Qualifications 5–8 years of experience in Data Modelling, Data Warehousing or Data Engineering. Strong hands-on experience with Databricks. Strong understanding of Delta Lake and Lakehouse architecture. Excellent ...
... is hiring across all levels and candidates are individually assessed and appropriately leveled based upon their skills and experience. We're looking for an engineer to ensure the reliable operation of production environments for our Data Infrastructure and products, running at scale on large-volume distributed cloud systems. ...
... maintain scalable data transformation pipelines using Python (with tools like PySpark, ADF) and SQL in Azure Databricks.- Implement transformation logic to move data from Bronze to Silver/Gold layers following data engineering best practices.- Apply strong data engineering principles to ensure data reliability, quality, performance, ...
... attribution).- Support partnership conversations with Databricks by contributing technical https://jobeax.com/link/pcbWmnsMMdladkQy Candidate : - Strong hands-on Databricks Architect/Engineer profile with end-to-end build-and-solution capability and Databricks professional certification.- 5+ years in Data engineering/Data architect ...