Lead the design, architecture, and optimization of enterprise-scale data pipelines using Azure Databricks for ACDP projects, driving innovation in DATA & AI initiatives.
Leverage 7-8 years of expertise to mentor teams, ensure scalability, and deliver mission-critical data solutions.
Key Responsibilities:
Architect and implement end-to-end data pipelines with Azure Databricks (notebooks, jobs, clusters) for ingestion, ETL/ELT, and complex transformations, incorporating Medallion Architecture (Bronze, Silver, Gold) using Delta Lake for ACID compliance, versioning, schema evolution, and data quality.
Provide hands-on leadership in Databricks Spark (PySpark, Scala, SQL), Delta Live Tables, Structured Streaming, and Unity Catalog for advanced governance, security, and multi-workspace management.
Integrate with Azure ecosystem including ADLS Gen2, Azure SQL Database, Serverless/Dedicated SQL Pools; orchestrate via Databricks workflows and hybrid ADF triggers for high-volume ACDP workloads.
Design and deploy CI/CD pipelines using Azure DevOps, Bicep for infrastructure-as code, and advanced automation for Databricks workspaces, emphasizing zero downtime deployments and cost optimization.
Champion data quality, validation, cleansing with Databricks expectations/constraints, and compliance via RBAC, encryption, and Purview; mentor on best practices for Photon engine, auto-scaling, spot instances, and performance tuning
Collaborate with senior analysts, data scientists, and executives to translate complex business requirements into reusable Databricks components, fostering AI/ML integration.
Drive monitoring, troubleshooting, and optimization for petabyte-scale processing, including unstructured data handling, data modeling, and streaming from Kafka/Event Hubs
Requirements
Required Qualifications:
7-8+ years in data engineering, with deep Azure Databricks expertise for big data processing in high-demand environments
Advanced proficiency in Python/PySpark, SQL, Scala; strong data architecture and distributed systems experience
Proven track record in cloud-native solutions (AZ-900+ preferred), CI/CD, and mentoring junior engineers
Excellent communication for stakeholder alignment and documentation
Preferred: Power BI, Power Automate, Azure ML, enterprise governance tools.
... scalable ETL/ELT pipelines to integrate disparate data sources, ensuring high data quality and availability for downstream analytics.- Develop and optimize complex data models within Snowflake and Databricks environments to support high-performance reporting and business intelligence requirements.- Collaborate with engineering ...
Data Engineer (AWS & Databricks)Experience : 3 - 5 YearsRole Summary :We are looking for a Data Engineer with 3 - 5 years of experience specializing in Databricks and AWS to design, build, and optimize scalable data pipelines for analytical and BI https://jobeax.com/link/br6coHVFBehnqWEZ ideal candidate brings hands-on ...
Job Overview : We are seeking a highly experienced Senior Data Engineer to design, develop, and optimize scalable data pipelines in a cloud-based https://jobeax.com/link/2rce0yCTCWx0P6oA ideal candidate will have deep expertise in PySpark, SQL, Azure Databricks, and experience with either AWS or GCP.A strong foundation ...
... batch and streaming data pipelines.- Exposure to CI/CD practices for Databricks and cloud-based data engineering solutions.- Experience working with healthcare datasets or healthcare technology https://jobeax.com/link/KOtvo7ctwAi2sjdb :- Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, ...
Bangalore, भारत
View vacancy
HOLLISTER GLOBAL BUSINESS SERVICES INDIA PRIVATE L
... Description : Sr. Data Developer ITotal Experience : 5-9 yearsWe are seeking an experienced Data Developer with expertise in Microsoft Fabric, Azure Synapse Analytics, Databricks, and strong SQL development https://jobeax.com/link/H2BkKpqLFNhzT57K ideal candidate will work on end-to-end data solutions supporting analytics initiatives ...
... developing high-performing data engineering teams - Bachelor's degree in computer science/engineering with 12+ years experience, or Master's with 11+ years - Strong problem-solving, analytical, and critical thinking abilities for complex data challenges - AWS Certified Data Engineer or Databricks Certificate preferred []
... looking for an experienced Sr. QA Data Engineer to join our Data Engineering team and take ownership of the quality, accuracy, consistency, and reliability of our data pipelines and data-driven systems. The ideal candidate will have strong hands-on experience in Data Testing, ETL/ELT Testing, Database Testing, SQL, Python, Apache ...
... Perform exploratory data analysis, feature engineering, and support data quality assessment Requirements - Bachelor's or Master's degree in Computer Science, Engineering, or related field - 5–8 years of IT or Information Systems experience - Hands-on experience with big data platforms such as Databricks, PySpark, and SparkSQL ...
Thermo Fisher Scientific seeks a Senior Data Platform Engineer with 10+ years of analytics platform experience and 5+ years of hands-on Databricks expertise. This role involves designing, building, and operating secure, scalable data platforms that deliver reliable data, AI, and analytics solutions across the organization ...