Lead the design, architecture, and optimization of enterprise-scale data pipelines using Azure Databricks for ACDP projects, driving innovation in DATA & AI initiatives.
Leverage 7-8 years of expertise to mentor teams, ensure scalability, and deliver mission-critical data solutions.
Key Responsibilities:
Architect and implement end-to-end data pipelines with Azure Databricks (notebooks, jobs, clusters) for ingestion, ETL/ELT, and complex transformations, incorporating Medallion Architecture (Bronze, Silver, Gold) using Delta Lake for ACID compliance, versioning, schema evolution, and data quality.
Provide hands-on leadership in Databricks Spark (PySpark, Scala, SQL), Delta Live Tables, Structured Streaming, and Unity Catalog for advanced governance, security, and multi-workspace management.
Integrate with Azure ecosystem including ADLS Gen2, Azure SQL Database, Serverless/Dedicated SQL Pools; orchestrate via Databricks workflows and hybrid ADF triggers for high-volume ACDP workloads.
Design and deploy CI/CD pipelines using Azure DevOps, Bicep for infrastructure-as code, and advanced automation for Databricks workspaces, emphasizing zero downtime deployments and cost optimization.
Champion data quality, validation, cleansing with Databricks expectations/constraints, and compliance via RBAC, encryption, and Purview; mentor on best practices for Photon engine, auto-scaling, spot instances, and performance tuning
Collaborate with senior analysts, data scientists, and executives to translate complex business requirements into reusable Databricks components, fostering AI/ML integration.
Drive monitoring, troubleshooting, and optimization for petabyte-scale processing, including unstructured data handling, data modeling, and streaming from Kafka/Event Hubs
Requirements
Required Qualifications:
7-8+ years in data engineering, with deep Azure Databricks expertise for big data processing in high-demand environments
Advanced proficiency in Python/PySpark, SQL, Scala; strong data architecture and distributed systems experience
Proven track record in cloud-native solutions (AZ-900+ preferred), CI/CD, and mentoring junior engineers
Excellent communication for stakeholder alignment and documentation
Preferred: Power BI, Power Automate, Azure ML, enterprise governance tools.
... and Kappa architecture patterns Lead architecture design, technical feasibility assessments, and solution reviews Design and support cloud data platforms and data modernization initiatives Work on migration and transformation projects involving Azure Databricks and Databricks Lakehouse Guide data engineering, analytics, ...
... scalable ETL/ELT pipelines to integrate disparate data sources, ensuring high data quality and availability for downstream analytics.- Develop and optimize complex data models within Snowflake and Databricks environments to support high-performance reporting and business intelligence requirements.- Collaborate with engineering ...
... seeking a skilled Data Engineer with a strong focus on the Microsoft Fabric ecosystem and Application Database management, who can design and implement scalable data solutions for our clients. Key Responsibilities: - Hands-on experience building and maintaining OLTP databases, including relational data modeling, schema design, ...
... product teams, business partners, and cybersecurity, to identify and clarify requirements for data-driven features and products. They independently understand the data ecosystem, security, privacy, and retention requirements necessary to support business and product features. Sr. Data Engineers create and document technical ...
... Experience : - 6 - 12 years of relevant professional experience, including strong hands-on experience in Azure Data Engineering.- Strong proficiency in Azure Databricks, Python, PySpark and SQL.- Experience with Azure Data Lake Storage (ADLS).- Hands-on experience with Azure Data Factory (ADF) or Databricks Workflows.- ...
Chennai, भारत
View vacancy
CGI Information Systems and Management Consultants
... Warehouse and Lakehouse solutions using Python, Databricks, and Microsoft https://jobeax.com/link/oyyb1fI6qDK3K0Ol is an excellent opportunity for an emerging data professional to grow their skills in cloud data engineering and distributed https://jobeax.com/link/iKnD8scqnw079qNN Experience : 8 - 10 yearsKey Responsibilities ...
Job Title : Azure Data Engineer (Pyspark, Databricks, Delta lake)We are looking for a Data Engineer with 8 - 10 years of experience, including 4+ years of hands-on expertise in Azure Databricks and Azure data https://jobeax.com/link/v8qrFLRTZgSSorDg Skills : - Azure Databricks, PySpark/Scala, Spark SQL, Delta Lake- Azure ...
Job Overview : We are seeking a highly experienced Senior Data Engineer to design, develop, and optimize scalable data pipelines in a cloud-based https://jobeax.com/link/2rce0yCTCWx0P6oA ideal candidate will have deep expertise in PySpark, SQL, Azure Databricks, and experience with either AWS or GCP.A strong foundation ...
... developing high-performing data engineering teams - Bachelor's degree in computer science/engineering with 12+ years experience, or Master's with 11+ years - Strong problem-solving, analytical, and critical thinking abilities for complex data challenges - AWS Certified Data Engineer or Databricks Certificate preferred []