We are seeking an experienced Data Engineering Manager with strong Databricks and modern cloud data engineering expertise to lead the design, development and operation of scalable enterprise data solutions. The role combines technical leadership, people management, architecture oversight and delivery accountability. The successful candidate will establish engineering standards, reusable frameworks and robust practices for secure, governed, reliable and cost-efficient Lakehouse solutions.
Key Responsibilities
Lead, mentor and support a team of data engineers, setting technical direction and encouraging engineering best practices.
Design, build and operate scalable data pipelines and Lakehouse solutions using Databricks, Apache Spark, Delta Lake, SQL, Python and cloud-native services.
Define standards for ingestion, transformation, orchestration, automated testing, deployment, observability, monitoring and production support.
Collaborate with business, analytics, data science, architecture, security and platform teams to translate requirements into high-quality data solutions.
Champion reusable and metadata-driven frameworks, automation, Medallion Architecture patterns and modern data engineering practices.
Embed data quality, security, governance, privacy and compliance controls across pipelines and data products.
Review solution designs, code and deployment approaches to maintain consistency, resilience, maintainability and operational reliability.
Optimize Databricks workloads for performance, scalability, reliability and cost efficiency.
Support Agile delivery, including backlog refinement, sprint planning, estimation, risk management, continuous improvement and technical roadmap development.
Required Qualifications & Experience
Bachelor's or Master's degree in Computer Science, Engineering, Information Technology, Data Science or a related discipline.
Significant professional experience in data engineering, including experience leading or managing data engineering teams.
Strong hands-on experience with Databricks and Lakehouse-based architectures.
Advanced proficiency in Apache Spark / PySpark, SQL, Python, Delta Lake, ETL/ELT, data pipeline development and orchestration.
Strong understanding of Medallion Architecture, distributed processing and modern cloud data-platform patterns.
... 4+ years of experience with Big Data pipelines or DAG Tools (such as Airflow, Dbt, Data Factory, or similar). - 4+ years of Spark experience, especially with Databricks Spark and Delta Lake. - 4+ years of hands-on experience implementing Big Data solutions in a cloud ecosystem, including Data/Delta Lakes. - 5+ years of ...
Databricks Architect with AIRole Overview:We are looking for an experienced Databricks Data Architect to lead the design and implementation of enterprise-scale data platforms using Databricks Lakehouse Architecture. The candidate should have strong expertise in modern data engineering, data architecture, dimensional modeling, ...
... datasets in distributed computing environments. 4. AWS & Databricks - Hands-on experience with AWS cloud-based data engineering solutions. - Experience working with Databricks for large-scale data processing and transformation. - Understanding of cloud-based data platforms, data lakes, and modern data architectures. - Exposure ...
... Lakehouse, Lambda, and Kappa architecture patterns Lead architecture design, technical feasibility assessments, and solution reviews Design and support cloud data platforms and data modernization initiatives Work on migration and transformation projects involving Azure Databricks and Databricks Lakehouse Guide data engineering, ...
Coimbatore, भारत
View vacancy
CGI Information Systems and Management Consultants
... move data from various sources into our Azure Data Lake and Data Warehouse.- Coding & Transformation : Write clean, efficient PySpark and Python code in Azure Databricks to clean, transform, and aggregate data.- Data Integration : Utilize Azure Data Factory (ADF) to orchestrate data movement and schedule automated workflows.- ...
... proficiency in Azure Databricks, Python, PySpark and SQL.- Experience with Azure Data Lake Storage (ADLS).- Hands-on experience with Azure Data Factory (ADF) or Databricks Workflows.- Good understanding of Delta Lake, ETL / ELT pipelines and dimensional data modelling.- Experience integrating or processing SAP S/4HANA data.- ...
Job Title : Azure Data Engineer (Pyspark, Databricks, Delta lake)We are looking for a Data Engineer with 8 - 10 years of experience, including 4+ years of hands-on expertise in Azure Databricks and Azure data https://jobeax.com/link/v8qrFLRTZgSSorDg Skills : - Azure Databricks, PySpark/Scala, Spark SQL, Delta Lake- Azure ...
... project manager, architect, and customer teams to ensure the technical components of the engagement are delivered to meet customer needs.- Mentor and guide junior data engineers and team https://jobeax.com/link/XBLYbA0mrhLjAKbc : - 6+ years of experience in data engineering, data platforms, and analytics on the Databricks platform.- ...
... experienced Data Engineer with strong hands-on expertise in Python, PySpark, SQL, and cloud-based data engineering. The role involves designing and developing scalable data processing solutions, building reusable frameworks, optimizing data transformation pipelines, and delivering reliable data solutions using Azure Databricks and ...