Azure Databricks Architect - ETL/PySpark in Bangalore, भारत
SP Staffing Servcies
Bangalore, भारत
Work Location : Pan IndiaTotal Experience : 8 - 20 yearsRole Overview :Design and implement end-to-end data architecture using Azure Databricks and Microsoft Azure https://jobeax.com/link/v8qrFLRTZgSSorDg Responsibilities :- Architect scalable data lakehouse solutions using Delta Lake and medallion architecture (Bronze, Silver, Gold).- Define data ingestion, transformation, integration, and orchestration strategies.- Develop and optimize complex data pipelines using PySpark, Spark SQL, and Python.- Design and implement ETL/ELT pipelines using Azure Data Factory and Azure Databricks.- Implement data governance, security, access control, and compliance using Unity Catalog, RBAC, and Azure security services.- Optimize Spark workloads, cluster configurations, job performance, and cloud costs.- Integrate Databricks with Azure Data Lake Storage Gen2, Azure Synapse Analytics, and other enterprise data sources.- Establish CI/CD, deployment automation, monitoring, and data quality standards. (ref:hirist.tech)
... 8+ years of experience in Software Engineering or Data Engineering, with significant technical leadership exposure.- Strong hands-on experience with Microsoft Azure and Azure Databricks.- Expertise in Data Engineering, ETL/ELT, SQL, Spark/PySpark, and Data Modelling.- Excellent Python programming skills and experience building ...
... 23 large-scale migrations from legacy to modern cloud platforms.- Technical Depth : - Expert-level Python/PySpark and SQL.- Deep expertise in Snowflake or Databricks (Certifications like SnowPro or Databricks Architect are highly preferred).- Familiarity with "Traditional" ETL (Informatica/SSIS) to manage legacy-to-cloud ...
... support. - Design data models (dimensional SCD Medallion Bronze/Silver/Gold) that match how the business actually uses the data. - Build and maintain scalable ETL/ELT pipelines using Python SQL PySpark and Azure Data Factory on Azure Databricks or Microsoft Fabric. - Implement batch incremental/CDC and streaming loads and ...
... drive efficient and scalable data engineering https://jobeax.com/link/XnflZD2dGSZYgXFG Responsibilities : - Design, develop, and optimize data pipelines using Databricks and Apache Airflow.- Implement PySpark-based transformations and processing in Databricks for handling large-scale data.- Develop and maintain SQL-based data ...
... performance Fun activities and team engagement sessions throughout the year Work Location: In person Required Skills: SQL Data Validation Data Quality Testing ETL/ELT Testing Database Testing Data Profiling Data Reconciliation MS Excel Python Power BI Microsoft Fabric / Azure Databricks / PySpark JIRA Required Education: ...
... support. - Design data models (dimensional, SCD, Medallion Bronze/Silver/Gold) that match how the business actually uses the data. - Build and maintain scalable ETL/ELT pipelines using Python, SQL, PySpark and Azure Data Factory on Azure Databricks or Microsoft Fabric. - Implement batch, incremental/CDC and streaming loads, ...
... maintain scalable data engineering solutions using Microsoft Azure services. - Build and optimize ETL/ELT pipelines using Azure Data Factory, Azure Databricks, and Azure Synapse Analytics. - Develop distributed data processing solutions using Apache Spark, PySpark, and Python. - Implement and manage data lake architectures using ...