Azure Databricks Architect - ETL/PySpark in Bangalore, भारत
SP Staffing Servcies
Bangalore, भारत
Work Location : Pan IndiaTotal Experience : 8 - 20 yearsRole Overview :Design and implement end-to-end data architecture using Azure Databricks and Microsoft Azure https://jobeax.com/link/v8qrFLRTZgSSorDg Responsibilities :- Architect scalable data lakehouse solutions using Delta Lake and medallion architecture (Bronze, Silver, Gold).- Define data ingestion, transformation, integration, and orchestration strategies.- Develop and optimize complex data pipelines using PySpark, Spark SQL, and Python.- Design and implement ETL/ELT pipelines using Azure Data Factory and Azure Databricks.- Implement data governance, security, access control, and compliance using Unity Catalog, RBAC, and Azure security services.- Optimize Spark workloads, cluster configurations, job performance, and cloud costs.- Integrate Databricks with Azure Data Lake Storage Gen2, Azure Synapse Analytics, and other enterprise data sources.- Establish CI/CD, deployment automation, monitoring, and data quality standards. (ref:hirist.tech)
... requirements and translate them into technical specifications and architecture https://jobeax.com/link/SH3XIHB7XPgGBBvi Integration & Pipelines : - Build and oversee ETL/ELT pipelines using Azure Data Factory, Airflow, SSIS, or equivalent tools.- Use Azure Databricks and PySpark for large-scale data processing.- Own pipeline reliability ...
... enterprise-scale data pipelines using Azure Databricks, PySpark, Delta Lake, Python, SQL, and Azure Data Factory, including both batch and streaming workloads.- Hands-on Databricks experience, including notebooks, workflows/jobs, Spark concepts, Delta Lake patterns, and production support.- Python and/or PySpark for transformation, validation, ...
... support. - Design data models (dimensional SCD Medallion Bronze/Silver/Gold) that match how the business actually uses the data. - Build and maintain scalable ETL/ELT pipelines using Python SQL PySpark and Azure Data Factory on Azure Databricks or Microsoft Fabric. - Implement batch incremental/CDC and streaming loads and ...
... experience with ETL concepts & technologies, Data migration testing, Data warehouse testing, and BI Reporting. Hands on experience in working with CSV and JSON using PySpark. High level of Proficiency in writing complex SQL queries in Azure data bricks from Business requirements to read data from source DB hosted in Azure and compare ...
... complex joins and CTEs for deep data validation.- ETL & DWH : Deep understanding of ETL and Data Warehouse https://jobeax.com/link/foMcZv4qpsOyQhfA to have :- Databricks Experience : Hands-on experience with Databricks Notebooks and cluster management.- Programming : Strong skills in Python, specifically PySpark and Pandas ...
... complex joins and CTEs for deep data validation.- ETL & DWH : Deep understanding of ETL and Data Warehouse https://jobeax.com/link/foMcZv4qpsOyQhfA to have :- Databricks Experience : Hands-on experience with Databricks Notebooks and cluster management.- Programming : Strong skills in Python, specifically PySpark and Pandas ...
... data warehousing and analytics. - Strong hands-on architecture expertise in Snowflake and/or Databricks . - Excellent SQL skills and experience with Python PySpark or equivalent data processing technologies. - Strong understanding of cloud platforms such as Azure AWS or GCP. - Expertise in data modeling dimensional modeling ...
... Snowflake.- Excellent SQL skills and experience with Python, PySpark, or equivalent data processing technologies.- Strong understanding of cloud platforms such as Azure, AWS, or GCP.- Expertise in data modeling, dimensional modeling, data warehousing, lakehouse architecture, and ETL/ETL patterns.- Strong BI expertise in Power ...
... implementation of Delta Lake architecture. - Be highly skilled in developing ETL pipelines using Azure Data Factory (ADF) and executing big data processing with PySpark. - Have a solid command of Python and SQL for data manipulation, scripting, and query optimization. - Experience working within Databricks environments for ...