Senior Data Engineer - Azure Databricks/PySpark in Bangalore, भारत

Experience : 7 - 11 YearsLocation : Hyderabad / Chennai / BangaloreRole & Responsibilities : - Design, develop, and maintain scalable data pipelines using Azure Databricks, PySpark, and Spark SQL.- Build and orchestrate complex, parameterized data pipelines using Azure Data Factory (ADF).- Develop and implement end-to-end data engineering solutions using Azure Data Lakehouse and Medallion Architecture.- Process structured and unstructured data across batch and streaming environments.- Optimize Spark jobs, SQL queries, and data pipelines for performance, scalability, and reliability.- Implement metadata-driven frameworks for data ingestion, transformation, and data quality.- Develop exception handling, logging, monitoring, and troubleshooting mechanisms for data pipelines.- Work with Databricks Notebooks, Workflows, Unity Catalog, and Delta Lake.- Ensure data security, governance, and lifecycle management across the Azure environment.- Collaborate with data architects, data scientists, BI teams, and other stakeholders to deliver data solutions.- Take ownership of critical modules, conduct code reviews, and guide team members on best https://jobeax.com/link/wJIgo45kO3tINlTo Skills : - Strong experience in Azure Databricks, PySpark, and Spark SQL.- Hands-on experience in Azure Data Factory (ADF) and pipeline orchestration.- Strong understanding of Apache Spark architecture, including RDDs and DataFrames.- Experience in designing and developing scalable data pipelines and ETL/ELT workflows.- Strong SQL skills, including complex query development, optimization, and troubleshooting.- Experience with Azure Data Lakehouse and Medallion Architecture.- Knowledge of Delta Lake, Parquet, Avro, JSON, and CSV file formats.- Experience in batch and streaming data processing.- Strong understanding of data modeling, data warehousing, and dimensional modeling.- Experience with Databricks Workflows, Unity Catalog, and cloud data platforms.- Proficiency in Python, PySpark, SQL, debugging, and unit testing.- Familiarity with Git, CI/CD, and automated deployment practices.- Strong problem-solving, collaboration, and ownership skills. (ref:hirist.tech)