LogixHealth - Data Engineer - Azure Databricks in Bangalore, भारत

About the Role :We are looking for an experienced Data Engineer to join our global engineering team and build scalable, reliable, and high-performance data solutions for the healthcare technology https://jobeax.com/link/oDjHfcAJ4htFUbSA role will focus on designing and developing robust data pipelines using Apache Spark, PySpark, and Databricks, while leveraging the Azure ecosystem to process and manage large-scale https://jobeax.com/link/izyK9Gp1149KIRva Responsibilities :- Design, develop, and maintain scalable data pipelines using Apache Spark, PySpark, and Databricks.- Build efficient batch and near-real-time data processing solutions for large and complex datasets.- Develop data transformation workflows using Databricks and Delta Lake.- Work extensively with Delta Live Tables (DLT) to build reliable and maintainable data pipelines.- Implement and manage data governance and access controls using Unity Catalog.- Develop reusable and optimized PySpark/Spark code for large-scale data processing.- Design data ingestion and transformation frameworks across various source systems.- Implement data quality, validation, error handling, and reconciliation mechanisms within data pipelines.- Optimize Spark jobs, SQL queries, cluster configurations, partitioning, joins, and data storage strategies to improve performance.- Work with Azure-based data services to build scalable and secure cloud data solutions.- Collaborate with data architects, software engineers, analysts, and other stakeholders to understand data requirements and deliver technical solutions.- Troubleshoot production data pipeline issues and resolve performance, reliability, and data-quality challenges.- Implement monitoring and operational practices to ensure data pipelines meet availability and performance requirements.- Follow engineering best practices around version control, testing, CI/CD, documentation, and deployment.- Contribute to the continuous improvement of data architecture, engineering standards, and development frameworks.- Ensure solutions follow appropriate security, governance, and compliance requirements relevant to healthcare technology https://jobeax.com/link/72xljNf4GfrhBYcv Technical Skills :- 9+ years of hands-on experience in Data Engineering or related technology roles.- Strong hands-on experience with Apache Spark and Databricks.- Strong programming experience in PySpark and Python.- Strong understanding of Spark architecture, transformations, actions, partitioning, joins, caching, and optimization.- Experience working with Databricks Delta Lake.- Hands-on experience with Delta Live Tables (DLT).- Experience with Databricks Unity Catalog and data governance concepts.- Strong understanding of Data Structures and Algorithms (DSA) fundamentals.- Strong SQL skills and experience working with relational and analytical data.- Experience working with Microsoft Azure and Azure-based data engineering services.- Strong understanding of data warehousing, data modelling, ETL/ELT, and data processing concepts.- Experience optimizing large-scale Spark workloads and improving pipeline https://jobeax.com/link/krF72eFyZ8GYItSd to Have :- Experience with Azure Data Factory (ADF).- Experience with Azure Event Hubs or similar event-streaming technologies.- Experience with Apache Airflow or other workflow orchestration tools.- Experience with advanced SQL and query optimization.- Experience building batch and streaming data pipelines.- Exposure to CI/CD practices for Databricks and cloud-based data engineering solutions.- Experience working with healthcare datasets or healthcare technology https://jobeax.com/link/KOtvo7ctwAi2sjdb :- Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.- Strong analytical and problem-solving skills.- Ability to work independently on complex data engineering problems.- Strong communication and collaboration skills.- Ability to work effectively with global and cross-functional engineering teams.- Strong focus on code quality, scalability, reliability, and performance. (ref:hirist.tech)