PySpark Developer - ETL Pipelines in Hyderabad, भारत

Key Responsibilities :- Design, develop, and maintain scalable ETL pipelines using PySpark.- Build robust data ingestion frameworks for processing data from multiple source systems.- Develop efficient batch and near real-time data processing workflows.- Perform data extraction, transformation, cleansing, validation, and loading activities.- Optimize PySpark jobs for performance, memory utilization, and execution time.- Develop reusable data transformation components and utility functions.- Write complex SQL queries for data analysis, validation, and reconciliation.- Process large datasets using distributed computing techniques.- Collaborate with business teams to understand data requirements and translate them into technical solutions.- Develop data quality checks and validation frameworks to ensure data accuracy and consistency.- Troubleshoot ETL failures, production issues, and data discrepancies.- Perform root cause analysis and implement permanent fixes for recurring issues.- Integrate data from relational databases, cloud storage, APIs, and flat files.- Support production deployments and monitor ETL job execution.- Participate in code reviews and ensure adherence to coding standards and best practices.- Create technical documentation, data mappings, and process flow diagrams.- Work closely with DevOps and infrastructure teams to improve deployment automation and operational efficiency.- Contribute to continuous improvement initiatives for data engineering https://jobeax.com/link/UuZ31CweuYCeQSOI Skills :- Experience with Azure Databricks or similar Spark-based data platforms.- Understanding of distributed computing and big data processing.- Familiarity with cloud platforms such as Azure, AWS, or GCP.- Experience with data pipeline orchestration tools.- Knowledge of dimensional data modeling and data warehouse architecture.- Experience working with large-scale enterprise data platforms.- Exposure to DevOps practices and automated deployment pipelines. (ref:hirist.tech)