PYTHON PYSPARK DEVELOPER in Hyderabad, India - Jobeax
Vacancy description
PYTHON PYSPARK DEVELOPER in Hyderabad, भारत
Vrinda International
Full-timeStandard weekly hours
Rs 20 - 25 lakhs p.a.
Hyderabad, भारत
Python & PySpark Developer
Summary
We are seeking an experienced Python and PySpark Developer to join our IT Services team. In this pivotal role, you will architect and implement robust data solutions, focusing on the design, development, and optimization of complex data pipelines. You will play a critical part in transforming raw data into actionable insights by leveraging cloud-native technologies and advanced processing frameworks. This position requires a deep understanding of data engineering principles to ensure high availability, scalability, and efficiency across our data infrastructure.
Responsibilities
Design, build, and maintain scalable ETL and ELT pipelines to facilitate seamless data movement and transformation.
Develop high-performance data processing applications using Python and PySpark to handle large-scale datasets.
Integrate and manage various AWS data services to support cloud-based data operations and storage requirements.
Create and optimize SQL queries for data extraction, manipulation, and reporting within complex database environments.
Implement and maintain REST APIs to enable efficient data integration between disparate systems and applications.
Utilize cloud technologies to deploy and manage data solutions, ensuring security and compliance with industry standards.
Collaborate with cross-functional teams to define data requirements and deliver data engineering solutions that meet business objectives.
Requirements
Requirements:
Experience: 6 to 9 years of professional experience in data engineering or related software development roles.
Core Programming: Proficiency in Python and Scala for developing data-intensive applications.
Big Data Technologies: Strong expertise in PySpark for distributed data processing and analytics.
Data Engineering: Demonstrated ability to design and implement ETL/ELT pipelines and data integration strategies.
Cloud & Infrastructure: Hands-on experience with AWS data services and general cloud technologies.
Database Skills: Advanced knowledge of SQL and relational database management systems.
API Development: Competence in building and consuming REST APIs for data connectivity.
Industry Context: Familiarity with the IT Services sector and its specific data handling challenges.
... primary contact for development queries and concerns. Analyze and resolve development issues accurately. Skills Must have 8+ years of experience in data intensive Pyspark development. Experience as a core Python developer. Basic experience in Gen/Agentic AI Experience developing Classes, OOPS, exception handling, parallel processing ...
... maintain the reliability of the AEP https://jobeax.com/link/hs6VQjYW4dlN5SYR Skillset :- Demonstrated expertise in building high-performance data pipelines using Python and PySpark within AWS cloud environments.- Proven ability to manage and query large-scale datasets using both SQL and NoSQL databases, with a focus on performance ...
Key Responsibilities :- Design, develop, and maintain scalable data pipelines using Azure Databricks and PySpark.- Build and optimize large-scale data processing solutions using Python and Spark.- Develop robust ETL/ELT pipelines for batch and near-real-time data processing.- Work with Azure Data Lake Storage (ADLS) and ...
... SQL queries to extract, manipulate, and analyze large-scale datasets from distributed storage environments.- Build and maintain robust data pipelines using Python and Spark to ensure high-quality data availability for downstream engineering and research teams.- Collaborate with autonomous vehicle domain experts to identify ...
... Key Vault .Strong Experience in Azure Data Bricks, Notebooks, ADLS, Delta Lake Responsibilities Experience in building data pipelines and analysis tools using Python, PySpark, Scala, SQL .Proficient in integrating, transforming, and consolidating data from various structured and unstructured data. .Knowledge on Autosys jobs, ...
... 10 - 13 years of experience in Data & Analytics, with 5+ years of hands-on experience in Azure data engineering technologies.- Strong hands-on expertise in Python, SQL, and PySpark.- Strong experience with Databricks Unity Catalog for data governance and security.- Strong sense of ownership, accountability, and execution ...
Key Responsibilities :- Design, develop, and maintain scalable ETL pipelines using PySpark.- Build robust data ingestion frameworks for processing data from multiple source systems.- Develop efficient batch and near real-time data processing workflows.- Perform data extraction, transformation, cleansing, validation, and ...
... Framework Architecture & Technical Leadership . Architect & Enhance Frameworks: Design, build, and maintain scalable, reusable ETL automation test frameworks using Python, PySpark, Pandas, and Robot Framework. . Data Orchestration: Develop PySpark based orchestration mechanisms to execute SQL queries and perform high performance ...
... Presto, Tez, Hive, Spark etc.Hands-on experience in work with bigdata platforms and related programming languages (https://jobeax.com/link/UVPpUiV6Uuv02TIK, Python, PySpark, GIT, Jenkins).Behavioral Competencies : Business Acumens Organization leadership skills Coaching and mentoring Excellent analytical skillsDemonstrate ...