We are seeking a highly skilled and experienced Python/PySpark Lead to drive the design, development, and delivery of large-scale data engineering solutions. The ideal candidate should possess strong technical expertise in Python and PySpark, coupled with proven leadership capabilities to manage project deliverables, mentor development teams, and engage effectively with business and technology stakeholders
Responsibilities
Lead the end-to-end delivery of Python/PySpark-based solutions, ensuring adherence to project timelines and quality standards.
Provide technical leadership and guidance to development teams across design, development, testing, and deployment phases.
Collaborate with business stakeholders, project managers, and technical teams to understand requirements and translate them into scalable solutions.
Drive stakeholder communication by providing regular project updates, highlighting risks, dependencies, and mitigation plans.
Perform hands-on development and code reviews to ensure adherence to coding standards, best practices, and performance optimization techniques.
Foster collaboration, accountability, and a high-performance culture within the team.
Support estimation, planning, resource coordination, and delivery governance activities.
Ensure production stability, resiliency, and continuous improvement of data processing frameworks.
Qualifications
10+ years of overall IT experience with strong expertise in Python and PySpark.
Extensive experience in designing and developing scalable data processing and data transformation solutions.
Strong knowledge of Spark architecture, distributed computing, performance tuning, and optimization.
Experience working with Hadoop ecosystem components, Hive, SQL, and large-scale data platforms.
Strong understanding of software engineering principles, design patterns, and Agile delivery methodologies.
Excellent problem-solving, debugging, and analytical skills.
Proven experience in leading technical teams and driving project execution.
Excellent verbal and written communication skills with demonstrated stakeholder management experience.
Preferred Qualifications
Experience in Banking and Financial Markets domain is an added advantage.
Exposure to cloud-based data platforms such as Databricks, AWS, or Azure is desirable.
Experience in mentoring and developing high-performing engineering teams.
... with business objectives, ensuring solutions are both technically sound and strategically relevant. - Author, review, and approve technical requirements and architectural designs, and lead process re-engineering efforts to deliver cost-effective, high-impact business solution - Drives team adoption of enterprise-authorized ...
Technical Lead / Data Architect Azure | 8+ Years | BangaloreHiring an experienced Technical Lead / Data Architect to drive end-to-end technical architecture and delivery of cloud and data engineering projects. This is an excellent opportunity for professionals with strong expertise in Azure, Azure Databricks, Python, Data ...
... Architecture.- Strong hands-on experience in Azure Databricks, Python, PySpark, and SQL.- Proven experience architecting and developing reusable, metadata-driven Python frameworks.- Strong expertise in enterprise data modeling and complex data pipeline design.- Experience leading Data Engineering teams while remaining actively ...
ROLE SUMMARY: We are looking for an AI Solution Architect who can take an enterprise AI initiative from vision to execution. You will shape AI strategy with C-level stakeholders, architect cloud-native GenAI and ML platforms on Azure and Databricks, and lead cross-functional teams across data science, MLOps and product ...
... Engineering and Architecture. You must have led at least 23 large-scale migrations from legacy to modern cloud platforms.- Technical Depth : - Expert-level Python/PySpark and SQL.- Deep expertise in Snowflake or Databricks (Certifications like SnowPro or Databricks Architect are highly preferred).- Familiarity with "Traditional" ...
... finance users to define benchmark queries and validate numbers during UAT and parallel runs. Report progress risks and decisions to stakeholders and delivery leadership. Required skills Area Databricks Unity Catalog Fine-grained security Delta Lake Performance PySpark and SQL Python engineering Azure DevOps Data modelling ...
... platforms. Required Skills & Experience - 10-15 years of experience in data engineering data architecture data warehousing and analytics. - Strong hands-on architecture expertise in Snowflake and/or Databricks . - Excellent SQL skills and experience with Python PySpark or equivalent data processing technologies. - Strong ...
... layers. - Design and optimize large-scale distributed data processing pipelines using Hadoop, Apache Spark, and PySpark. - Build reusable, production-grade Python frameworks and pipelines for batch and streaming data engineering workloads. - Define and implement cloud data architecture on AWS and/or Azure, covering storage, ...
... adoption. Job Responsibilities - Lead and manage a high-performing team of data engineers across multiple projects and geographies. - Define and drive the architecture of scalable secure and cost-effectivecloud-based data warehouse solutions. - Oversee the development of robust ETL/ELT pipelines usingPySparkPython andSQL. ...
... stores, containerization, orchestration, and model serving frameworks.- Partner with data engineering, application engineering, DevOps, security, and cloud architecture teams to ensure seamless integration with enterprise platforms.- Lead technical reviews, architecture discussions, code reviews, and root-cause analysis ...