Loading description…
Vacancy description
ScaleupAlly
Noida, भारत
We are looking for a Senior Data Engineer who is equally strong on the data side (modelling, data quality, SQL, understanding the business) and the engineering side (Python, PySpark, CI/CD, performance, reliability). You will own the design and delivery of production data platforms from requirements to production support, work directly with clients, and share your knowledge with the wider engineering team.
Your stack will include Python, SQL, PySpark, Azure Databricks or Microsoft Fabric, Azure Data Factory, Delta Lake, and Lakehouse and Data Warehouse designs. Essential Qualifications
Bachelor's degree in any discipline
Information Systems, or a related field.
Holds at least 5-8 years in data engineering
Skills ADF l Microsoft Fabric l SQL l ETL/ELT l Power BI | Python
Duties
Own the design and delivery of data pipelines and platforms end to end: requirements, architecture, build, testing, deployment and production support.
Design data models (dimensional, SCD, Medallion Bronze/Silver/Gold) that match how the business actually uses the data.
Build and maintain scalable ETL/ELT pipelines using Python, SQL, PySpark and Azure Data Factory on Azure Databricks or Microsoft Fabric.
Implement batch, incremental/CDC and streaming loads, and tune Spark and SQL performance and cost.
Build data quality checks, monitoring and alerting into every pipeline, and apply governance, lineage and access controls.
Automate testing and deployment with Git and CI/CD, and follow engineering best practices in code reviews.
Work directly with client stakeholders to gather requirements, explain design choices and agree acceptance criteria.
Integrate data from APIs, transactional systems and external platforms.
Investigate and resolve complex production issues and document root causes.
Document your solutions and share knowledge with the team, including peer code reviews.
Required Qualifications
Data side: strong SQL, dimensional and Medallion/Lakehouse modelling, data quality practices, and the ability to understand business requirements.
Engineering side: strong Python and PySpark, Git and CI/CD, automated testing, and performance tuning of high-volume workloads.
Hands-on experience with Azure Databricks or Microsoft Fabric, and Azure Data Factory or an equivalent orchestration tool.
Proven delivery of batch and incremental/CDC pipelines on Delta Lake or similar.
Client-facing experience: requirement gathering, stakeholder communication and clear technical documentation.
Measurable impact you can explain, such as runtime, cost or data quality improvements.
Good to Have
Streaming experience (Spark Structured Streaming, Kafka).
Governance and lineage tools (Unity Catalog, Azure Purview).
Infrastructure as code (Terraform).
Power BI and semantic-layer knowledge, to work effectively with BI teams.
Microsoft certifications such as DP-203 or DP-600.
Benefits and Additional Information
Working hours: 10:00 AM – 7:00 PM
Working days: 5 days a week (plus 1st & 3rd Saturdays working)
Medical Insurance coverage for employees
Provident Fund (PF) facility
Quarterly parties and yearly outings/trips for team bonding
Regular check-ins with leadership for growth and feedback
Recognition awards to celebrate high performance
Fun activities and team engagement sessions throughout the year