Job Summary
We are seeking an experienced Senior Data Engineer to lead the design, development, and optimization of scalable, secure, and high-performance data engineering solutions.
The ideal candidate should have strong expertise in Python, SQL, Databricks, PySpark, Delta Lake, CDC, and incremental data processing. The candidate will own data architecture decisions, design reusable data ingestion frameworks, implement Medallion Architecture, and develop reliable integration and orchestration solutions across AWS and Microsoft Azure.
The role also involves building API integration services using TypeScript and https://jobeax.com/link/JSd7UYCSg572jJDY, implementing serverless workflows using Azure Functions and Durable Functions, optimizing production workloads, establishing engineering best practices, and mentoring team members.
Key ResponsibilitiesData Engineering and Architecture
- Design and lead the development of scalable data architectures and end-to-end data pipelines.
- Define data engineering standards, architectural patterns, reusable frameworks, and best practices.
- Build and optimize complex ETL/ELT workflows supporting large-scale data processing.
- Design data ingestion frameworks for databases, REST APIs, files, event streams, and third-party platforms.
- Develop reusable connectors supporting authentication, pagination, rate limits, retries, error handling, and incremental synchronization.
- Design and maintain data warehouses, data lakes, lakehouses, and analytical data models.
- Optimize complex SQL queries, database schemas, partitioning strategies, and processing performance.
Databricks, PySpark, and Delta Lake
- Design and optimize data processing solutions using Databricks, PySpark, and Delta Lake.
- Establish and maintain Medallion Architecture across Bronze, Silver, and Gold layers.
- Define data ingestion, cleansing, standardization, enrichment, and aggregation strategies.
- Design Delta Lake tables and implement schema enforcement, schema evolution, and transactional data processing.
- Optimize Spark jobs, joins, partitioning, file sizes, and execution plans.
- Apply Delta Lake table optimization and maintenance strategies where appropriate.
- Develop curated datasets for business intelligence, analytics, and downstream applications.
CDC and Incremental Data Processing
- Design and implement scalable Change Data Capture (CDC) and incremental loading frameworks.
- Handle inserts, updates, deletes, duplicate records, late-arriving data, and reprocessing scenarios.
- Implement watermarking, checkpointing, and recovery mechanisms.
- Ensure idempotent processing and reliable data consistency during retries and reprocessing.
- Establish source-to-target reconciliation and automated data quality validation.
- Design historical backfill, replay, and recovery strategies.
- Monitor data freshness, pipeline failures, and data quality issues.
API Integration and Serverless Development
- Design reusable API integration frameworks using TypeScript, https://jobeax.com/link/JSd7UYCSg572jJDY, and Python where appropriate.
- Develop secure and reliable connectors for third-party platforms and external data sources.
- Implement authentication, pagination, rate-limit management, retries, token refresh, and timeout handling.
- Design scalable serverless processing solutions using Azure Functions.
- Design orchestrated, long-running workflows using Azure Durable Functions.
- Implement workflow orchestration, retry policies, failure recovery, and parallel processing patterns where appropriate.
- Integrate serverless workflows with data pipelines, cloud storage, and downstream processing systems.
- Establish standards for API security, observability, performance, and maintainability.
AWS and Azure Cloud Engineering
- Design, implement, and optimize data engineering solutions across AWS and Microsoft Azure.
- Select appropriate cloud services based on scalability, reliability, security, performance, and cost.
- Work with AWS services such as Amazon S3, AWS Glue, AWS Lambda, and Amazon CloudWatch, as applicable.
- Work with Azure services such as Azure Data Lake Storage Gen2, Azure Data Factory, Azure Functions, and Azure Monitor, as applicable.
- Implement secure access controls, identity management, encryption, and secrets management.
- Establish cloud monitoring, alerting, deployment, and operational standards.
- Optimize cloud resource utilization, storage costs, and data processing performance.
Data Quality, Security, and Reliability
- Establish data quality standards, validation rules, automated tests, and reconciliation frameworks.
- Implement logging, metrics, monitoring, alerting, and pipeline observability.
- Ensure data accuracy, completeness, freshness, and availability.
- Follow security, privacy, access control, and data governance practices.
- Troubleshoot critical production issues and perform root-cause analysis.
- Improve pipeline resilience, recovery, and operational reliability.
DevOps and Engineering Best Practices
- Drive CI/CD, automated testing, deployment automation, and release management.
- Establish Git workflows, code review standards, and maintainable coding practices.
- Support infrastructure automation and environment configuration management.
- Maintain architecture diagrams, technical documentation, data mappings, and operational runbooks.
- Evaluate technologies and recommend solutions based on scalability, reliability, performance, and cost.
Technical Leadership and Collaboration
- Lead technical design reviews and architectural decisions.
- Mentor junior and mid-level engineers through code reviews and knowledge sharing.
- Collaborate with product managers, architects, analysts, backend engineers, and business stakeholders.
- Translate business requirements into scalable technical data solutions.
- Identify performance bottlenecks, technical risks, and automation opportunities.
- Own end-to-end technical delivery and continuous improvement of data engineering solutions.
Required Skills
- 6–10+ years of experience in data engineering, data platforms, or related engineering roles.
- Advanced proficiency in Python, SQL, and PySpark.
- Strong hands-on experience with Databricks and distributed data processing.
- Expertise in Medallion Architecture and lakehouse design.
- Strong experience with Delta Lake, schema evolution, transactional data processing, and table optimization.
- Proven experience designing CDC and incremental loading frameworks.
- Strong understanding of ETL/ELT architecture, data modeling, and source-to-target reconciliation.
- Experience developing API integration services using TypeScript and Node.js.
- Hands-on experience with Azure Functions and Azure Durable Functions.
- Strong expertise in at least one of AWS or Microsoft Azure, with practical experience in the other.
- Experience with cloud security, monitoring, logging, and cost optimization.
- Experience with Git, CI/CD pipelines, automated testing, and production deployment practices.
- Strong troubleshooting, problem-solving, communication, and technical decision-making skills.
- Experience mentoring engineers and leading technical initiatives.
Preferred Skills
- Experience operating data platforms across both AWS and Azure.
- Experience with AWS Glue, Amazon S3, AWS Lambda, and CloudWatch.
- Experience with Azure Data Factory, ADLS Gen2, and Azure Monitor.
- Familiarity with Databricks Workflows and Unity Catalog.
- Experience with structured streaming and event-driven data processing where applicable.
- Knowledge of Terraform or other infrastructure automation tools.
- Experience with Docker and containerized services.
- Familiarity with data governance, lineage, metadata management, and data observability.
- Experience designing multi-tenant data platforms and reusable integration frameworks.
- Experience supporting analytics, reporting, and AI/ML data workloads.
Qualifications
Bachelor’s or Master’s degree in Computer Science, Information Technology, Engineering, or a related field, or equivalent practical experience.
Success Criteria
- Scalable, secure, and highly reliable data platform architecture.
- Improved pipeline performance, data quality, and system availability.
- Reduced production incidents and faster issue resolution.
- Optimized cloud infrastructure and data processing costs.
- Reusable data ingestion frameworks and consistent engineering standards.
- High-quality documentation, automated testing, and operational practices.
- Successful delivery of complex data engineering initiatives.
- Effective technical mentorship and cross-functional collaboration.