Intraedge Technologies Ltd.
Senior Data Engineer - ETL Pipelines
Pune, भारत
Vacancy description
TekPioneers - A TekGence Company
Hyderabad, भारत
This is a hybrid role based in Hyderabad, India.
Role Summary
Seeking a hands-on senior Python engineer to own data pipelines end to end (ingestion, transformation, orchestration, quality, and delivery) at enterprise scale. Pipelines feed analytics, search, and GenAI/Agentic AI platforms.
Responsibilities
- Design, build, and own production data pipelines in Python (batch and near-real-time).
- Operate orchestrated workflows (AWS Step Functions, Airflow, Dagster, or Prefect) with retries, idempotency, and backfills.
- Model data at scale: partitioning, Parquet/Avro/Iceberg, schema evolution, CDC.
- Establish data quality and observability (Great Expectations/Pandera, SLAs, lineage, alerting).
- Build document-processing and RAG retrieval pipelines (chunking, embeddings, vector and hybrid search).
- Build serving APIs with FastAPI, Flask, or Django REST.
- Set Python engineering standards; mentor junior engineers; liaise with business stakeholders.
Qualifications
- 8–12 years of software engineering, 5–6+ years building production data pipelines in Python.
- Expert modern Python (3.10+), typing, asyncio/concurrency, pandas/Polars, PyArrow, PySpark/Dask/Ray.
- Strong SQL and data modeling; Snowflake, Redshift, Iceberg/Delta, dbt.
- Strong AWS: Lambda, Step Functions, S3, Glue, EMR, IAM, CloudWatch; IaC (CDK/Terraform).
- pytest, packaging (uv/Poetry), CI/CD.
- Strongly preferred: RAG, AWS Bedrock, LangChain/LlamaIndex, agentic frameworks, MCP servers, GenAI evaluation.
- Nice to have: Kafka/Kinesis/Flink streaming, C#/.NET, regulated-industry experience.
Similar vacancies
Intraedge Technologies Ltd.
Pune, भारत
Calcutta, भारत
Bangalore, भारत
Bangalore, भारत
Valam Solutions
Chennai, भारत
Bangalore, भारत
GANIT BUSINESS SOLUTIONS PRIVATE LIMITED
Gurgaon, भारत
BNP Paribas India Solutions Pvt. Ltd.
Chennai, भारत
$80 USD
Hyderabad, भारत
BNP Paribas India Solutions Pvt. Ltd.
Bombay, भारत