Vacancy description
Luminite Services
India, भारत
Data EngineerExperience: 6+ years in data engineering, ETL/ELT, analytics engineering, data warehouse development, or big data pipelines; minimum 4 years on GCP cloud data platforms https://jobeax.com/link/bPH40YQHlBdyx8oe we're looking for:- Strong SQL and Python.- Data modelling, warehousing concepts, partitions, clustering, indexing, optimization.- ETL/ELT pipeline design and production support.- Batch and streaming pipeline fundamentals.- Orchestration using Airflow, Composer, Control-M, ADF, dbt, or equivalent.- Big data processing using Spark, PySpark, Beam, Dataflow, Dataproc, Databricks, EMR, or equivalent.- Cloud storage and data lake concepts.- Data quality, reconciliation, schema validation, lineage, monitoring, and alerting.- CI/CD for data pipelines and code versioning.- Ability to explain scale scenarios, not just tool https://jobeax.com/link/w20nDoSA2f7Mp1HD Skills:- Python and advanced SQL.- Data quality checks, reconciliation, alerting, retries, and backfills.- Production pipeline monitoring and https://jobeax.com/link/3mvkfheMPtV9f85w Skills:- Mandatory: BigQuery: datasets, tables, views, partitioning, clustering, SQL optimisation, cost awareness.- Option 1: Dataflow / Apache Beam or equivalent streaming/batch framework.- Option 2: Dataproc / PySpark or strong Spark experience.- Pub/Sub or equivalent event streaming.- Cloud Composer / Airflow DAG https://jobeax.com/link/iDCqPrhlKH5MkOB3 to have:- Dataform, dbt, CI/CD for data.- BigQuery ML.- Dataflow templates.- Looker / LookML basics.- Terraform for data infrastructure. (ref:hirist.tech)