eClerx - Data Engineer - Google Cloud Platform in Pune, भारत

Role : GCP Data EngineerExperience : Not specifiedLocation : Navi Mumbai / PuneEmployment Type : Full-time, PermanentIndustry : IT Services & ConsultingDepartment : Engineering - Software & QARole Category : Software DevelopmentAbout the Role : We are looking for a skilled GCP Data Engineer to design, develop, and maintain scalable data pipelines and ETL/ELT frameworks on Google Cloud Platform (GCP).The role will involve working with large-scale campaign and clickstream datasets, building reliable data processing workflows, optimizing BigQuery performance and costs, and implementing automation, monitoring, and alerting for critical data https://jobeax.com/link/hEiNRx4W5ZDbjxHN ideal candidate should have strong hands-on expertise in GCP, BigQuery, SQL, Python/PySpark, and Airflow/Composer, along with an understanding of CI/CD and Git-based development https://jobeax.com/link/JcPnJNS8HEL7JdHn Responsibilities : - Design, develop, and implement scalable data pipelines on GCP using services such as BigQuery, Cloud Storage, and Airflow/Composer.- Build robust ETL/ELT pipelines to process and transform large volumes of campaign, clickstream, and other business data.- Develop efficient data ingestion and transformation workflows to support analytics and downstream applications.- Write optimized SQL queries in BigQuery for large-scale data processing and analytics.- Optimize BigQuery query performance and processing costs through effective data modelling, partitioning, clustering, query optimization, and other best practices.- Develop reusable and scalable data engineering frameworks for batch and scheduled data processing.- Automate data workflows and pipeline execution using Airflow/Cloud Composer.- Implement appropriate monitoring, logging, alerting, and failure-handling mechanisms for critical data pipelines.- Identify and troubleshoot pipeline failures, data quality issues, performance bottlenecks, and integration-related problems.- Build and maintain reliable data processing workflows with appropriate data validation and quality checks.- Collaborate with analytics, product, engineering, and business teams to understand data requirements and deliver scalable solutions.- Follow best practices for version control, code quality, testing, deployment, and documentation.- Support CI/CD processes for data engineering workflows using tools such as Jenkins or equivalent platforms.- Work with Git-based development workflows for managing and deploying data engineering code.- Continuously identify opportunities to improve pipeline reliability, scalability, performance, and operational https://jobeax.com/link/GK6JLJWOsFhSBApv Skills & Experience : - Strong hands-on experience in Data Engineering and building scalable data pipelines.- Strong proficiency in SQL, preferably with extensive experience in Google BigQuery.- Strong programming skills in Python and/or PySpark.- Hands-on experience with Google Cloud Platform (GCP).- Strong experience with BigQuery for data storage, transformation, querying, and optimization.- Experience working with Google Cloud Storage (GCS).- Experience developing ETL/ELT frameworks for large-scale datasets.- Hands-on experience with Apache Airflow or Google Cloud Composer for workflow orchestration.- Understanding of data pipeline monitoring, logging, alerting, and troubleshooting.- Familiarity with CI/CD practices and tools such as Jenkins.- Good understanding of Git and version-control workflows.- Strong analytical and problem-solving https://jobeax.com/link/ZitjVqvb84V4AOyQ to Have : - Experience working with campaign, marketing, customer, or clickstream data.- Experience handling large-volume datasets on cloud platforms.- Knowledge of BigQuery partitioning and clustering.- Experience with data quality frameworks and validation processes.- Exposure to other GCP services used in data engineering.- Experience working in Agile development environments (ref:hirist.tech)