Senior Big Data Engineer - Apache Spark in India, भारत

Role & Responsibilities :- Design, develop, and maintain scalable and high-performance big data processing pipelines.- Develop and optimize data pipelines using technologies such as Apache Spark, PySpark, Kafka, Hadoop, Hive, and Airflow.- Build robust ETL/ELT workflows to process large volumes of structured and unstructured data.- Work with cloud-based data platforms such as AWS, Azure, or GCP and implement cloud-native data solutions.- Design and implement data lakes, data warehouses, and distributed data processing architectures.- Optimize Spark jobs, SQL queries, data pipelines, and storage for performance, scalability, and cost efficiency.- Develop real-time and batch data processing solutions using Kafka and Spark Streaming/Structured Streaming.- Ensure data quality, accuracy, availability, security, and compliance across data pipelines.- Troubleshoot production issues, perform root-cause analysis, and implement permanent fixes.- Collaborate with Data Scientists, Data Analysts, Software Engineers, Architects, and Business teams to understand data requirements and deliver scalable solutions.- Participate in system design, architecture discussions, code reviews, and technical documentation.- Establish and follow engineering best practices for version control, CI/CD, testing, monitoring, and deployment.- Mentor junior engineers and provide technical guidance to the team.- Stay current with emerging big data, cloud, data engineering, and distributed computing https://jobeax.com/link/T08X3WP3qGUc4E8N Candidate Profile :- 5+ years of experience in Big Data Engineering, Data Engineering, or a closely related role.- Strong hands-on experience with Apache Spark / PySpark and distributed data processing.- Strong programming skills in Python, Scala, or Java.- Strong understanding of data lake, data warehouse, dimensional modeling, and distributed system concepts.- Hands-on experience with at least one major cloud platform: AWS, Microsoft Azure, or Google Cloud Platform.- Experience with modern data platforms such as Databricks, Snowflake, Delta Lake, or similar technologies is preferred.- Knowledge of Docker, Kubernetes, Git, CI/CD, and DevOps practices is an advantage.- Strong understanding of data quality, data governance, security, monitoring, and performance optimization.- Ability to analyze complex technical problems and develop scalable, reliable, and maintainable solutions.- Good communication and collaboration skills, with the ability to work effectively across technical and business teams.- Experience mentoring engineers or taking ownership of technical design and delivery is preferred.- Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field is preferred. (ref:hirist.tech)