Big Data Platform Administrator - Apache Spark & Airflow… - Jobeax
Vacancy description
Big Data Platform Administrator - Apache Spark & Airflow in Chennai, भारत
Samridhi Placements
Chennai, भारत
Job Description :Key Responsibilities :- Administer and support Hadoop ecosystem components (HDFS, YARN) across Production, Test, and Development environments- Manage and maintain Apache Spark and Airflow infrastructure- Monitor cluster health, troubleshoot performance issues, and optimize system performance- Implement and maintain High Availability (HA), Disaster Recovery (DR), backup, and restore strategies- Manage logging, monitoring, and alerting systems using Prometheus, Grafana, and ELK Stack- Perform Linux (Ubuntu) system administration, including patching, upgrades, and security hardening- Develop and maintain automation scripts using Shell and/or Python- Handle Kerberos authentication setup and maintenance- Support incident management, root cause analysis (RCA), and ensure SLA adherence- Manage capacity planning and system scalability- Follow CI/CD and DevOps best practices for platform improvements and deploymentsRequired Skills :- Strong experience with Hadoop ecosystem (HDFS, YARN)- Handson expertise in Apache Spark and Apache Airflow- Proficiency in Linux system administration (Ubuntu), including security hardening- Scripting skills in Shell and Python for automation- Familiarity with monitoring and observability tools: Prometheus, Grafana, ELK Stack- Knowledge of High Availability, Disaster Recovery, backup, and restore concepts- Experience with Kerberos authentication setup and maintenance (ref:hirist.tech)
Professional Summary : Experienced Hadoop Administrator with 8+ years of expertise in designing, implementing, administering, monitoring, troubleshooting, and supporting enterprise Big Data https://jobeax.com/link/DMuNz6lbkIBcCnpO hands-on experience with Cloudera CDP, Cloudera Manager, HDFS, YARN, Hive, Spark, Impala, ...
... high-performance big data processing pipelines.- Develop and optimize data pipelines using technologies such as Apache Spark, PySpark, Kafka, Hadoop, Hive, and Airflow.- Build robust ETL/ELT workflows to process large volumes of structured and unstructured data.- Work with cloud-based data platforms such as AWS, Azure, or ...
... https://jobeax.com/link/wJIgo45kO3tINlTo Skills & Experience :- 6 - 11 years of experience in Big Data, Data Engineering, or a related technology domain.- Strong hands-on expertise in Apache Spark and Scala.- Strong understanding of Big Data concepts, distributed computing, and parallel data processing.- Experience developing large-scale data processing, ...
... Hadoop ecosystem technologies for distributed data processing and storage.- Integrate data from multiple sources including databases, APIs, files, and streaming platforms.- Build and manage data orchestration workflows using Azure Data Factory (ADF).- Optimize Spark jobs, Databricks clusters, and data pipelines for performance ...
WEKA Data Platform Administrator About the job WEKA is architecting a new approach to the enterprise data stack built for the age of reasoning. NeuralMesh by WEKA sets the standard for agentic AI data infrastructure with a cloud and AI-native software solution that can be deployed anywhere. It transforms legacy data silos ...
... build, and maintain scalable data pipelines and architectures for large-scale data environments. - Translate business and technical requirements into reliable data engineering solutions. - Develop data integration, transformation, and processing workflows using Python and appropriate big data technologies. - Build, manage, ...
... https://jobeax.com/link/wJIgo45kO3tINlTo Skills & Experience :- 5+ years of experience with AWS Services including RDS, AWS Lambda, AWS Glue, EMR, S3, and related big data technologies.- 5+ years of experience working with Apache Spark, Spark Streaming, Kafka, and Hive.- Strong experience in SQL and NoSQL databases (MySQL, ...
... scalable, reliable, high-performance data pipelines and data platforms.EDUCATION:Bachelor's degree in computer science, Software Engineering, MIS or equivalent combination of education and https://jobeax.com/link/NfKu17xn3PQ6gnFL SKILLS:Python, PySpark, Hadoop, Spark, Hive, Data Engineering, Big Data, SQL (ref:hirist.tech) []
... traditional relational databases and/or distributed systems (Snowflake or Redshift), required.- 1+ years of experience programming languages (e.g. Python, Pyspark), preferred.- 1+ years of experience with data orchestration/ETL tools (Airflow, Nifi), preferred.- Experience with Snowflake, Databricks/EMR/Spark, and/or https://jobeax.com/link/2xnbqOXKXQKBt9BL ...
... containerized environments.- Troubleshoot production issues and continuously improve platform https://jobeax.com/link/AMuFfOH9R4FgRv8m : - 5+ years of experience in Data Engineering or Big Data Engineering.- Strong hands-on experience with Apache Spark and Scala.- Experience building and maintaining ETL pipelines.- Familiarity ...