Job Overview
Nextlink is seeking for a Data Engineer with strong expertise in Palantir Foundry, Python, and PySpark to design, develop, and maintain scalable data engineering solutions. The ideal candidate will have hands-on experience with Foundry Ontology, data pipelines, modular applications, and end-to-end data solution implementation, along with a solid understanding of data engineering principles and distributed computing.
Key Responsibilities
1. Palantir Foundry Platform Expertise
- Develop and maintain data pipelines, Foundry Ontology, and modular applications within the Palantir Foundry platform.
- Design and implement end-to-end data solutions, from data ingestion and transformation to data modeling and consumption.
- Integrate data from multiple source systems into Palantir Foundry.
- Build and maintain scalable, reusable data workflows aligned with business requirements.
- Leverage Foundry capabilities to support data-driven applications and analytics.
2. Programming and Scripting – Python & PySpark
- Develop Python and PySpark scripts for data processing, transformation, cleansing, and automation.
- Build reusable Python functions and libraries to extend Palantir Foundry capabilities.
- Develop modular and maintainable code for ETL workflows and data transformation processes.
- Process large datasets efficiently using Python libraries such as Pandas and NumPy, along with PySpark for distributed processing.
- Optimize code for performance, scalability, and reliability.
- Use SQL for data querying, validation, and data modeling, as required.
3. Data Engineering Fundamentals
- Design, develop, and maintain ETL/ELT pipelines for data ingestion and transformation.
- Build efficient data models and optimize data processing workflows.
- Work with structured and unstructured data from multiple sources.
- Implement data quality checks, validation, and error-handling mechanisms.
- Ensure data consistency, reliability, and availability across pipelines.
4. Big Data and Distributed Systems
- Develop scalable data processing solutions using Apache Spark and PySpark.
- Apply distributed computing concepts to handle large-scale data processing requirements.
- Troubleshoot pipeline failures and resolve data processing issues.
- Perform performance tuning and optimization of Spark jobs and data workflows.
- Follow best practices for scalable and efficient data engineering.
Requirements
Essential Skills
- Hands-on experience with Palantir Foundry and its core capabilities.
- Strong programming skills in Python and PySpark .
- Experience developing and maintaining ETL/ELT pipelines.
- Knowledge of Foundry Ontology, data pipelines, and modular applications.
- Strong understanding of data transformation, data modeling, and data integration.
- Experience with Apache Spark and distributed data processing.
- Understanding of performance optimization and scalable data processing techniques.
Desirable Skills
- Familiarity with SQL for querying and data modeling.
- Experience with Pandas and NumPy for data manipulation and analysis.
- Knowledge of data quality, validation, and pipeline monitoring practices.
- Experience integrating data from multiple source systems.
- Understanding of data governance and enterprise data engineering best practices.
Keywords
Palantir Foundry, Palantir Core, PySpark, Python, Apache Spark, Data Engineering, ETL/ELT, Foundry Ontology, Data Pipelines, Modular Applications, SQL, Data Transformation, Data Modeling, Distributed Computing, Performance Optimization.
Primary Skills: Digital – PySpark, Palantir Core
Role Focus: Palantir Foundry development, Python/PySpark-based data engineering, and end-to-end data pipeline implementation.