Senior Data Engineer

Role Summary

EFutures is seeking a seasoned professional to join their team to drive significant data and Master Data Management (MDM) projects. The successful candidate will focus on constructing scalable data pipelines and robust solutions designed for large-scale enterprise environments.

Key Responsibilities

• Engineer ingestion pipelines utilizing REST APIs, file-based sources, and CDC/database integrations.
• Develop data transformations using PySpark and Delta Lake within the Azure Databricks ecosystem.
• Apply architectural patterns including Bronze, standardized, and governed data layers.
• Build reusable frameworks for data validation, standardization, and quality assurance prior to MDM processes.
• Manage incremental loading, schema evolution, and system recovery protocols.
• Execute source-to-canonical transformations while ensuring data lineage is maintained.
• Fine-tune Spark jobs to ensure optimal performance, cost-efficiency, and reliability.
• Conduct automated testing and participate in collaborative peer reviews.
• Coordinate Databricks workflows with Profisee and other Azure-based services.
• Assist with production monitoring, troubleshooting, and the development of operational runbooks.

Requirements & Qualifications

• Essential: Over 7 years of professional experience, with at least 5 years dedicated to data engineering and production-level PySpark/Spark development.
• Technical Expertise: Proficiency in Azure Databricks, Delta Lake, SQL, and ADLS Gen2.
• Integration Skills: Proven track record with REST/API ingestion and database integration.
• Data Management: Experience with incremental processing, CDC, and schema evolution.
• Best Practices: Strong background in data quality, testing, Git, and CI/CD workflows.
• Preferred: Databricks certification, experience with MDM or entity resolution, Azure Data Factory, PySpark MLlib (fuzzy matching), and governance tools like Purview or Unity Catalog.

Employment Details

This is a 2-year contract position offered on a part-time basis. The role follows a hybrid work model, requiring 3 to 4 days onsite in Colombo. Standard working hours are Monday to Friday, 9:00 a.m. to 6:00 p.m.

How to Apply

Please contact the employer directly to submit your application for this position.