Job Title: Senior Data Engineer
Job type: Permanent
Job Location: Dublin/Hybrid
Company Overview:
A global data and artificial intelligence ("AI") company that offers services and solutions to reinvent client business models, drive better outcomes, and unlock growth with speed. The company harnesses the power of data, AI, and deep industry knowledge to transform businesses, including the world's leading corporations in industries such as insurance, healthcare, banking and financial services, media, and retail, among others. Founded in 1999, the company operates with the core values of innovation, collaboration, excellence, integrity, and respect.
The Role:
Our client is seeking a hands-on Databricks Data Engineer to support a high-impact data transformation initiative focused on automating reporting and dashboard processes.
You will work with structured and unstructured data sources, building and maintaining scalable data ingestion and transformation pipelines within a Databricks medallion architecture.
This role is delivery-focused, working closely with Data Architects and senior stakeholders to implement data solutions, develop pipelines across Bronze / Silver / Gold layers, and ensure data quality, performance, and reliability within a modern cloud-based data platform.
Key Responsibilities:
- Data Pipeline Development:
Design, build, and maintain data ingestion and transformation pipelines using Databricks, working across structured and unstructured data sources.
- Medallion Architecture Implementation:
Develop and support data pipelines across bronze, silver and gold layers, ensuring data is cleansed, transformed, and optimised for downstream reporting and analytics. - Data Integration & Processing:
Implement batch and streaming data pipelines using tools such as Databricks, Spark, and Azure Data Factory (ADF) to enable efficient data movement and processing. - Collaboration with Architects & Stakeholders:
Work closely with Data Architects and business stakeholders to understand requirements and translate them into reliable, maintainable data engineering solutions. - Data Quality & Validation:
Support the implementation of data quality checks, validation rules, and monitoring processes to ensure accuracy and reliability of data pipelines. - Performance Optimisation:
Assist in optimising data pipeline performance, scalability, and cost efficiency within Databricks and cloud environments. - Data Governance & Standards:
Follow established best practices for data governance, lineage, and metadata management, ensuring compliance with organisational standards. - Testing, Deployment & Release Support:
Support testing, deployment, and release processes, ensuring solutions are robust and production ready. - Documentation:
Create and maintain technical documentation, including pipeline logic, data flows, and transformation rules. - Continuous Improvement:
Stay up to date with Databricks features and modern data engineering practices, contributing to ongoing improvements in tooling, frameworks, and ways of working.
Experience Required:
- Education & Experience:
Bachelor's degree in IT, Computer Science, or a related field, with 3-6 years' experience in data engineering, data warehousing, or related roles. - Databricks & Data Engineering Skills:
Hands-on experience with Databricks and Spark-based data processing, including building and maintaining data pipelines. - Data Integration & Processing:
Experience working with ETL/ELT pipelines using tools such as Databricks, Azure Data Factory (ADF), or similar frameworks. - Data Platform & Database Knowledge:
Experience working with enterprise data stores such as SQL Server, PostgreSQL, Oracle, Synapse, or similar. - Cloud Experience:
Exposure to cloud-based data platforms (Azure preferred) and distributed data processing environments. - Business Intelligence Awareness:
Basic understanding of reporting tools such as Power BI or Tableau, and how data pipelines support analytics and dashboards. - Industry Experience (Desirable):
Experience within Financial Services or regulated environments is beneficial.
If you are interested in this role or would like to discuss it further, please call Nidhi at +353 1 645 5244 or email [email protected].
