Data Engineer

OverviewDeepLightAI is a specialist AI and data consultancy with extensive experience implementing intelligent enterprise systems across multiple industries, with particular depth in financial services and banking. Our team combines deepexpertise in data science, statistical modeling, AI/ML technologies, workflow automation, and systems integration with a practical understanding of complex business operations.ResponsibilitiesDesign, build, and optimise scalable data solutions, primarily using the Lakehouse architecture to unify data warehousing and data lake capabilities.Advise stakeholders on strategic choice between Data Warehouse, Data Lake, and Lakehouse architectures based on specific business needs, cost, and latency requirements.Design, develop, and maintain scalable and reliable data pipelines to ingest, transform, and load diverse datasets from various sources, including structured and unstructured data, streaming data, and real‑time feeds.Implement standards and tooling to ensure ACID properties, schema evolution, and high data quality within the Lakehouse environment.Implement robust data governance frameworks (security, privacy, integrity, compliance, auditing).Continuously optimise data storage, compute resources, and query performance across the data platform to reduce costs and improve latency for both BI and ML workloads, leveraging techniques such as indexing, partitioning, and parallel processing.Develop and maintain CI/CD pipelines to automate the entire machine learning lifecycle, from data validation and model training to deployment and infrastructure provisioning.Deploy, manage, and scale machine learning models into production environments, utilizing MLOps principles for reliable and repeatable operations.Establish and manage monitoring systems to track model performance metrics, detect data drift and model decay.Ensure rigorous version control and tracking for all components: code, datasets, and trained model artifacts (using tools like MLflow or similar).Create comprehensive documentation, including technical specifications, data flow diagrams, and operational procedures, to facilitate understanding, collaboration, and knowledge sharing.QualificationsProven practical experience in designing, building, and optimising solutions using Data Lakehouse architectures (e.g., Databricks, Delta Lake).Strong hands‑on experience with managing data ingestion, schema enforcement, ACID properties, and utilizing big data technologies/frameworks like Spark and Kafka.Expertise in data modeling, ETL/ELT processes, and data warehousing concepts.Proficiency in SQL and scripting languages (e.g., Python, Scala).Demonstrated practical experience implementing MLOps pipelines for production systems, including automation, governance, and monitoring of ML models throughout the entire lifecycle.Experience with CI/CD tools, containerization/orchestration technologies (e.g., Docker, Kubernetes), model serving frameworks (e.g., TensorFlow Serving, SageMaker), and experiment tracking (e.g., MLflow).Experience with production monitoring tools to detect data drift or model decay.Strong hands‑on experience with major cloud platforms (e.g., AWS, Azure, GCP) and familiarity with DevOps practices.Excellent analytical, problem‑solving, and communication skills, with the ability to translate complex technical concepts into clear and actionable insights.Proven ability to work effectively in a fast‑paced, collaborative environment, with a passion for innovation and continuous learning.Benefits & Growth OpportunitiesCompetitive salary and performance bonusesComprehensive health insuranceProfessional development and certification supportOpportunity to work on cutting‑edge AI projectsFlexible working arrangementsCareer advancement opportunities in a rapidly growing AI companyDiversity & InclusionAt DeepLightAI, we recognise that diversity drives innovation. We are committed to fostering an inclusive environment where individuals with different thinking styles can thrive and contribute their unique strengths to our specialised AI and data solutions. Our goal is to ensure our application and interview process is accessible, predictable, and fair for all candidates. If you require any specific adjustments to the application process, or if you require any reasonable adjustments should you be successful in being processed to the interview stage, please do let us know. This information will be kept strictly confidential and will not impact hiring decisions.Additional InformationThe successful candidate will play a crucial role in driving our company's success in delivering transformative AI solutions to our clients. As an AI consultancy, our greatest asset is the expertise of our people. While technical mastery is the foundation of what we do, the ability to bridge the gap between complex data science and actionable business value is what defines your success with DeepLightAI. We’re looking for individuals who are not only world‑clas