Job description
Senior Data Engineer — Databricks & PySpark
Location: Fully remote. Open to candidates based anywhere in Europe, and in countries such as Georgia, Armenia and Moldova, provided there is meaningful overlap with Central European working hours.
Language: Professional working English (B2 / Intermediate+ or above) - the role is client-facing.
Experience: 5+ years hands-on software development, with substantial recent Databricks delivery.
The opportunity
We are working with an international technology delivery partner building enterprise applications and AI-driven tooling for a large professional services organisation. The platform supports the delivery of tax and advisory services at scale, and the team combines software engineering, domain expertise, change management and project delivery.
The work is genuinely modern: a large-scale migration of legacy SQL logic into Spark, alongside greenfield build on a current Azure and AI stack. This is a hands-on senior engineering role with real ownership of standards and quality, not a maintenance seat.
Tech environment
Azure Cloud · Databricks · PySpark · Delta Lake · Python (Polars, Pandas) · Azure SQL · MongoDB · Microservices architecture · .NET 8 / ASP.NET Core services · Angular 18 · GitHub Enterprise with Copilot · LangGraph, LangChain, RAG pipelines and multi-modal LLMs.
What you will be doing
Freelance / contract | Fully remote | Europe, Caucasus and neighbouring markets
Role type: Freelance contract (B2B), full-time engagementLocation: Fully remote. Open to candidates based anywhere in Europe, and in countries such as Georgia, Armenia and Moldova, provided there is meaningful overlap with Central European working hours.
Language: Professional working English (B2 / Intermediate+ or above) - the role is client-facing.
Experience: 5+ years hands-on software development, with substantial recent Databricks delivery.
The opportunity
We are working with an international technology delivery partner building enterprise applications and AI-driven tooling for a large professional services organisation. The platform supports the delivery of tax and advisory services at scale, and the team combines software engineering, domain expertise, change management and project delivery.
The work is genuinely modern: a large-scale migration of legacy SQL logic into Spark, alongside greenfield build on a current Azure and AI stack. This is a hands-on senior engineering role with real ownership of standards and quality, not a maintenance seat.
Tech environment
Azure Cloud · Databricks · PySpark · Delta Lake · Python (Polars, Pandas) · Azure SQL · MongoDB · Microservices architecture · .NET 8 / ASP.NET Core services · Angular 18 · GitHub Enterprise with Copilot · LangGraph, LangChain, RAG pipelines and multi-modal LLMs.
What you will be doing
- Designing, building and maintaining robust, scalable Spark applications on Databricks.
- Migrating existing SQL stored procedures into Spark SQL and PySpark, and re-engineering the logic properly rather than lifting and shifting it.
- Optimising Spark workloads for performance and cost in production, including cluster configuration and workflow automation.
- Defining and enforcing coding standards and engineering best practice across the project.
- Running thorough code reviews and holding the quality bar for the wider team.
- Giving clear direction to other engineers and helping keep day-to-day delivery moving.
- Working directly with the client on a regular basis — explaining decisions, trade-offs and progress.
- Diagnosing and resolving technical issues affecting reliability and performance.
- Producing and maintaining clear documentation for code, processes and workflows.
- 5+ years of hands-on software development experience.
- Extensive practical expertise with PySpark on Databricks, covering Delta tables, cluster management and workflow automation.
- Demonstrable experience developing SQL stored procedures and migrating them into Spark SQL or PySpark.
- A proven track record of Spark performance optimisation on production-grade projects — specific, evidenced examples matter more than tool familiarity here.
- Strong Python for complex data manipulation, using libraries such as Polars or Pandas.
- Solid grasp of columnar storage formats, particularly Parquet, with real Delta Table experience.
- Proven capability across data processing, analysis and transformation workflows.
- A working understanding of microservices architecture and how it applies in scalable systems.
- Strong analytical and problem-solving ability, with genuine attention to detail.
- A pragmatic approach — able to balance standardised process against what a project actually needs to ship.
- Azure Cloud services (or another major cloud), including Service Bus, Data Lake, Blob Storage, Redis and similar.
- Familiarity with FastAPI.
- Containerisation and orchestration — Docker and Kubernetes.
