We are looking for an experienced Azure Databricks Data Engineer to help build and operate scalable, production-grade data platforms on Azure Databricks and Microsoft Azure.
The ideal candidate combines strong Spark engineering expertise with a production engineering mindset, treating data pipelines as reliable, enterprise-grade products rather than ad-hoc solutions. This role requires hands-on experience delivering secure, governed, high-performance batch and streaming data platforms at scale.
Tasks- Design, build, and maintain scalable batch and streaming pipelines using bronze/silver/gold architectures
- Develop reusable transformation frameworks and production-grade Spark workloads
- Implement and optimize Delta Lake solutions, including: ACID transactions, Partitioning strategies, Schema evolution, OPTIMIZE/ZORDER, Incremental and CDC processing patterns
- Configure and manage Unity Catalog permissions, governance, lineage, and auditability
- Perform Spark performance tuning and observability monitoring
- Integrate Databricks with Azure services including ADLS, ADF/Synapse, Key Vault, and Microsoft Entra ID
- Implement CI/CD pipelines and deployment automation using Azure DevOps or GitHub Actions
- Collaborate with architects, analysts, data scientists, and engineers to align technical implementation with business needs
- Continuously improve platform performance, reliability, cost efficiency, and standardization
-
Strong hands-on expertise with Azure Databricks engineering
-
Advanced Apache Spark experience using:
-
PySpark or Scala
-
Spark SQL
-
Deep understanding of Delta Lake architecture and optimization techniques
-
Experience building scalable enterprise data platforms and real-time pipelines
-
Strong governance and security experience in multi-team environments
-
Experience with service principals, managed identities, lineage, and auditability
-
Strong troubleshooting and Spark performance optimization skills
-
Azure integration experience across storage, orchestration, and security services
-
Solid CI/CD and Infrastructure-as-Code exposure
-
Comfortable working independently in consulting-style delivery environments
Cypher Consulting Europe takes pride in its ability to deliver expert consultants across a wide spectrum of technologies and industry sectors.