-
Book Overview & Buying
-
Table Of Contents
Data Engineering with Azure Databricks
By :
In the previous chapters, we focused on techniques and approaches for building data pipelines. You ingested data from source systems, built transformations, materialized the results as Delta Lake tables, and applied initial performance optimizations - the pipeline works.
The next question is how to run this pipeline in production. How do you safely deploy it and evolve it over time? What happens when you need to fix bugs, improve performance, or introduce new features? And how do you manage the development and maintenance of multiple data-engineering projects, with multiple developers working in parallel?
These questions move us beyond pipeline implementation and into production readiness, collaborative development, and the operational lifecycle of data pipelines.
That's where DevOps practices for data engineering come in - such as version control, CI/CD, environment separation, testing, release management, and operational...