Summary
In this chapter, we deviated from our usual personas of data analysts and database administrators. Instead, we put on the hat of data engineers who are responsible for delivering data to analysts and administrators. Data engineers are wholly responsible for optimizing the data layout on the Lakehouse so that data analysts get the best possible query experience.
That said, we saw how the inputs of data analysts will be integral to getting the data layout correct. Inputs on query-filtering patterns, ordering of data columns, and frequency of optimizations are required for data engineers to make the best decisions.
In this chapter, we learned how Delta Lake is the storage layer for Databricks SQL. We learned how it provides the best out-of-the-box query experience. We also learned about additional features in Delta Lake that can elevate query performance even more. Finally, we discussed the internal workings of Delta Lake and how it enables all the features that Delta Lake...