Summary

In this overview of the business problem, you have learned a number of foundational elements that will be elaborated on in subsequent chapters. This chapter introduced the topics needed to gain an understanding of the current state of data engineering and the creation of future-proof designs. You have learned that businesses are faced with an ever-changing technological landscape. Competition requires one to innovate at scale to remain relevant. This causes a constant implementation stream of total-cost-of-ownership (TCO) budget allocations for refactoring and re-envisioning during what would normally be a run/manage phase of a system’s lifespan. In this chapter, and in subsequent chapters, we make many references to the engineering solution’s TCO. These references will be reminders to all stakeholders that the solutions developed are within the real world business setting. They are not created in some abstract vacuum, devoid of budgeting constraints that will, at times, limit possibilities. It is important to note that when the TCO is clear, yet constrained by budgets, these constraints repeatedly appear on the monthly and quarterly radar reports presented to the enterprise. These constraints will most likely have imposed risk. Without a constant stream of reminders, the business will forget how these constraints have impacted the solution.

Additionally, building a system that perpetuates false facts, even if spun as true facts, is foolish. Make the future data solution smart! We are entering an exciting future where data and information solutions will become smarter and support knowledge and intelligence capabilities. Embrace the change and know its implications on your data engineering choices. DataOps needs to be adopted by data professionals as a critical approach to managing data in today’s complex, data-driven world.

One size does not fit all and as such, building with data contracts in mind will force the development of data stores with the same logical data into the physical data architecture as fit-for-purpose parallel instantiations. Correctly building data solutions to be future-proof requires a vision, strategy, mission, and architectural approach to prevent the implementation from dying an untimely death due to the juggling needed to get the solution serviceable for the business.

Third-party vendors and cloud providers will produce well architected solutions that do not integrate, or worse yet, that foster architectural anti-patterns that must be avoided. As such, the data mesh and the cloud provider’s data fabrics are only buzzwords until the concepts are fully understood and rationalized into your architecture and organization’s objectives. Design data solutions consistently to the architecture you develop, develop use cases across the system, and test, regression test, and monitor them for continual service in order to maintain the trust established through data contracts.

Lastly, stay agile! Read! Learn! Be innovative! Once the big picture is grasped, the forward-looking perspective will grant you the foresight to look beyond the obstacles that will be encountered. You will be able to keep the data solution and its data fresh and current with a governed, agile architectural process.

In the next chapter, you will be presented with the architectural background challenges that build on this overview.