Understanding the problem and source data
Data engineering often involves collecting, storing, and analyzing data. But nearly all data engineering landscapes start with ingesting raw data into a data lake or a data warehouse. In this chapter, we will be discussing one such typical use case and build an end-to-end solution for the problem discussed in the following section.
Problem statement
Company XYZ is a third-party vendor that provides services for building and maintaining data centers. Now, Company XYZ is planning to build a data center monitoring tool for its customer. The customer wants to see various useful metrics, such as the number of incidents reported for any device on an hourly, monthly, or quarterly basis. They also want reports on closure ratios and average closure duration. They are also interested in searching incidents based on the type of device or incident type. They are also interested to find time-based outage patterns to predict seasonal or hourly usage...