What this book covers
Chapter 1, Hi Optimus!, shows us what Optimus is, why it was created, and the goals of the project. We will get a good understanding of how Optimus works internally, how it is different from the current technology, and how it works in tandem with users to bring the best of the technology to serve users' data processing needs.
Chapter 2, Data Loading, Saving, and File Formats, is all about how to use Optimus to load and save data from text data formats such as CSV and JSON files. Also, we will explore binary files such as Excel and some optimized for columnar data processing such as Parquet, Avro, and OCR. Lastly, we will learn how to connect to databases such as SQLite and remote data storage such as Redshift.
Chapter 3, Data Wrangling, demonstrates how to concatenate data row- and column-wise and how to use SQL-like syntax to merge data using left, right, inner, and outer methods. Also, we'll learn how to pivot data tables to put data in the shape needed for the next step in the data pipeline.
Chapter 4, Combining, Reshaping, and Aggregating Data, teaches us how to group columns of data and apply summary statistics to each group of data. From count, min, and max aggregation to more advanced stats such as kurtosis and skew, we'll have all the tools to calculate any stats needed.
Chapter 5, Data Visualization and Profiling, demonstrates the profiler and the profiler data types, an Optimus DataFrame unique feature that lets the user have an overview of the data quality on every column. Data types such as email, dates, URLs, string, and float let the user easily standardize mismatches and missing data like any other library.
Chapter 6, String Clustering, uses string clustering techniques to let us easily identify groups of similar strings and replace them with a unique value.
Chapter 7, Feature Engineering, teaches us how to create new features for machine learning models to learn from. We'll look at generating them by combining fields, extracting values from messy columns, or encoding them for better results.
Chapter 8, Machine Learning, shows us how to easily create machine learning models and how Optimus will take care of the implementation details and make the feature engineering work when possible, as well as how to save the model after training and load it for future use.
Chapter 9, Natural Language Processing, shows us how to easily prepare data to apply techniques such as a word cloud, data summarization, and sentiment analysis, along with examples.
Chapter 10, Hacking Optimus, explores how to add new profiler data types to better approach quality problems. Also, we will learn how to create data cleaning and transformation functions and how to add a hypothetical new data processing engine. To close, we will talk about the Optimus Community, how to contribute to the project, and the next step in the Optimus project.
Chapter 11, Optimus as a Web Service, demonstrates how Optimus can be used as a web service with the help of various tools and plugins.