Book Image

Learn MongoDB 4.x

By : Doug Bierer

Book Image

Learn MongoDB 4.x

By: Doug Bierer

Overview of this book

When it comes to managing a high volume of unstructured and non-relational datasets, MongoDB is the defacto database management system (DBMS) for DBAs and data architects. This updated book includes the latest release and covers every feature in MongoDB 4.x, while helping you get hands-on with building a MongoDB database app. You’ll get to grips with MongoDB 4.x concepts such as indexes, database design, data modeling, authentication, and aggregation. As you progress, you’ll cover tasks such as performing routine operations when developing a dynamic database-driven website. Using examples, you’ll learn how to work with queries and regular database operations. The book will not only guide you through design and implementation, but also help you monitor operations to achieve optimal performance and secure your MongoDB database systems. You’ll also be introduced to advanced techniques such as aggregation, map-reduce, complex queries, and generating ad hoc financial reports on the fly. Later, the book shows you how to work with multiple collections as well as embedded arrays and documents, before finally exploring key topics such as replication, sharding, and security using practical examples. By the end of this book, you’ll be well-versed with MongoDB 4.x and be able to perform development and administrative tasks associated with this NoSQL database.

Preface

Who this book is for

What this book covers

To get the most out of this book

Download the example code files

Download the color images

Conventions used

Section 1: Essentials

Section 1: Essentials

Free Chapter

Introducing MongoDB 4.x

Introducing MongoDB 4.x

High-level technology overview of MongoDB 4.x

MongoDB is based upon documents

High availability

Horizontal scaling

Discovering what's new and different in MongoDB 4.x

Significant new features

Multidocument ACID transaction support

Nonblocking secondary reads

In-progress index build interruption

Other noteworthy new features

Aggregation pipeline type conversions

Client-side field-level encryption

Password prompt

Wildcard indexes

Extended JSON v2 support

Improved logging and diagnostics

TCP fast open support

Internal enhancements

Timestamps in the storage engine

Shard migration

Important new security enhancements

SHA-256 support

Avoiding problems when upgrading from MongoDB 3.x to 4.x

MMAPv1 storage engine

Replica set protocol version

Verifying that at least one oplog entry has replicated

Upgrading the primary to protocol version 1

Feature compatibility

User authentication

Removed and deprecated items

Significant removed items

Deprecated SSL configuration options

Setting Up MongoDB 4.x

Setting Up MongoDB 4.x

Technical requirements

Installing MongoDB on a customer site

Available versions of MongoDB 4.x

Understanding RAM and CPU requirements

Examining storage requirements

Choosing the filesystem type

Other server considerations

Installing MongoDB 4.x

MongoDB 3 and MongoDB 4.x installation differences

Installing MongoDB 4.x on Linux

Installing on Windows and macOS

Configuring MongoDB 4.x

The name and location of the MongoDB configuration file

YAML configuration file syntax

Configuration options

Storage configuration options

Logging configuration options

Network configuration options

Example MongoDB configuration file

Command-line options

Expansion directives

Using the __rest expansion directive

Using the __exec expansion directive

Installing the PyMongo driver package

Python driver compatibility

Installing PyMongo using pip

Testing the connection between Python and MongoDB

Loading the sample data

Installing the source code and sample data using Git

Installing source and sample data by downloading and unzipping

Understanding the source code directory structure

Creating the demo environment using Docker and Docker Compose

Installing Docker and Docker Compose

Configuring Docker for Chapter 2

Configuring Docker for Chapters 3 to 12

Running the demo environment

Connecting to the container from your host computer

Essential MongoDB Administration Techniques

Essential MongoDB Administration Techniques

Technical requirements

Connecting with the mongo shell

The mongo shell command-line switches

Customizing the mongo shell

Shell command helpers

Using mongo shell methods

Running scripts using the shell

Running a script from the command line

Running a script from inside the shell

Shell script syntax

Using the Mongo shell for common database operations

Creating a new database and collection

Inserting documents

Querying a collection

Building complex queries using query operators

Applying projections to your query

Updating a document

Update operators

Updating customer contact information

Deleting a document

Performing backup and restore operations

Using mongodump to back up a local database

mongodump options summary

Things to consider before you back up

Backing up a specific database

Restoring using mongorestore

Things to consider before restoring data

Restoring the purchases collection

Using the --nsInclude switch to restore

Performance monitoring

Mongo shell stats() methods

Monitoring server status

Section 2: Building a Database-Driven Web Application

Section 2: Building a Database-Driven Web Application

Fundamentals of Database Design

Fundamentals of Database Design

Technical requirements

Reviewing customer requirements

Introducing Sweets Complete Inc.

What data comes out of the application?

What data goes into the application?

What needs to be stored?

A quick word on software design

Building MongoDB document structures

Defining the products document structure

Primary key considerations

Defining the purchases document structure

Defining the common document structure

Developing a corresponding Python module

Determining common needs in a base class

Defining classes for the products collection

Demo script for product

Unit test for the product entity

Defining classes for the remaining collections

Mission-Critical MongoDB Database Tasks

Mission-Critical MongoDB Database Tasks

Technical requirements

Creating a Connection class

pymongo.MongoClient

Creating the connection

Defining domain service classes

Products collection domain service

Generating a product sales report

Using the pymongo.collection.Collection.find() class

Modeling the query in JavaScript

Adding the appropriate methods to the domain service class

Generic Query Method approach

Custom Query Method approach

Updating product information

Handling a customer purchase

Web environment support classes

The web.session.Session class

The web.auth.SimpleAuth class

The web.responder.Html class

The pymongo.collection.Collection.insert_one() and insert_many() class

Inserting documents into the purchase domain service

Web application for product purchase

Configuring Apache for Python CGI

Simple customer authentication

Selecting products from a web page

Capturing and processing purchase information from a web form

Canceling a purchase

The pymongo.collection.Collection.delete_one() and delete_many() methods

Integrating purchase cancellation into the web flow

Using AJAX and REST to Build a Database-Driven Website

Using AJAX and REST to Build a Database-Driven Website

Technical requirements

Paginating MongoDB database queries

Modifying the purchases domain service

Adding pagination to the responder class

Consolidating configuration

Adding pagination to a web application

Handling AJAX requests

A brief overview of jQuery and DataTables

Incorporating jQuery and DataTables

Defining a JSON responder class

Delivering data from MongoDB

Creating an AJAX handler web action script

Modifying the product select web action script

Modifying the product purchase web action script

Serving binary data directly from MongoDB

Producing a Base64 image tag

Creating an HTML template for the product details

Creating a display script

Modifying the AJAX response

Revisiting the product selection action script

Responding to REST requests

Understanding REST

Understanding HTTP

HTTP status codes

JSON standards for a REST response

Strategies for creating a REST responder

Response strategy base class

The JSON:API strategy

The hal+json strategy

Creating a REST responder

Modifying the products domain service

REST request action handler

Section 3: Digging Deeper

Section 3: Digging Deeper

Advanced MongoDB Database Design

Advanced MongoDB Database Design

Technical requirements

Reviewing customer requirements

Introducing Book Someplace, Inc.

What data is produced by the application?

What data goes into the application?

What needs to be stored?

Building document structures

Defining a name structure

Defining a location structure

Defining a contact structure

Additional structures pertaining to users

Defining a property information structure

Defining a structure for booking

Defining intermediary structures

Customer booking structures

Property booking structures

Putting it all together

Customer collection document structure

Partner collection document structure

Property collection document structure

Booking collection document structure

Common collection document structure

Developing the corresponding Python entity classes

Defining the booking module

Defining the Common module

Testing an entity class

Defining domain service classes

Base domain service class

Property domain service class

Using MongoDB with Django

Determining which web framework to use

Installing the web framework and MongoDB library

Apache configuration

Book Someplace Django project structure

Making modifications to the main project

Presenting the view

Displaying results in the view template

Using Documents with Embedded Lists and Objects

Using Documents with Embedded Lists and Objects

Technical requirements

Adding a document with embedded objects and arrays

Defining forms for property subclasses

Defining templates to match sub-forms

Defining a FormService class

Defining view logic to handle form rendering and posting

Defining a route to addEdit()

Generating the property key

Updating the property domain service

Adding a new property

Adding to arrays embedded within documents

Defining an add room form

Defining the view method and URL route

Updating the database

MongoDB update operators

Domain service method to add rooms

Updating documents with embedded objects

Choosing the partner

Partner and property AJAX lookups

Contact information form and view

Defining JavaScript functions to generate AJAX requests

Using operators to update contact information

Using the contact update form

Querying properties of embedded objects

Defining a property listing view method

Room types view method

Adding the appropriate URL routes

Property listing template

jQuery AJAX function requesting room types

Listing results

Removing a document

Defining a property domain service deletion method

Choosing the partner and property

Defining the choose and confirm delete template

Defining a URL route

Handling Complex Queries in MongoDB

Handling Complex Queries in MongoDB

Technical requirements

Modeling complex queries using MongoDB Compass

Understanding MongoDB Compass

Installing MongoDB Compass

MongoDB Compass initial connection

SSL connection options

Making the connection

Formulating queries involving embedded objects

Building queries involving embedded arrays

Using the aggregation framework

Working with the MongoDB aggregate() method

Pipeline stage operators

Pipeline expression operators

String expression operators

Date expression operators

Other important expression operators

Accumulator expression operators

Working with Map-Reduce functions

Understanding the mapReduce() method syntax

Reviewing a Map-Reduce example

Handling geospatial data

Working with the flat 2D model

Using aggregation to create the geospatial field

Defining an index

Understanding GeoJSON objects

Working with the spherical 2D geospatial model

Using geospatial query operators

Examining the geospatial aggregation pipeline stage operator

Using aggregation to produce financial reports

Using aggregation to generate revenue reports

Using the mongo shell to build a query

Generating 30-60-90 day aging reports using MongoDB aggregation

Performing risk analysis through revenue trends

Setting up the Django infrastructure

Defining a form to capture query parameters

Defining view methods

Defining the view template

Defining a method to generate trend data

Viewing the results

Section 4: Replication, Sharding, and Security in a Financial Environment

Section 4: Replication, Sharding, and Security in a Financial Environment

Working with Complex Documents Across Collections

Working with Complex Documents Across Collections

Technical requirements

Introducing BigLittle Micro Finance Ltd.

Handling monetary data in MongoDB

User collection data structures

Loan collection data structure

Common collections

Storing financial data

Referencing documents across collections

Techniques for referencing documents across collections

The building block approach

Embedded documents

Document references

Database references (DBRefs)

$lookup aggregation pipeline stage

Practical example using document references

Gathering information to generate a loan proposal

Creating a loan document using document references

Practical example using the $lookup aggregation pipeline operator

Connecting two collections using $lookup

Producing Python code from the JSON query

Performing secondary updates

Accepting a loan payment

Django view logic for choosing the borrower

Django view logic for payment processing

Payment processing domain service methods

Loan payment view template

Defining code to process secondary updates

The publish/subscribe design pattern

Defining listeners for loan events

Event notification

Avoiding cross-collection problems

Ensuring uniqueness

Using ObjectId as a unique key

Defining a custom unique key

Enforcing uniqueness

Avoiding creating orphans across collections

Using embedded arrays to avoid orphans

Using embedded documents to avoid orphans

Keeping values properly synchronized

Uploading files into GridFS

Why use GridFS?

How to use GridFS from the command line

GridFS command-line options

GridFS commands

Command-line usage examples

PyMongo GridFS support

Example using GridFS to store loan documents

Administering MongoDB Security

Administering MongoDB Security

Technical requirements

Enabling database security

Understanding MongoDB authentication

Authentication versus authorization

Understanding the authentication database

Authentication mechanisms

Creating a database user

Enabling database security

Understanding role-based access control

Understanding privilege actions (rights)

Operational rights

Maintenance rights

Understanding built-in roles

Assigning built-in roles applying to a single database

Creating a user with rights on a single database

Assigning built-in roles applying to all databases

Creating a user with rights on all databases

Administrative built-in roles

Creating a user with SCRAM-based authentication

Setting up transport layer security with X.509 certificates

Understanding TLS, SSL and the transport layer

Certificates needed for secure MongoDB communications

Configuring mongod for TLS/SSL

Establishing secure communications using PEM key files

Creating a client connection using TLS

Developing in a Secured Environment

Developing in a Secured Environment

Technical requirements

Developing applications with minimal rights

Creating a database user with read rights

Configuring the application for a database user

Limiting an application to a single database

Creating applications with expanded rights

Applications with rights to any database

Applications that perform database and collection administration

Applications that perform database user management

Configuring applications to communicate using TLS

PyMongo client TLS configuration options

Sample app connecting with TLS

Deploying a Replica Set

Deploying a Replica Set

Technical requirements

MongoDB replication overview

What is a replica set?

What are some potential replica set benefits?

What happens if a server goes down?

Replica set data synchronization

Proper replica set design

Establishing a business rationale

Modeling a replica set using Docker

Creating the Dockerfile

Bringing up multiple containers using Docker Compose

Replica set configuration and deployment

Replication configuration settings

Deploying the replica set

Network connectivity between replica set members

Replica set member configuration

Starting mongod instances

Initiating the replica set

Replica set configuration fields

Replica set members configuration directives

Replica set settings configuration directives

Replica set configuration example

Confirming the replica set status

Restoring the sample data

Replica Set Runtime Management and Development

Replica Set Runtime Management and Development

Technical requirements

Managing a running replica set

Managing the oplog size

Monitoring the replication status

Controlling replication lag

Backing up and restoring a replica set

Backing up using mongodump

Restoring a replica set

Resynchronizing a replica set member

Resynchronizing by forcing an initial sync

Resynchronizing by manual file copy

Managing replica set impact on application program code

Basic replica set connection

Manipulating read preferences

Managing write concerns

Deploying a Sharded Cluster

Deploying a Sharded Cluster

Technical requirements

MongoDB sharding overview

What is a sharded cluster?

Why deploy a sharded cluster?

Vertical scaling

Horizontal scaling

Developing a business rationale for sharded cluster deployment

Sharded cluster configuration

Understanding the shard key

Document distribution within shards

Choosing a shard key

Shard key strategies

Hashed sharding

Range-based sharding

Sharded cluster member configuration

Sharded cluster deployment

Using Docker to model a sharded cluster

Understanding the docker-compose.yml file

Additional files needed by Docker and MongoDB

Confirming sharded cluster network communications

Deploying the config server replica set

Deploying shard replica sets

Starting a mongos instance

Restoring the sample data

Adding shards to the cluster

Enabling sharding on the database

Choosing a shard key

Sharding the collection

Sharded Cluster Management and Development

Sharded Cluster Management and Development

Technical requirements

Managing a sharded cluster

Monitoring the sharded cluster status

Working with sharded cluster zones

Assigning ranges to zones

Working with the sharded cluster balancer

Determining the balancer state

Disabling the balancer

Enabling the balancer

Managing chunk size

Adjusting chunk size

Splitting a chunk

Manually migrating a chunk

Backing up and restoring a sharded cluster

Sharded cluster impacts on program code

Connecting to a sharded cluster

Reading from a sharded cluster

Writing to a sharded cluster

Other Books You May Enjoy

Other Books You May Enjoy

Leave a review - let other readers know what you think

Customer Reviews

5 star

0

4 star

0

3 star

0

2 star

0

1 star

0

Understanding the source code directory structure

The source code and sample data for the examples shown in the book are located in these directories under /path/to/repo:

chapters: Under this subdirectory, you will find source code for each chapter, by number.
www: This directory structure contains source code based upon Django.
sample_data: In this directory, you will find JavaScript files that can be executed to populate the database with sample data. There is no need to insert data just yet: as the need arises, you will be instructed in the chapter to insert the sample data appropriate for that section of the book.

Next, we will have a look at creating a demo environment you can use to test the examples shown in the book using Docker and Docker Compose.