Book Image

Learn MongoDB 4.x

By : Doug Bierer
Book Image

Learn MongoDB 4.x

By: Doug Bierer

Overview of this book

When it comes to managing a high volume of unstructured and non-relational datasets, MongoDB is the defacto database management system (DBMS) for DBAs and data architects. This updated book includes the latest release and covers every feature in MongoDB 4.x, while helping you get hands-on with building a MongoDB database app. You’ll get to grips with MongoDB 4.x concepts such as indexes, database design, data modeling, authentication, and aggregation. As you progress, you’ll cover tasks such as performing routine operations when developing a dynamic database-driven website. Using examples, you’ll learn how to work with queries and regular database operations. The book will not only guide you through design and implementation, but also help you monitor operations to achieve optimal performance and secure your MongoDB database systems. You’ll also be introduced to advanced techniques such as aggregation, map-reduce, complex queries, and generating ad hoc financial reports on the fly. Later, the book shows you how to work with multiple collections as well as embedded arrays and documents, before finally exploring key topics such as replication, sharding, and security using practical examples. By the end of this book, you’ll be well-versed with MongoDB 4.x and be able to perform development and administrative tasks associated with this NoSQL database.

Related Content you might be interested in

Current Title:

Small book image
Learn MongoDB 4.x
Doug Bierer
Sep, 2020 | 610 pages
No titles found
Table of Contents (22 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Download the example code files
Download the color images
Conventions used
Get in touch
Reviews
1
Section 1: Essentials
Section 1: Essentials
Free Chapter
2
Introducing MongoDB 4.x
Introducing MongoDB 4.x
High-level technology overview of MongoDB 4.x
MongoDB is based upon documents
High availability
Horizontal scaling
Discovering what's new and different in MongoDB 4.x
Significant new features
Multidocument ACID transaction support
Nonblocking secondary reads
In-progress index build interruption
Other noteworthy new features
Aggregation pipeline type conversions
Client-side field-level encryption
Password prompt
Wildcard indexes
Extended JSON v2 support
Improved logging and diagnostics
Hedged reads
TCP fast open support
Natural sort
Internal enhancements
Timestamps in the storage engine
Shard migration
Change streams
Important new security enhancements
SHA-256 support
TLS handling
Avoiding problems when upgrading from MongoDB 3.x to 4.x
MMAPv1 storage engine
Replica set protocol version
Verifying that at least one oplog entry has replicated
Upgrading the primary to protocol version 1
Feature compatibility
User authentication
Removed and deprecated items
Removed items
Significant removed items
Deprecated SSL configuration options
Summary
3
Setting Up MongoDB 4.x
Setting Up MongoDB 4.x
Technical requirements
Installing MongoDB on a customer site
Available versions of MongoDB 4.x
Understanding RAM and CPU requirements
Examining storage requirements
Choosing the filesystem type
Other server considerations
Installing MongoDB 4.x
MongoDB 3 and MongoDB 4.x installation differences
Installing MongoDB 4.x on Linux
Installing on Windows and macOS
Configuring MongoDB 4.x
The name and location of the MongoDB configuration file
YAML configuration file syntax
Configuration options
Storage configuration options
Logging configuration options
Network configuration options
Example MongoDB configuration file
Command-line options
Expansion directives
Using the __rest expansion directive
Using the __exec expansion directive
Installing the PyMongo driver package
Python driver compatibility
pip3
Installing PyMongo using pip
Testing the connection between Python and MongoDB
Loading the sample data
Installing the source code and sample data using Git
Installing source and sample data by downloading and unzipping
Understanding the source code directory structure
Creating the demo environment using Docker and Docker Compose
Installing Docker and Docker Compose
Configuring Docker for Chapter 2
Configuring Docker for Chapters 3 to 12
Running the demo environment
Connecting to the container from your host computer
Summary
4
Essential MongoDB Administration Techniques
Essential MongoDB Administration Techniques
Technical requirements
Connecting with the mongo shell
The mongo shell command-line switches
Customizing the mongo shell
Shell command helpers
Using mongo shell methods
Running scripts using the shell
Running a script from the command line
Running a script from inside the shell
Shell script syntax
Using the Mongo shell for common database operations
Creating a new database and collection
Inserting documents
Querying a collection
Simple queries
Building complex queries using query operators
Applying projections to your query
Updating a document
Update operators
Updating customer contact information
Data cleanup
Deleting a document
Performing backup and restore operations
Using mongodump to back up a local database
mongodump options summary
Things to consider before you back up
Backing up a specific database
Restoring using mongorestore
Things to consider before restoring data
Restoring the purchases collection
Using the --nsInclude switch to restore
Performance monitoring
Mongo shell stats() methods
Monitoring server status
Summary
5
Section 2: Building a Database-Driven Web Application
Section 2: Building a Database-Driven Web Application
6
Fundamentals of Database Design
Fundamentals of Database Design
Technical requirements
Reviewing customer requirements
Introducing Sweets Complete Inc.
What data comes out of the application?
What data goes into the application?
What needs to be stored?
A quick word on software design
Building MongoDB document structures
Defining the products document structure
Primary key considerations
Defining the purchases document structure
Defining the common document structure
Developing a corresponding Python module
Determining common needs in a base class
Defining classes for the products collection
Demo script for product
Unit test for the product entity
Defining classes for the remaining collections
Summary
7
Mission-Critical MongoDB Database Tasks
Mission-Critical MongoDB Database Tasks
Technical requirements
Creating a Connection class
pymongo.MongoClient
Creating the connection
Defining domain service classes
Products collection domain service
Generating a product sales report
Using the pymongo.collection.Collection.find() class
Modeling the query in JavaScript
Adding the appropriate methods to the domain service class
Generic Query Method approach
Custom Query Method approach
Updating product information
Handling a customer purchase
Web environment support classes
The web.session.Session class
The web.auth.SimpleAuth class
The web.responder.Html class
The pymongo.collection.Collection.insert_one() and insert_many() class
Inserting documents into the purchase domain service
Web application for product purchase
Configuring Apache for Python CGI
Simple customer authentication
Selecting products from a web page
Capturing and processing purchase information from a web form
Canceling a purchase
The pymongo.collection.Collection.delete_one() and delete_many() methods
Integrating purchase cancellation into the web flow
Summary
8
Using AJAX and REST to Build a Database-Driven Website
Using AJAX and REST to Build a Database-Driven Website
Technical requirements
Paginating MongoDB database queries
Modifying the purchases domain service
Adding pagination to the responder class
Consolidating configuration
Adding pagination to a web application
Handling AJAX requests
A brief overview of jQuery and DataTables
Incorporating jQuery and DataTables
Defining a JSON responder class
Delivering data from MongoDB
Creating an AJAX handler web action script
Modifying the product select web action script
Modifying the product purchase web action script
Serving binary data directly from MongoDB
Producing a Base64 image tag
Creating an HTML template for the product details
Creating a display script
Modifying the AJAX response
Revisiting the product selection action script
Responding to REST requests
Understanding REST
Understanding HTTP
HTTP status codes
HTTP methods
JSON standards for a REST response
JSend
JSON:API
hal+json
Strategies for creating a REST responder
Response strategy base class
JSend strategy
The JSON:API strategy
The hal+json strategy
Creating a REST responder
Modifying the products domain service
REST request action handler
Summary
9
Section 3: Digging Deeper
Section 3: Digging Deeper
10
Advanced MongoDB Database Design
Advanced MongoDB Database Design
Technical requirements
Reviewing customer requirements
Introducing Book Someplace, Inc.
What data is produced by the application?
What data goes into the application?
What needs to be stored?
Building document structures
Defining a name structure
Defining a location structure
Defining a contact structure
Additional structures pertaining to users
Defining a property information structure
Defining rooms
Defining a structure for booking
Defining intermediary structures
Customer booking structures
Property booking structures
Putting it all together
Customer collection document structure
Partner collection document structure
Property collection document structure
Booking collection document structure
Common collection document structure
Developing the corresponding Python entity classes
Defining the booking module
Defining the Common module
Testing an entity class
Defining domain service classes
Base domain service class
Property domain service class
Using MongoDB with Django
Determining which web framework to use
Installing the web framework and MongoDB library
Apache configuration
Book Someplace Django project structure
Making modifications to the main project
Presenting the view
Displaying results in the view template
Summary
11
Using Documents with Embedded Lists and Objects
Using Documents with Embedded Lists and Objects
Technical requirements
Overview
Adding a document with embedded objects and arrays
Defining forms for property subclasses
Defining templates to match sub-forms
Defining a FormService class
Defining view logic to handle form rendering and posting
Defining a route to addEdit()
Generating the property key
Updating the property domain service
Adding a new property
Adding to arrays embedded within documents
Defining an add room form
Defining the view method and URL route
Updating the database
MongoDB update operators
Domain service method to add rooms
Using the form
Updating documents with embedded objects
Choosing the partner
Partner and property AJAX lookups
Contact information form and view
Defining JavaScript functions to generate AJAX requests
Using operators to update contact information
Using the contact update form
Querying properties of embedded objects
Defining a property listing view method
Room types view method
Adding the appropriate URL routes
Property listing template
jQuery AJAX function requesting room types
Listing results
Removing a document
Defining a property domain service deletion method
Choosing the partner and property
Defining the choose and confirm delete template
Defining a URL route
Summary
12
Handling Complex Queries in MongoDB
Handling Complex Queries in MongoDB
Technical requirements
Modeling complex queries using MongoDB Compass
Understanding MongoDB Compass
Installing MongoDB Compass
MongoDB Compass initial connection
SSL connection options
Making the connection
Formulating queries involving embedded objects
Building queries involving embedded arrays
Using the aggregation framework
Working with the MongoDB aggregate() method
Pipeline stage operators
Pipeline expression operators
String expression operators
Date expression operators
Other important expression operators
Accumulator expression operators
Working with Map-Reduce functions
Understanding the mapReduce() method syntax
Reviewing a Map-Reduce example
Handling geospatial data
Working with the flat 2D model
Using aggregation to create the geospatial field
Defining an index
Understanding GeoJSON objects
Working with the spherical 2D geospatial model
Using geospatial query operators
Examining the geospatial aggregation pipeline stage operator
Using aggregation to produce financial reports
Using aggregation to generate revenue reports
Using the mongo shell to build a query
Generating 30-60-90 day aging reports using MongoDB aggregation
Performing risk analysis through revenue trends
Setting up the Django infrastructure
Defining a form to capture query parameters
Defining view methods
Defining the view template
Defining a method to generate trend data
Viewing the results
Summary
13
Section 4: Replication, Sharding, and Security in a Financial Environment
Section 4: Replication, Sharding, and Security in a Financial Environment
14
Working with Complex Documents Across Collections
Working with Complex Documents Across Collections
Technical requirements
Introducing BigLittle Micro Finance Ltd.
Handling monetary data in MongoDB
User collection data structures
Loan collection data structure
Common collections
Storing financial data
Referencing documents across collections
Techniques for referencing documents across collections
The building block approach
Embedded documents
Document references
Database references (DBRefs)
$lookup aggregation pipeline stage
Practical example using document references
Gathering information to generate a loan proposal
Creating a loan document using document references
Practical example using the $lookup aggregation pipeline operator
Connecting two collections using $lookup
Producing Python code from the JSON query
Performing secondary updates
Accepting a loan payment
Django view logic for choosing the borrower
Django view logic for payment processing
Payment processing domain service methods
Loan payment view template
Defining code to process secondary updates
The publish/subscribe design pattern
Defining listeners for loan events
Event notification
Avoiding cross-collection problems
Ensuring uniqueness
Using ObjectId as a unique key
Defining a custom unique key
Enforcing uniqueness
Avoiding creating orphans across collections
Using embedded arrays to avoid orphans
Using embedded documents to avoid orphans
Keeping values properly synchronized
Uploading files into GridFS
Why use GridFS?
How to use GridFS from the command line
GridFS command-line options
GridFS commands
Command-line usage examples
PyMongo GridFS support
Example using GridFS to store loan documents
Summary
15
Administering MongoDB Security
Administering MongoDB Security
Technical requirements
Enabling database security
Understanding MongoDB authentication
Authentication versus authorization
Understanding the authentication database
Authentication mechanisms
Creating a database user
Enabling database security
Understanding role-based access control
Understanding privilege actions (rights)
Operational rights
Maintenance rights
Understanding built-in roles
Assigning built-in roles applying to a single database
Creating a user with rights on a single database
Assigning built-in roles applying to all databases
Creating a user with rights on all databases
Administrative built-in roles
Creating a user with SCRAM-based authentication
Setting up transport layer security with X.509 certificates
Understanding TLS, SSL and the transport layer
Certificates needed for secure MongoDB communications
Configuring mongod for TLS/SSL
Establishing secure communications using PEM key files
Creating a client connection using TLS
Summary
16
Developing in a Secured Environment
Developing in a Secured Environment
Technical requirements
Developing applications with minimal rights
Creating a database user with read rights
Configuring the application for a database user
Limiting an application to a single database
Creating applications with expanded rights
Applications with rights to any database
Applications that perform database and collection administration
Applications that perform database user management
Configuring applications to communicate using TLS
PyMongo client TLS configuration options
Sample app connecting with TLS
Summary
17
Deploying a Replica Set
Deploying a Replica Set
Technical requirements
MongoDB replication overview
What is a replica set?
What are some potential replica set benefits?
What happens if a server goes down?
Replica set data synchronization
Proper replica set design
Establishing a business rationale
Modeling a replica set using Docker
Creating the Dockerfile
Bringing up multiple containers using Docker Compose
Replica set configuration and deployment
Replication configuration settings
Deploying the replica set
Network connectivity between replica set members
Replica set member configuration
Starting mongod instances
Initiating the replica set
Replica set configuration fields
Replica set members configuration directives
Replica set settings configuration directives
Replica set configuration example
Confirming the replica set status
Restoring the sample data
Summary
18
Replica Set Runtime Management and Development
Replica Set Runtime Management and Development
Technical requirements
Managing a running replica set
Managing the oplog size
Monitoring the replication status
Controlling replication lag
Backing up and restoring a replica set
Backing up using mongodump
Restoring a replica set
Resynchronizing a replica set member
Resynchronizing by forcing an initial sync
Resynchronizing by manual file copy
Managing replica set impact on application program code
Basic replica set connection
Manipulating read preferences
Managing write concerns
Summary
19
Deploying a Sharded Cluster
Deploying a Sharded Cluster
Technical requirements
MongoDB sharding overview
What is a sharded cluster?
Why deploy a sharded cluster?
Vertical scaling
Horizontal scaling
Developing a business rationale for sharded cluster deployment
Sharded cluster configuration
Understanding the shard key
Document distribution within shards
Choosing a shard key
Cardinality
Frequency
Change
Shard key strategies
Hashed sharding
Range-based sharding
Sharded cluster member configuration
Sharded cluster deployment
Using Docker to model a sharded cluster
Understanding the docker-compose.yml file
Additional files needed by Docker and MongoDB
Confirming sharded cluster network communications
Deploying the config server replica set
Deploying shard replica sets
Starting a mongos instance
Restoring the sample data
Adding shards to the cluster
Enabling sharding on the database
Choosing a shard key
Sharding the collection
Summary
20
Sharded Cluster Management and Development
Sharded Cluster Management and Development
Technical requirements
Managing a sharded cluster
Monitoring the sharded cluster status
Working with sharded cluster zones
Defining zones
Assigning ranges to zones
Working with the sharded cluster balancer
Determining the balancer state
Disabling the balancer
Enabling the balancer
Managing chunk size
Adjusting chunk size
Splitting a chunk
Manually migrating a chunk
Backing up and restoring a sharded cluster
Sharded cluster impacts on program code
Connecting to a sharded cluster
Reading from a sharded cluster
Writing to a sharded cluster
Summary
21
Other Books You May Enjoy
Other Books You May Enjoy
Leave a review - let other readers know what you think

Configuration options

It is important to provide configuration to direct how the mongod server daemon starts. Although you could theoretically start mongod using command-line options, most DevOps prefer to place a permanent configuration into a configuration file. Configuration options fall into these major categories:

  • Core: Storage, logging and network
  • Process management
  • Security
  • Profiling
  • Replication and sharding

In addition, the Enterprise (that is, paid) version of MongoDB features auditing and Simple Network Management Protocol (SNMP) options. Of these options in this chapter, we will only focus on the Core category. Other options will be covered in later chapters as those topics arise.

NOTE: The name and location of the configuration file vary depending on the operating system. See the earlier section on installing MongoDB to get an idea of the default name and location of this file.
Previous Section
End of Section 16
Next Section