One of the most frequently asked questions by newcomers to Express is how to structure an application. There is no definitive answer for this, and we may choose different solutions based on how big our application is or what problem we are trying to tackle. Luckily for us, Express is easy to customize, and we can apply whatever structure we deem necessary.
If the code base is small, we can include everything into a few files or even a single one. This might be the case when exposing a database over HTTP (such as LevelDB and PouchDB) and creating mountable applications (these tend to be small and solve a specific problem) or other small applications.
When dealing with medium and large projects, the best thing to do is to split them into smaller pieces, making them easier to debug and test. If there are parts of the application that can be reused for other projects, the best thing to do is to move them into their separate repository.
An interesting technique to structure an application is to group files by the features they provide instead of grouping them by their function. In MVC, the controllers, models, and views live inside their own folder; however, with this approach, we have folders that group files with the same role. For example, consider the following folders:
One could go even further and choose to include things such as tests and static assets that belong to a feature inside its folder.
If there's something that can be reused for multiple features such as the general layout or models, we can group them inside their own folder. Each of these folders can export an Express application with its own view engine, middleware, and other customizations. These folders can reside in a parent lib
folder, for example. We will then require them in the main app.js
file like we would any regular middleware. It's a good way to separate concerns, although they are not necessarily complete, independent pieces because they rely on application-specific logic.
An advantage this structure offers is that when we are working on a certain section of an application, all the files that need to be created/edited are in the same location, so there's no need to switch between controllers, models, and views like with MVC.
It's worth mentioning that the creator of Express, TJ Holowaychuk, recommends this approach for larger applications instead of MVC.
The most common technique to structure web applications with Express is MVC. When generating a project using the Express CLI, it almost provides an MVC structure, omitting the models
folder. The following screenshot lists all the files and folders generated for a sample application using the CLI tool:
The package.json
file is automatically populated with the name of the application, the dependencies, the private attribute, and the starting script. This starting script is named app.js
and loads all the middleware, assigns the route handlers, and starts the server. There are three folders in the root:
Apart from these existing folders and the models
folder, which we need to create ourselves, we might also create folders for tests, logs, or configuration. The best thing about this structure is that it's easy to get started with and is known to most developers.
Let's apply the theory in practice now and create an MVC file manager application using Express 4.x and Mongoose (an object modeling library for MongoDB). The application should allow users to register and log in and enable them to view, upload, and delete their files.
We will start by creating the folder structure. First, we'll use the Express CLI tool in the terminal to create the boilerplate. Apart from the public
, routes
, and views
folders, we also need to add folders for models
, helpers
(view helpers), files
(the files uploaded by users will be stored in subfolders here), and lib
(used for internal app libraries):
$ express FileManager $ cd FileManager $ mkdir {models,helpers,files,lib}
By default, the CLI tool will create two dependencies in your package.json
file—express
and jade
—but it won't install them, so we need to manually execute the following install
command:
$ npm install .
In addition to these two modules, we also need to install mongoose
to interact with MongoDB, async
for control flow, pwd
to hash and compare passwords, connect-flash
to store messages for the user (which are then cleared after being displayed), and connect-multiparty
to handle file uploads. We can use the following shortcut to install the packages and have them declared in package.json
at the same time if we call NPM with the –save
flag:
$ npm install –save mongoose async pwd connect-flash connect-multiparty
Express 3.x came bundled with the Connect middleware, but that's not the case in the 4.x version, so we need to install them separately using the following command:
$ npm install –save morgan cookie-parser cookie-session body-parser method-override errorhandler
Note
The middleware libraries from Connect were extracted into their separate repos, so starting with Express 4.x, we need to install them separately. Read more about this topic on the Connect GitHub page at https://github.com/senchalabs/connect#middleware.
We can always check what modules are installed by entering the following command in the terminal at the root of our project:
$ npm ls
That command will output a tree with the dependencies.
Note
It's worth noting that the versions for the dependencies listed in the package.json
file will not be exact when we use the –save
flag; instead, they will be using the default npm
semver range operator. You can read more from the official
npm
documentation (https://www.npmjs.org/doc/cli/npm-install.html) and the node-semver
page (https://www.npmjs.org/package/semver).
We can get as inventive as we want with the configuration parameters of a project, like have multiple subfolders based on the environment or hierarchical configuration, but for this simple application, it's enough to have a single config.json
file. The configuration variables we need to define in this file are the MongoDB
database URL, the application port, the session secret key, and its maximum age so that our file will look like the following code:
{ "mongoUrl": "mongodb://localhost/filestore", "port": 3000, "sessionSecret": "random chars here", "sessionMaxAge": 3600000 }
Tip
Downloading the example code
You can download the example code files for all Packt books you have purchased from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.
You can also download the example code files for the book from GitHub: https://github.com/alessioalex/mastering_express_code.
In the main file of the application, named app.js
, we handle the view setup, load the middleware required for the project, connect to the database, and bind the Express application to a port. Later on, we modify this file to set up the route handling as well, but at the moment, the file contains the following code:
// Module dependencies var express = require('express'); var app = express(); var morgan = require('morgan'); var flash = require('connect-flash'); var multiparty = require('connect-multiparty'); var cookieParser = require('cookie-parser'); var cookieSession = require('cookie-session'); var bodyParser = require('body-parser'); var methodOverride = require('method-override'); var errorHandler = require('errorhandler'); var config = require('./config.json'); var routes = require('./routes'); var db = require('./lib/db'); // View setup app.set('view engine', 'jade'); app.set('views', __dirname + '/views'); app.locals = require('./helpers/index'); // Loading middleware app.use(morgan('dev')); app.use(bodyParser.json()); app.use(bodyParser.urlencoded({ extended: true })); app.use(methodOverride(function(req, res){ if (req.body && typeof req.body === 'object' && '_method' in req.body) { // look in url - encoded POST bodies and delete it var method = req.body._method; delete req.body._method; return method; } })); app.use(cookieParser()); app.use(cookieSession({ secret: config.sessionSecret, cookie: { maxAge: config.sessionMaxAge } })); app.use(flash()); if (app.get('env') === 'development') { app.use(errorHandler()); } // static middleware after the routes app.use(express.static(__dirname + '/public')); // Establishing database connection and binding application to specified port db.connect(); app.listen(config.port); console.log('listening on port %s', config.port);
Note that the preceding app.js
file contains the code for the database connection. Later on, we will need other database-related functions such as checking for failed data validation, duplicate keys, or other specific errors. We can group this logic into a separate file called db.js
inside the lib
folder and move the connection functionality there as well, as shown in the following code:
var mongoose = require('mongoose'); var config = require('../config.json'); exports.isValidationError = function(err) { return ((err.name === 'ValidationError') || (err.message.indexOf('ValidationError') !== -1)); }; exports.isDuplicateKeyError = function(err) { return (err.message.indexOf('duplicate key') !== -1); }; exports.connect = /* database connection function extracted from app.js should move here */
The routes
folder will have a file for each controller (files.js
, users.js
, and sessions.js
), another file for the application controller (main.js
), and an index.js
file that will export an object with the controllers as properties, so we don't have to require every single route in app.js
.
The users.js
file contains two functions: one to display the user registration page and another to create a user and its subfolder inside /files
, as shown in the following code:
var User = require('../models/user'); var File = require('../models/file'); var db = require('../lib/db'); exports.new = function(req, res, next) { res.render('users/new', { error: req.flash('error')[0] }); }; exports.create = function(req, res, next) { var user = new User({ username: req.body.username }); user.saveWithPassword(req.body.password, function(err) { if (err) { if (db.isValidationError(err)) { req.flash('error', 'Invalid username/password'); return res.redirect('/users/new'); } else if (db.isDuplicateKeyError(err)) { req.flash('error', 'Username already exists'); return res.redirect('/users/new'); } else { return next(err); } } File.createFolder(user._id, function(err) { if (err) { return next(err); } req.flash('info', 'Username created, you can now log in!'); res.redirect('/sessions/new'); }); }); };
The sessions.js
file handles user authentication and sign out as well as renders the login page. When the user logs in successfully, the username
and userId
properties are populated on the session object and deleted on sign out:
var User = require('../models/user'); exports.new = function(req, res, next) { res.render('sessions/new', { info: req.flash('info')[0], error: req.flash('error')[0] }); }; exports.create = function(req, res, next) { User.authenticate(req.body.username, req.body.password, function(err, userData) { if (err) { return next(err); } if (userData !== false) { req.session.username = userData.username; req.session.userId = userData._id; res.redirect('/'); } else { req.flash('error', 'Bad username/password'); res.redirect('/sessions/new'); } }); }; exports.destroy = function(req, res, next) { delete req.session.username; delete req.session.userId; req.flash('info', 'You have successfully logged out'); res.redirect('/sessions/new'); };
The files.js
controller performs CRUD-type operations; it displays all the files or a specific file for the logged-in user and saves the files or deletes them. We use res.sendfile
to display individual files because it automatically sets the correct content type and handles the streaming for us. Since the bodyParser
middleware from Express was deprecated, we replaced it with connect-multiparty
(a connect
wrapper around the multiparty
module), one of the recommended alternatives. Luckily, this module has an API similar to bodyParser
, so we won't notice any differences. Check out the complete source code of files.js
as follows:
var File = require('../models/file'); exports.index = function(req, res, next) { File.getByUserId(req.session.userId, function(err, files) { if (err) { return next(err); } res.render('files/index', { username: req.session.username, files: files, info: req.flash('info')[0], error: req.flash('error')[0] }); }); }; exports.show = function(req, res, next) { var file = new File(req.session.userId, req.params.file); file.exists(function(exists) { if (!exists) { return res.send(404, 'Page Not Found'); } res.sendfile(file.path); }); }; exports.destroy = function(req, res, next) { var file = new File(req.session.userId, req.params.file); file.delete(function(err) { if (err) { return next(err); } req.flash('info', 'File successfully deleted!'); res.redirect('/'); }); }; exports.create = function(req, res, next) { if (!req.files.file || (req.files.file.size === 0)) { req.flash('error', 'No file selected!'); return res.redirect('/'); } var file = new File(req.session.userId, req.files.file.originalFilename); file.save(req.files.file.path, function(err) { if (err) { return next(err); } req.flash('info', 'File successfully uploaded!'); res.redirect('/'); }); };
The general routes used to require user authentication or other middleware that needs to be reused for different paths can be put inside main.js
, as shown in the following code:
exports.requireUserAuth = function(req, res, next) { // redirect user to login page if they're not logged in if (!req.session.username) { return res.redirect('/sessions/new'); } // needed in the layout for displaying the logout button res.locals.isLoggedIn = true; next(); };
The index.js
file is pretty simple; it just exports all the controllers into a single object so they're easier to require in the start script of our application:
exports.main = require('./main'); exports.users = require('./users'); exports.sessions = require('./sessions'); exports.files = require('./files');
Now that we have seen what the controllers look like, we can add them to our existing app.js
file:
var routes = require('./routes'); // Declaring application routes app.get('/', routes.main.requireUserAuth, routes.files.index); app.get('/files/:file', routes.main.requireUserAuth, routes.files.show); app.del('/files/:file', routes.main.requireUserAuth, routes.files.destroy); app.post('/files', multiparty(), routes.main.requireUserAuth, routes.files.create); app.get('/users/new', routes.users.new); app.post('/users', routes.users.create); app.get('/sessions/new', routes.sessions.new); app.post('/sessions', routes.sessions.create); app.del('/sessions', routes.sessions.destroy);
Note that we included the requireUserAuth
route for all the URLs that need the user to be logged in, and that the multiparty
middleware is added just for the URL assigned to file uploads (which would just slow the rest of the routes with no reason).
A similarity between all the controllers is that they tend to be slim and delegate the business logic to the models.
The application manages users and files, so we need to create models for both. Since the users will be saved to the database, we will work with Mongoose and create a new schema. The files will be saved to disk, so we will create a file prototype that we can reuse.
The file model is a class that takes the user ID and the filename as parameters in the constructor and sets the file path automatically. Some basic validation is performed before saving the file to ensure that it only contains letters, numbers, or the underscore character. Each file is persisted to disk in a folder named after userId
(generated by Mongoose). The methods used to interact with the filesystem use the native Node.js fs
module. The first part of the code is as follows:
var fs = require('fs'); var async = require('async'); var ROOT = __dirname + '/../files'; var path = require('path'); function File(userId, name) { this.userId = userId; this.name = name; this.path = this._getPath(); } File.prototype._getPath = function() { return path.resolve(File.getUserPath(this.userId) + '/' + this.name); }; File.prototype.isValidFileName = function() { return /[a-z0-9_]/i.test(this.name); }; File.prototype.exists = function(callback) { if (!this.isValidFileName()) { // keep the function async return process.nextTick(function() { callback(null, false) }); } fs.exists(this.path, callback); }; File.prototype.delete = function(callback) { this.exists((function(exists) { if (!exists) { return callback(); } fs.unlink(this.path, callback); }).bind(this)); }; File.prototype.getStats = function(callback) { fs.stat(this.path, callback); }; File.getUserPath = function(userId) { return ROOT + '/' + userId; }; // create a folder if it doesn't exist already File.createFolder = function(userId, callback) { var userPath = File.getUserPath(userId); fs.exists(userPath, function(exists) { if (!exists) { fs.mkdir(userPath, callback); } }); };
The most interesting methods in this model are the ones used to save a file and get all the files that belong to a user. When uploading a file, the multiparty
module saves it at a temporary location, and we need to move it to the user's folder. We solve this by piping readStream
into writeStream
and executing the callback on the close
event of the latter. The method to save a file should look like the following:
File.prototype.save = function(tempPath, callback) { if (!this.isValidFileName()) { return process.nextTick(function() { callback(null, new Error('Invalid filename')) }); } var readStream = fs.createReadStream(tempPath); var writeStream = fs.createWriteStream(this.path); // if an error occurs invoke the callback with an error param readStream.on('error', callback); writeStream.on('error', callback); writeStream.on('close', callback); readStream.pipe(writeStream); };
The function that retrieves all the files of a user reads the directory to get the files, then it calls the getStats
function in parallel for every file to get its stats, and finally, it executes the callback once everything is done. In case there is an error returned because the user's folder does not exist, we call the File.createFolder()
method to create it:
File.getByUserId = function(userId, callback) { var getFiles = function(files) { if (!files) { return callback(null, []); } // get the stats for every file async.map(files, function(name, done) { var file = new File(userId, name); file.getStats(function(err, stats) { if (err) { return done(err); } done(null, { name: name, stats: stats }); }); }, callback); }; fs.readdir(File.getUserPath(userId), function(err, files) { if (err && err.code === 'ENOENT') { File.createFolder(userId, function(err) { if (err) { return callback(err); } getFiles(files); }); } else if (!err) { getFiles(files); } else { return callback(err); } }); };
The only things that we need to store in the database are the users, so the user.js
file contains the Mongoose schema for the User model, field validation functions, and functions related to hashing and comparing passwords (for authentication). The following code contains the module dependencies along with the validation functions and schema declaration:
var mongoose = require('mongoose'); var pass = require('pwd'); var validateUser = function(username) { return !!(username && /^[a-z][a-z0-9_-]{3,15}$/i.test(username)); }; var validatePassword = function(pass) { return !!(pass && pass.length > 5); }; var User = new mongoose.Schema({ username: { type: String, validate: validateUser, unique: true }, salt: String, hash: String }, { safe: true });
Since we don't store the password in plain text but use a salt and a hash instead, we cannot add password
as a field on the schema (in order to enforce its validation rules) nor create a virtual setter for it (because the hashing function is asynchronous). Due to this, we need to create custom functions such as setPassword
, saveWithPassword
, and validateAll
as shown in the following code:
User.methods.setPassword = function(password, callback) { pass.hash(password, (function(err, salt, hash) { if (err) { return callback(err); } this.hash = hash; this.salt = salt; callback(); }).bind(this)); }; // validate schema properties (username) && password User.methods.validateAll = function(props, callback) { this.validate((function(err) { if (err) { return callback(err); } if (!validatePassword(props.password)) { return callback(new Error('ValidationError: invalid password')); } return callback(); }).bind(this)); }; User.methods.saveWithPassword = function(password, callback) { this.validateAll({ password: password }, (function(err) { if (err) { return callback(err); } this.setPassword(password, (function(err) { if (err) { return callback(err); } this.save(callback); }).bind(this)); }).bind(this)); };
The authentication function is pretty straightforward; it gets the username and then compares the hash stored in the database with the hash generated by the password, which is sent as a parameter:
User.statics.authenticate = function(username, password, callback) { // no call to database for invalid username/password if (!validateUser(username) || !validatePassword(password)) { // keep this function async in all situations return process.nextTick(function() { callback(null, false) }); } this.findOne({ username: username }, function(err, user) { if (err) { return callback(err); } // no such user in the database if (!user) { return callback(null, false); } pass.hash(password, user.salt, function(err, hash) { if (err) { return callback(err); } // if the auth was successful return the user details return (user.hash === hash) ? callback(null, user) : callback(null, false); }); }); }; module.exports = mongoose.model('User', User);
The first thing to do here is to create a global layout for our application, since we want to reuse the header and footer and only customize the unique part of every web page. We use jade
as the templating language, so in order to declare the extendable part of the layout, we use the block
function. The layout.jade
file will be created inside the views
folder as follows:
!!! 5 html head title File Store link(rel='stylesheet', href='http://fonts.googleapis.com/css?family=IM+Fell+Great+Primer') link(rel='stylesheet', href='/stylesheets/normalize.css', type='text/css') link(rel='stylesheet', href='/stylesheets/style.css', type='text/css') body header h1 File Store if isLoggedIn div form(action='/sessions', method='POST') input(type='hidden', name='_method', value='DELETE') input(type='submit', class='sign-out', value='Sign out') div.container block content script(src='http://code.jquery.com/jquery-1.10.1.min.js') script(src='/javascripts/file-upload.js')
Note
An interesting detail in the preceding code is that we override the method interpreted on the server side from POST
to DELETE
by passing a hidden field called _method
. This functionality is provided by the methodOverride
middleware of Express, which we included in the app.js
file.
Sometimes, we need to use functions for date formatting and size formatting or as a link to use some parameters and other similar tasks. This is where view helpers come in handy. In our application, we want to display the size of the files in kilobytes, so we need to create a view helper that will convert the size of a file from bytes to kilobytes. We can replicate the same structure from the routes
folder for the helpers as well, which means that we will have an index.js
file that will export everything as an object. Besides this, we will only create the helper for the files at the moment, namely files.js
, since that's all we need:
exports.formatSize = function(sizeInBytes) { return (sizeInBytes / 1024).toFixed(2) + ' kb'; };
To make the view helpers accessible inside the view, we need to add another piece of code into our app.js
main file after the view setup, as shown in the following line of code:
app.locals = require('./helpers/index');
This will ensure that whatever is assigned to the locals
property is globally accessible in every view file.
In the views
folder, we create subfolders for files, sessions, and users. The sessions
and users
folders will contain a new.jade
file, each with a form (user login and signup page). The biggest view file from the files subfolder is index.jade
since it's the most important page of the application. The page will contain dynamic data such as the logged-in username or the number of files stored and other stuff such as an upload form and a dashboard with a list of files. The code for the index.jade
file will look like the following:
extends ../layout block content h2 Hello #{username} if !files.length h3 You don't have any files stored! else h3 You have #{files.length} files stored! if info p.notification.info= info if error p.notification.error= error div#upload-form form(action='/files', method='POST', enctype="multipart/form-data") div.browse-file input(type='text', id='fake-upload-box', placeholder='Upload new file!') input(type='file', name='file') button(type='submit') Go! if files.length table.file-list thead tr th Name th Size th Delete tbody each file in files tr td a(href="/files/#{encodeURIComponent(file.name)}") #{file.name} td #{helpers.files.formatSize(file.stats.size)} td form(action="/files/#{encodeURIComponent(file.name)}", method='POST') input(type='hidden', name="_method", value='DELETE') input(type='submit', value='delete')
We have not covered the JavaScript static files or stylesheets used by the application, but you can fill in the missing pieces by yourself as an exercise or just copy the example code provided with the book.
To run the application, you need to have Node and NPM installed and MongoDB up and running, and then execute the following commands in the terminal from the project root:
$ npm install . $ npm start
The first command will install all the dependencies and the second one will start the application. You can now visit http://localhost:3000/
and see the live demo!