One of the most frequently asked questions by newcomers to Express is how to structure an application. There is no definitive answer for this, and we may choose different solutions based on how big our application is or what problem we are trying to tackle. Luckily for us, Express is easy to customize, and we can apply whatever structure we deem necessary.
If the code base is small, we can include everything into a few files or even a single one. This might be the case when exposing a database over HTTP (such as LevelDB and PouchDB) and creating mountable applications (these tend to be small and solve a specific problem) or other small applications.
When dealing with medium and large projects, the best thing to do is to split them into smaller pieces, making them easier to debug and test. If there are parts of the application that can be reused for other projects, the best thing to do is to move them into their separate repository.
An interesting technique to structure an application is to group files by the features they provide instead of grouping them by their function. In MVC, the controllers, models, and views live inside their own folder; however, with this approach, we have folders that group files with the same role. For example, consider the following folders:
One could go even further and choose to include things such as tests and static assets that belong to a feature inside its folder.
If there's something that can be reused for multiple features such as the general layout or models, we can group them inside their own folder. Each of these folders can export an Express application with its own view engine, middleware, and other customizations. These folders can reside in a parent lib folder, for example. We will then require them in the main app.js file like we would any regular middleware. It's a good way to separate concerns, although they are not necessarily complete, independent pieces because they rely on application-specific logic.
An advantage this structure offers is that when we are working on a certain section of an application, all the files that need to be created/edited are in the same location, so there's no need to switch between controllers, models, and views like with MVC.
It's worth mentioning that the creator of Express, TJ Holowaychuk, recommends this approach for larger applications instead of MVC.
The most common technique to structure web applications with Express is MVC. When generating a project using the Express CLI, it almost provides an MVC structure, omitting the models folder. The following screenshot lists all the files and folders generated for a sample application using the CLI tool:
The package.json file is automatically populated with the name of the application, the dependencies, the private attribute, and the starting script. This starting script is named app.js and loads all the middleware, assigns the route handlers, and starts the server. There are three folders in the root:
Apart from these existing folders and the models folder, which we need to create ourselves, we might also create folders for tests, logs, or configuration. The best thing about this structure is that it's easy to get started with and is known to most developers.
Let's apply the theory in practice now and create an MVC file manager application using Express 4.x and Mongoose (an object modeling library for MongoDB). The application should allow users to register and log in and enable them to view, upload, and delete their files.
We will start by creating the folder structure. First, we'll use the Express CLI tool in the terminal to create the boilerplate. Apart from the public, routes, and views folders, we also need to add folders for models, helpers (view helpers), files (the files uploaded by users will be stored in subfolders here), and lib (used for internal app libraries):
$ express FileManager $ cd FileManager $ mkdir {models,helpers,files,lib}
By default, the CLI tool will create two dependencies in your package.json file—express and jade—but it won't install them, so we need to manually execute the following install command:
$ npm install .
In addition to these two modules, we also need to install mongoose to interact with MongoDB, async for control flow, pwd to hash and compare passwords, connect-flash to store messages for the user (which are then cleared after being displayed), and connect-multiparty to handle file uploads. We can use the following shortcut to install the packages and have them declared in package.json at the same time if we call NPM with the –save flag:
$ npm install –save mongoose async pwd connect-flash connect-multiparty
Express 3.x came bundled with the Connect middleware, but that's not the case in the 4.x version, so we need to install them separately using the following command:
$ npm install –save morgan cookie-parser cookie-session body-parser method-override errorhandler
Note
The middleware libraries from Connect were extracted into their separate repos, so starting with Express 4.x, we need to install them separately. Read more about this topic on the Connect GitHub page at https://github.com/senchalabs/connect#middleware.
We can always check what modules are installed by entering the following command in the terminal at the root of our project:
$ npm ls
That command will output a tree with the dependencies.
Note
It's worth noting that the versions for the dependencies listed in the package.json file will not be exact when we use the –save flag; instead, they will be using the default npm semver range operator. You can read more from the official
npm documentation (https://www.npmjs.org/doc/cli/npm-install.html) and the node-semver page (https://www.npmjs.org/package/semver).
We can get as inventive as we want with the configuration parameters of a project, like have multiple subfolders based on the environment or hierarchical configuration, but for this simple application, it's enough to have a single config.json file. The configuration variables we need to define in this file are the MongoDB database URL, the application port, the session secret key, and its maximum age so that our file will look like the following code:
{
"mongoUrl": "mongodb://localhost/filestore",
"port": 3000,
"sessionSecret": "random chars here",
"sessionMaxAge": 3600000
}Tip
Downloading the example code
You can download the example code files for all Packt books you have purchased from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.
You can also download the example code files for the book from GitHub: https://github.com/alessioalex/mastering_express_code.
In the main file of the application, named app.js, we handle the view setup, load the middleware required for the project, connect to the database, and bind the Express application to a port. Later on, we modify this file to set up the route handling as well, but at the moment, the file contains the following code:
// Module dependencies
var express = require('express');
var app = express();
var morgan = require('morgan');
var flash = require('connect-flash');
var multiparty = require('connect-multiparty');
var cookieParser = require('cookie-parser');
var cookieSession = require('cookie-session');
var bodyParser = require('body-parser');
var methodOverride = require('method-override');
var errorHandler = require('errorhandler');
var config = require('./config.json');
var routes = require('./routes');
var db = require('./lib/db');
// View setup
app.set('view engine', 'jade');
app.set('views', __dirname + '/views');
app.locals = require('./helpers/index');
// Loading middleware
app.use(morgan('dev'));
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));
app.use(methodOverride(function(req, res){
if (req.body && typeof req.body === 'object' && '_method' in req.body) {
// look in url - encoded POST bodies and delete it
var method = req.body._method;
delete req.body._method;
return method;
}
}));
app.use(cookieParser());
app.use(cookieSession({
secret: config.sessionSecret,
cookie: {
maxAge: config.sessionMaxAge
}
}));
app.use(flash());
if (app.get('env') === 'development') {
app.use(errorHandler());
}
// static middleware after the routes
app.use(express.static(__dirname + '/public'));
// Establishing database connection and binding application to specified port
db.connect();
app.listen(config.port);
console.log('listening on port %s', config.port);Note that the preceding app.js file contains the code for the database connection. Later on, we will need other database-related functions such as checking for failed data validation, duplicate keys, or other specific errors. We can group this logic into a separate file called db.js inside the lib folder and move the connection functionality there as well, as shown in the following code:
var mongoose = require('mongoose');
var config = require('../config.json');
exports.isValidationError = function(err) {
return ((err.name === 'ValidationError')
|| (err.message.indexOf('ValidationError') !== -1));
};
exports.isDuplicateKeyError = function(err) {
return (err.message.indexOf('duplicate key') !== -1);
};
exports.connect = /* database connection function extracted from app.js should move here */The routes folder will have a file for each controller (files.js, users.js, and sessions.js), another file for the application controller (main.js), and an index.js file that will export an object with the controllers as properties, so we don't have to require every single route in app.js.
The users.js file contains two functions: one to display the user registration page and another to create a user and its subfolder inside /files, as shown in the following code:
var User = require('../models/user');
var File = require('../models/file');
var db = require('../lib/db');
exports.new = function(req, res, next) {
res.render('users/new', {
error: req.flash('error')[0]
});
};
exports.create = function(req, res, next) {
var user = new User({ username: req.body.username });
user.saveWithPassword(req.body.password, function(err) {
if (err) {
if (db.isValidationError(err)) {
req.flash('error', 'Invalid username/password');
return res.redirect('/users/new');
} else if (db.isDuplicateKeyError(err)) {
req.flash('error', 'Username already exists');
return res.redirect('/users/new');
} else {
return next(err);
}
}
File.createFolder(user._id, function(err) {
if (err) { return next(err); }
req.flash('info', 'Username created, you can now log in!');
res.redirect('/sessions/new');
});
});
};The sessions.js file handles user authentication and sign out as well as renders the login page. When the user logs in successfully, the username and userId properties are populated on the session object and deleted on sign out:
var User = require('../models/user');
exports.new = function(req, res, next) {
res.render('sessions/new', {
info: req.flash('info')[0],
error: req.flash('error')[0]
});
};
exports.create = function(req, res, next) {
User.authenticate(req.body.username, req.body.password, function(err, userData) {
if (err) { return next(err); }
if (userData !== false) {
req.session.username = userData.username;
req.session.userId = userData._id;
res.redirect('/');
} else {
req.flash('error', 'Bad username/password');
res.redirect('/sessions/new');
}
});
};
exports.destroy = function(req, res, next) {
delete req.session.username;
delete req.session.userId;
req.flash('info', 'You have successfully logged out');
res.redirect('/sessions/new');
};The files.js controller performs CRUD-type operations; it displays all the files or a specific file for the logged-in user and saves the files or deletes them. We use res.sendfile to display individual files because it automatically sets the correct content type and handles the streaming for us. Since the bodyParser middleware from Express was deprecated, we replaced it with connect-multiparty (a connect wrapper around the multiparty module), one of the recommended alternatives. Luckily, this module has an API similar to bodyParser, so we won't notice any differences. Check out the complete source code of files.js as follows:
var File = require('../models/file');
exports.index = function(req, res, next) {
File.getByUserId(req.session.userId, function(err, files) {
if (err) { return next(err); }
res.render('files/index', {
username: req.session.username,
files: files,
info: req.flash('info')[0],
error: req.flash('error')[0]
});
});
};
exports.show = function(req, res, next) {
var file = new File(req.session.userId, req.params.file);
file.exists(function(exists) {
if (!exists) { return res.send(404, 'Page Not Found'); }
res.sendfile(file.path);
});
};
exports.destroy = function(req, res, next) {
var file = new File(req.session.userId, req.params.file);
file.delete(function(err) {
if (err) { return next(err); }
req.flash('info', 'File successfully deleted!');
res.redirect('/');
});
};
exports.create = function(req, res, next) {
if (!req.files.file || (req.files.file.size === 0)) {
req.flash('error', 'No file selected!');
return res.redirect('/');
}
var file = new File(req.session.userId, req.files.file.originalFilename);
file.save(req.files.file.path, function(err) {
if (err) { return next(err); }
req.flash('info', 'File successfully uploaded!');
res.redirect('/');
});
}; The general routes used to require user authentication or other middleware that needs to be reused for different paths can be put inside main.js, as shown in the following code:
exports.requireUserAuth = function(req, res, next) {
// redirect user to login page if they're not logged in
if (!req.session.username) {
return res.redirect('/sessions/new');
}
// needed in the layout for displaying the logout button
res.locals.isLoggedIn = true;
next();
};The index.js file is pretty simple; it just exports all the controllers into a single object so they're easier to require in the start script of our application:
exports.main = require('./main');
exports.users = require('./users');
exports.sessions = require('./sessions');
exports.files = require('./files');Now that we have seen what the controllers look like, we can add them to our existing app.js file:
var routes = require('./routes');
// Declaring application routes
app.get('/', routes.main.requireUserAuth, routes.files.index);
app.get('/files/:file', routes.main.requireUserAuth, routes.files.show);
app.del('/files/:file', routes.main.requireUserAuth, routes.files.destroy);
app.post('/files', multiparty(), routes.main.requireUserAuth, routes.files.create);
app.get('/users/new', routes.users.new);
app.post('/users', routes.users.create);
app.get('/sessions/new', routes.sessions.new);
app.post('/sessions', routes.sessions.create);
app.del('/sessions', routes.sessions.destroy);Note that we included the requireUserAuth route for all the URLs that need the user to be logged in, and that the multiparty middleware is added just for the URL assigned to file uploads (which would just slow the rest of the routes with no reason).
A similarity between all the controllers is that they tend to be slim and delegate the business logic to the models.
The application manages users and files, so we need to create models for both. Since the users will be saved to the database, we will work with Mongoose and create a new schema. The files will be saved to disk, so we will create a file prototype that we can reuse.
The file model is a class that takes the user ID and the filename as parameters in the constructor and sets the file path automatically. Some basic validation is performed before saving the file to ensure that it only contains letters, numbers, or the underscore character. Each file is persisted to disk in a folder named after userId (generated by Mongoose). The methods used to interact with the filesystem use the native Node.js fs module. The first part of the code is as follows:
var fs = require('fs');
var async = require('async');
var ROOT = __dirname + '/../files';
var path = require('path');
function File(userId, name) {
this.userId = userId;
this.name = name;
this.path = this._getPath();
}
File.prototype._getPath = function() {
return path.resolve(File.getUserPath(this.userId) + '/' + this.name);
};
File.prototype.isValidFileName = function() {
return /[a-z0-9_]/i.test(this.name);
};
File.prototype.exists = function(callback) {
if (!this.isValidFileName()) {
// keep the function async
return process.nextTick(function() { callback(null, false) });
}
fs.exists(this.path, callback);
};
File.prototype.delete = function(callback) {
this.exists((function(exists) {
if (!exists) { return callback(); }
fs.unlink(this.path, callback);
}).bind(this));
};
File.prototype.getStats = function(callback) {
fs.stat(this.path, callback);
};
File.getUserPath = function(userId) {
return ROOT + '/' + userId;
};
// create a folder if it doesn't exist already
File.createFolder = function(userId, callback) {
var userPath = File.getUserPath(userId);
fs.exists(userPath, function(exists) {
if (!exists) {
fs.mkdir(userPath, callback);
}
});
};The most interesting methods in this model are the ones used to save a file and get all the files that belong to a user. When uploading a file, the multiparty module saves it at a temporary location, and we need to move it to the user's folder. We solve this by piping readStream into writeStream and executing the callback on the close event of the latter. The method to save a file should look like the following:
File.prototype.save = function(tempPath, callback) {
if (!this.isValidFileName()) {
return process.nextTick(function() {
callback(null, new Error('Invalid filename'))
});
}
var readStream = fs.createReadStream(tempPath);
var writeStream = fs.createWriteStream(this.path);
// if an error occurs invoke the callback with an error param
readStream.on('error', callback);
writeStream.on('error', callback);
writeStream.on('close', callback);
readStream.pipe(writeStream);
};The function that retrieves all the files of a user reads the directory to get the files, then it calls the getStats function in parallel for every file to get its stats, and finally, it executes the callback once everything is done. In case there is an error returned because the user's folder does not exist, we call the File.createFolder() method to create it:
File.getByUserId = function(userId, callback) {
var getFiles = function(files) {
if (!files) { return callback(null, []); }
// get the stats for every file
async.map(files, function(name, done) {
var file = new File(userId, name);
file.getStats(function(err, stats) {
if (err) { return done(err); }
done(null, {
name: name,
stats: stats
});
});
}, callback);
};
fs.readdir(File.getUserPath(userId), function(err, files) {
if (err && err.code === 'ENOENT') {
File.createFolder(userId, function(err) {
if (err) { return callback(err); }
getFiles(files);
});
} else if (!err) {
getFiles(files);
} else {
return callback(err);
}
});
};The only things that we need to store in the database are the users, so the user.js file contains the Mongoose schema for the User model, field validation functions, and functions related to hashing and comparing passwords (for authentication). The following code contains the module dependencies along with the validation functions and schema declaration:
var mongoose = require('mongoose');
var pass = require('pwd');
var validateUser = function(username) {
return !!(username && /^[a-z][a-z0-9_-]{3,15}$/i.test(username));
};
var validatePassword = function(pass) {
return !!(pass && pass.length > 5);
};
var User = new mongoose.Schema({
username: {
type: String,
validate: validateUser,
unique: true
},
salt: String,
hash: String
}, {
safe: true
});Since we don't store the password in plain text but use a salt and a hash instead, we cannot add password as a field on the schema (in order to enforce its validation rules) nor create a virtual setter for it (because the hashing function is asynchronous). Due to this, we need to create custom functions such as setPassword, saveWithPassword, and validateAll as shown in the following code:
User.methods.setPassword = function(password, callback) {
pass.hash(password, (function(err, salt, hash) {
if (err) { return callback(err); }
this.hash = hash;
this.salt = salt;
callback();
}).bind(this));
};
// validate schema properties (username) && password
User.methods.validateAll = function(props, callback) {
this.validate((function(err) {
if (err) { return callback(err); }
if (!validatePassword(props.password)) {
return callback(new Error('ValidationError: invalid password'));
}
return callback();
}).bind(this));
};
User.methods.saveWithPassword = function(password, callback) {
this.validateAll({ password: password }, (function(err) {
if (err) { return callback(err); }
this.setPassword(password, (function(err) {
if (err) { return callback(err); }
this.save(callback);
}).bind(this));
}).bind(this));
};The authentication function is pretty straightforward; it gets the username and then compares the hash stored in the database with the hash generated by the password, which is sent as a parameter:
User.statics.authenticate = function(username, password, callback) {
// no call to database for invalid username/password
if (!validateUser(username) || !validatePassword(password)) {
// keep this function async in all situations
return process.nextTick(function() { callback(null, false) });
}
this.findOne({ username: username }, function(err, user) {
if (err) { return callback(err); }
// no such user in the database
if (!user) { return callback(null, false); }
pass.hash(password, user.salt, function(err, hash) {
if (err) { return callback(err); }
// if the auth was successful return the user details
return (user.hash === hash) ? callback(null, user) : callback(null, false);
});
});
};
module.exports = mongoose.model('User', User);The first thing to do here is to create a global layout for our application, since we want to reuse the header and footer and only customize the unique part of every web page. We use jade as the templating language, so in order to declare the extendable part of the layout, we use the block function. The layout.jade file will be created inside the views folder as follows:
!!! 5
html
head
title File Store
link(rel='stylesheet', href='http://fonts.googleapis.com/css?family=IM+Fell+Great+Primer')
link(rel='stylesheet', href='/stylesheets/normalize.css', type='text/css')
link(rel='stylesheet', href='/stylesheets/style.css', type='text/css')
body
header
h1 File Store
if isLoggedIn
div
form(action='/sessions', method='POST')
input(type='hidden', name='_method', value='DELETE')
input(type='submit', class='sign-out', value='Sign out')
div.container
block content
script(src='http://code.jquery.com/jquery-1.10.1.min.js')
script(src='/javascripts/file-upload.js')Note
An interesting detail in the preceding code is that we override the method interpreted on the server side from POST to DELETE by passing a hidden field called _method. This functionality is provided by the methodOverride middleware of Express, which we included in the app.js file.
Sometimes, we need to use functions for date formatting and size formatting or as a link to use some parameters and other similar tasks. This is where view helpers come in handy. In our application, we want to display the size of the files in kilobytes, so we need to create a view helper that will convert the size of a file from bytes to kilobytes. We can replicate the same structure from the routes folder for the helpers as well, which means that we will have an index.js file that will export everything as an object. Besides this, we will only create the helper for the files at the moment, namely files.js, since that's all we need:
exports.formatSize = function(sizeInBytes) {
return (sizeInBytes / 1024).toFixed(2) + ' kb';
};To make the view helpers accessible inside the view, we need to add another piece of code into our app.js main file after the view setup, as shown in the following line of code:
app.locals = require('./helpers/index');This will ensure that whatever is assigned to the locals property is globally accessible in every view file.
In the views folder, we create subfolders for files, sessions, and users. The sessions and users folders will contain a new.jade file, each with a form (user login and signup page). The biggest view file from the files subfolder is index.jade since it's the most important page of the application. The page will contain dynamic data such as the logged-in username or the number of files stored and other stuff such as an upload form and a dashboard with a list of files. The code for the index.jade file will look like the following:
extends ../layout
block content
h2 Hello #{username}
if !files.length
h3 You don't have any files stored!
else
h3 You have #{files.length} files stored!
if info
p.notification.info= info
if error
p.notification.error= error
div#upload-form
form(action='/files', method='POST', enctype="multipart/form-data")
div.browse-file
input(type='text', id='fake-upload-box', placeholder='Upload new file!')
input(type='file', name='file')
button(type='submit') Go!
if files.length
table.file-list
thead
tr
th Name
th Size
th Delete
tbody
each file in files
tr
td
a(href="/files/#{encodeURIComponent(file.name)}") #{file.name}
td #{helpers.files.formatSize(file.stats.size)}
td
form(action="/files/#{encodeURIComponent(file.name)}", method='POST')
input(type='hidden', name="_method", value='DELETE')
input(type='submit', value='delete')We have not covered the JavaScript static files or stylesheets used by the application, but you can fill in the missing pieces by yourself as an exercise or just copy the example code provided with the book.
To run the application, you need to have Node and NPM installed and MongoDB up and running, and then execute the following commands in the terminal from the project root:
$ npm install . $ npm start
The first command will install all the dependencies and the second one will start the application. You can now visit http://localhost:3000/ and see the live demo!