In this chapter we will code some JavaScript and we will learn techniques to reuse the code in different files and modules. We will also create our first node module and then publish it on the standard npm registry.
During the course of this section we will try to create a node module named simplemath. It will be a simple library. We won't delve much into programming paradigm in node.js, or nuances of JavaScript. Our library will simply provide basic operations like addition, subtraction, multiplication, division, and a few convenient constants. Through building this module we will learn how to use npm to create and publish node modules. We will follow the test-driven development strategy to build this library. We will write failing tests before starting to code our application and then try to pass those tests.
To begin with consider the sum function from our simplemath library:
function sum(num1, num2) {
return num1 + num2;
}This is a simple JavaScript program. It is a function which expects two integer arguments and returns the sum of the two numbers. To make this function available to other files and programs, we will have to add only a single line of code on top of the file. So the code looks like the following code listing:
module.exports = sum;
function sum(num1, num2) {
return num1 + num2;
}The module.exports object has a very special meaning in node.js. We can assign any entity to it, and we will be able to access that entity from outside that file. Let's try it out in node.js REPL. For those who don't know, a REPL (Read Eval Print Loop) is an interactive command-line program that one can use to try out the language features. We can enter node.js REPL by entering node on the command line.
$ node > var someVar = require('./sum.js'); > someVar; [Function: sum]
After we arrive at node (>) prompt we call require() with path of the file sum.js as argument and attached the returned value to a variable named someVar. On inspecting the someVar variable we come to know that it is a function named sum. Let's find out what toString() of someVar gives us:
> someVar.toString(); 'function sum(num1, num2) {\n\t\treturn num1 + num2;\n}'
This is actually the complete body of the sum function that we've defined in our file. Finally, when we call the function with appropriate arguments:
> someVar(1,2) 3
What actually happened is that the function that we had assigned to module.exports in our sum.js file, got returned by the require('./sum.js') call and was assigned to someVar. Behind the scenes, the JavaScript code in sum.js is run immediately on calling require. Keep in mind though that the function itself isn't called but just returned. We will have to call the function to run the code inside that. Anything that we assigned to module.exports is returned by the require call.
One caveat to keep in mind is that assignment to module.exports has to be done immediately and not asynchronously. So for example, although the following code works:
(function(){
module.exports = {"key": "value"};
})();The following doesn't:
setTimeout(function(){
module.exports = {"key": "value"};
}, 0);So that was using require from command line. Note the action performed by require is similar to that of import in programming languages such as python and Java. It gives access to code written in a file to somewhere outside the file. Now let's use require in another file that contains node.js code.
var sum = require("./sum"); console.log( sum(1,2) );
The preceding code again loads the sum.js file (the .js extension is optional, and as we will see, so is .node and .json) and prints the result of calling sum(1,2) on standard output. Save the preceding code in another file. Name it tests.js and keep it in the same directory as sum.js. Let's run tests.js from command line.
$ node tests.js 3
Thus, we have successfully used code from one file into another.
Using require is not limited to importing functions. We can import any JavaScript entity, including but not limited to strings, numbers, and objects. The require function also supports importing a .json file (a file whose body is JSON). For example, we can have following two files:
foo.js
module.exports = { "foo": "bar"};foo.json
{ "foo": "bar" }Calling require with ./foo.js and ./foo.json will be equivalent and it will load a JavaScript object with a key named foo and its value equal to bar. The filename that we pass to require should be a valid file name with path. So we can also pass paths which are outside the directory like require("../filename"). From node.js documentation (http://nodejs.org/api/modules.html):
If the exact filename is not found, then node will attempt to load the required filename with the added extension of
.js,.json, and then.node..jsfiles are interpreted as JavaScript text files, and.jsonfiles are parsed asJSONtext files.A module prefixed with
/is an absolute path to the file. For example,require('/home/marco/foo.js')will load the file at/home/marco/foo.js.
Another use of require is to load the core modules or native modules. Core modules are modules that are baked into the source code of node.js. They are the APIs that we talked about in the first section or rather the APIs that constitute node.js itself. For example, following code will load the http module from node.js core:
var http = require("http");
Note that there is no leading slash or dot for loading a core module or any external module for that matter. It's also worth noting that if the namespace of two modules clashes, that is, if two modules have the same name, the core module will always trump the external module.
What we can do with node.js is limited to what is already implemented in underlying meta language, which is C++ in our case. The current version of node.js provides a wide range of implementations of general usage. But if one wants to, one can further extend node.js by writing what is known as native modules. node.js provides a way of wrapping C++ code around JavaScript API and thus extending the V8 runtime just as core modules of node.js like fs and http does. There might be other reasons to create C++ modules. For example, to use legacy code or to gain performance. One can publish and distribute these packages just like usual node modules, the default entry point for a native module is index.node file. Writing native modules requires knowledge of a number of C++ libraries. We won't be discussing native modules in this book, that is much beyond the scope but worth mentioning.
Libraries hardly fit into a single file. The code for the simplest of libraries spans at least a few files. It makes sense to keep logically separate pieces of code in separate files, that is the whole point of modularization, and using require in the first place. Imagine you had to code your complete library, which is around 1000 lines of JavaScript code in a single file. How awful would that be? Let's turn that awful into awesome.
If node.js is unable to find the file with the specified name/path, node will look for a directory with the same name. On finding a directory with that exact name, the node, by default will try to load index.js file from the root of that directory. We can override that default but, this file should be the entry point into our library and because of the way node require works, it should be the single entry point.
So for our simplemath library, we shall have the following folder structure:
-simplemath
index.js
tests.js
-lib
sum.js
subtract.js
multiply.js
divide.js
constants.jsWhenever a file or module is required, the result obtained is cached. Subsequent require calls to the same file/module will serve the results from the cache. Let's enquire upon it. From the simplemath folder:
$ cd simplemath $ node > var sm = require("./index.js")
We have loaded the object in a variable named sm. Let's inspect the require object:
$ require { [Function: require] resolve: [Function], main: undefined, extensions: { '.js': [Function], '.json': [Function], '.node': [Function] }, registerExtension: [Function], cache: { '/home/juzer/simplemath/index.js': { id: '/home/juzer/simplemath/index.js', exports: [Object], parent: [Object], filename: '/home/juzer/simplemath/index.js', loaded: true, children: [Object], paths: [Object] }, … }
Check out the object, especially the cache property. It lists all the files that were required across our library and its dependencies. This object itself is the cache. Let's try to retrieve the function sum from the cache:
> var sum = require.cache["/home/juzer/simplemath/lib/sum.js"].exports > sum [Function] sum(1,3) 4
Hence we could retrieve the sum function from the cache. In general a cached file can be retrieved from following reference:
require.cache["absolute-file-path"].exports
On each subsequent require, the import will be served from this location. Assigning some other value to the exports property will return that value instead. Hacking with the cache object can introduce weird bugs in your program. It's good to stay away from this for the most part.
One implication of caching is that the code inside a file is run only once. So you should make sure your .js files are not having any side effects. For example, incrementing a counter in the database to keep track of number of times a module was required or loaded in your code is not a good idea.
Another implication is that the same object is returned each time a require on a file or module is called. It's not a copy of the object, it is the same object. Hope you haven't closed your REPL yet. After performing the previous commands do the following:
$ sm.x = "some value..." $ sm { PI: 3.141592653589793, E: 2.718281828459045, sum: [Function], subtract: [Function], multiply: [Function], divide: [Function], gcd: [Function: gcd], x: 'some value...' } $ var sm2 = require("./index") $sm2 { PI: 3.141592653589793, E: 2.718281828459045, sum: [Function], subtract: [Function], multiply: [Function], divide: [Function], gcd: [Function: gcd], x: 'some value...' }
The x property in both the cases proves that, using the require function on the module two times, returned the same object.
In tests.js add the following code, which tests the API of our library:
/* assert is used frequently during testing. assert.equal checks if the two arguments passed to it are equal, If they aren't, an error is immediately thrown.
*/
var assert = require("assert");
//index.js is the entrypoint into our library
var simplemath = require("./index");
assert.equal(simplemath.sum(1,2), 3);
assert.equal(simplemath.subtract(10, 6), 4);
assert.equal(simplemath.multiply(3, 5), 15);
assert.equal(simplemath.divide(33, 3),11);
assert.equal(simplemath.PI,3.141592653589793);
console.info("All tests passed successfully!");Running this test will of course result in us staring at a very ugly stack trace:
$ node tests.js /home/juzer/Desktop/Instant NPM Handbook/simplemath/tests.js:4 assert(math.sum(1,2) === 3); ^ TypeError: Object #<Object> has no method 'sum' at Object.<anonymous> (/home/juzer/Desktop/Instant NPM Handbook/simplemath/tests.js:4:13) at Module._compile (module.js:456:26) at Object.Module._extensions..js (module.js:474:10) at Module.load (module.js:356:32) at Function.Module._load (module.js:312:12) at Module.runMain (module.js:497:10) at process._tickCallback (node.js:427:13)
Calling math.sum failed because we still haven't written any code and calling require on an empty file will result in an empty object being assigned to simplemath variable. We will fix these errors one by one.
Let's write the actual business logic that our module promises to solve. Let's add subtract, multiply, divide functions, and constants objects to our library.
Add the following code in the files mentioned:
simplemath/lib/subtract.js
module.exports = function(num1, num2) {
return num1 – num2;
};
simplemath/lib/multiply.js
module.exports = function(num1, num2) {
return num1 * num2;
};
simplemath/lib/divide.js
module.exports = function(num, denom) {
return num / denom;
};
simplemath/lib/constants.js
module.exports = {
E: Math.E,
PI: Math.PI
};The functions subtracts, multiplies, and divides the provided arguments respectively. The constants.js file provides constants like PI and E borrowed directly from the in-built math library. It is an overkill to create a separate file for each function. In a typical scenario, we would put the complete simplemath code in a single file. Imagine if simplemath was a huge library and each of the functions were large components, then we could've probably justified splitting it in multiple files. For the sake of simplicity and illustration we have chosen to make ourselves believe that each function is really a large separate component.
Similarly, it is unnecessary to assign constants provided by the math library into our own object like we did in constants.js. We can just use the math library directly. But for illustrating how we can export objects, we chose the preceding example.
Now let's look at simplemath/index.js and see how we can glue the different components together. As a best practice, this file should only contain glue code and not any logic.
simplemath/index.js
// Attach all the components
var simplemath = require("./lib/constants.js");
simplemath.sum = require("./lib/sum.js");
simplemath.subtract = require("./lib/subtract.js");
simplemath.multiply = require("./lib/multiply");
simplemath.divide = require("./lib/divide.js");
//Expose the simplemath object and provide the entrypoint
module.exports = simplemath;Now let's run the test cases again:
$ node tests.js All test cases passed successfully!
So here is our simplemath library. The code is modularized into different files and yet we are able to obtain a single object that exposes the complete API.
Every project has lots of metadata associated with it. For example, the name of the module, name of the author, links to official repository and issue tracking, dependencies, license, and so on. In a node module, all that information is stored inside package.json file. The package.json file lives in the root of the package folder.
Let us try to create a relevant package.json file for our library. npm is of great help here. The npm init command interactively creates package.json file for the project, with providing convenient defaults.
$ npm init This utility will walk you through creating a package.json file. It only covers the most common items, and tries to guess sane defaults. See npm help json for definitive documentation on these fields and exactly what they do. Use npm install <pkg> --save afterwards, to install a package and save it as a dependency in the package.json file. Press ^C at any time, to quit. name: (simplemath)
npm has started collecting information regarding the package. The first piece of information it is asking for is the package name. Note that it is important, this is the name by which our package will be found and identified on registry. npm has already suggested the name based on our folder name. Just hit the return key.
version: (0.0.0) 0.0.1
Next npm asks for version. This field too is important, npm keeps all the versions of a package and serves the requested package. Although npm is suggesting 0.0.0, but we have a working package so we will version it 0.0.1. After clicking on the Enter key.
description:
This is optional. npm is asking for a description of our package. Although it isn't required, it is a good practice to drop a one line description, for example, a simple math library.
entry point: (index.js)
Npm is asking for an entry point to our application. As we already know index.js is the default, but this is where we can override the default setting. Right now we shall go ahead with the default.
test command: node tests.js
This too is optional, npm is asking for a command that will run our tests. Putting node tests.js there, we will be able to run our tests by issuing the command npm test on command line.
git repository:
Open source projects are usually hosted on github or some other hosting provider. Here we can specify the git URL of our project. For now, we will skip it. Hit return without typing anything.
keywords: math mathematics simple
Whatever keywords we provide will be indexed by npm so that our package can be searched by those keywords. Give a space-separated list of words.
author: yourname <[email protected]>
Enter the name of the author, that is, your name, and optionally, e-mail ID in the given format.
license: (BSD)
Enter the license under which you want to publish your code. The default BSD is a good copyleft license that places very little restrictions upon users and promotes the spirit of open source.
About to write to ~/simplemath/package.json: { "name": "simplemath", "version": "0.0.1", "description": "A simple math library", "main": "index.js", "devDependencies": {}, "scripts": { "test": "test" }, "repository": "", "keywords": [ "math", "mathematics", "simple" ], "author": "yourname <[email protected]>", "license": "BSD" } Is this ok? (yes)
So here is our first package.json carrying precious information about our project. Hit enter, and hooray! A package.json file appears in our project root with the above contents written in it. npm will warn you that there is no README.md file in our package. It is a good practice to include a README file with detailed description of the package. If you want, include such a file in project root, but it is not required.
Simplemath/README.md
simplemath ======= A simple math library
This is the most basic package.json file for a project; specifying name, description, version, and author name of the project. There is a lot more that package.json can do to share project related information with npm and automate repetitive tasks.
We suddenly realize that for our purposes, we will also need a function in our simplemath library that calculates greatest common divisor (gcd) of two numbers. It will take some effort to write a well-tested gcd function, why not look around to see if someone has already implemented it. Go to https://npmjs.org and search for gcd. You will get scores of results. You may find lots of node modules solving the same problem. It is often difficult to choose between seemingly identical node modules. In such situations, check out the credentials of the developer(s) who are maintaining the project. Compare the number of times each module was downloaded by users. You can get this information on the package's page on npmjs at https://npmjs.org/package/<pkg-name>.
You can also check out the repository where the project is hosted, or the home page of the project. You will get this information on the npmjs home page of the module. If it isn't available, this probably isn't the module you want to use. If, however, it is available, check out the number of people who have starred the project on github, the number of people who have forked it, and active developers contributing to the project. Perhaps check out and run the test cases, or dive into the source code.
If you are betting heavily on a node module which isn't actively maintained by reputed developers, or which isn't well tested, you might be setting yourself up for a major rework in the future.
While we search for the gcd keyword on npmjs website we come to know that there is a node module named mathutils (https://npmjs.org/package/mathutils) that provides an implementation of gcd. We don't want to write our own implementation especially after knowing that someone somewhere in the node community has already solved that problem and published the JavaScript code. Now we want to be able to reuse that code from within our library.
Note
Note that this use case is a little contrived and it is an overkill to include an external library for such simple tasks as to calculate GCD, which is as a matter of fact, very few lines of code, and is popular enough to be found easily. It is used here for the purpose of illustration.
We can do so very easily. Again npm command line will help us reduce the number of steps.
$npm install mathutils --save
We have asked npm to install mathutils and the --save flag, in the end saves it as a dependency in our package.json file. So the mathutils library is downloaded in node_modules folder inside our project. Our new package.json file looks like this.
{
"name": "simplemath",
"version": "0.0.1",
"description": "A simple math library",
"main": "index.js",
"dependencies": {
"mathutils": "0.0.1"
},
"devDependencies": {},
"scripts": {
"test": "test"
},
"repository": "",
"keywords": [
"math",
"mathematics",
"simple"
],
"author": "yourname <[email protected]>",
"license": "BSD"
}And thus, mathutils is ready for us to use it as we please. Let's proceed to make use of it in our library.
Add the test case: Add the following code to test
gcdfunction to the end oftests.jsfile but beforeconsole.info.assert.equal( simplemath.gcd(12, 8), 4 ); console.log("GCD works correctly");Glue the
gcdfunction frommathutilstosimplemathinindex.js.var mathutils = require("mathutils"); to load mathutils? var simplemath = require("./lib/constants.js"); simplemath.sum = require("./lib/sum.js"); simplemath.subtract = require("./lib/subtract .js"); simplemath.multiply = require("./lib/multiply.js"); simplemath.divide = require("./lib/divide.js"); simplemath.gcd = mathutils.gcd; // Assign gcd module.exports = simplemath;
We have imported the mathutil library in our index.js and assigned the gcd function from the mathutil library to simplemath property with the same name. Let's test it out. Since our package.json is aware of the test script, we can delegate it to npm.
$ npm test … All tests passed successfully
Thus we have successfully added a dependency to our project.
We do not want to litter our node.js application directory with code from external libraries or packages that we want to use, npm provides a way of keeping our application code and third party libraries or node modules into separate directories. That is why the node_modules folder. Code for any third-party modules will go into this folder. From node.js documentation (http://nodejs.org/api/modules.html):
If the module identifier passed to
require()is not a native module, and does not begin with'/','../', or'./', then node starts at the parent directory of the current module, and adds/node_modules, and attempts to load the module from that location.If it is not found there, then it moves to the parent directory, and so on, until the root of the tree is reached.
For example, if the file at
'/home/ry/projects/foo.js'calledrequire('bar.js'), then node would look in the following locations, in this order:
/home/ry/projects/node_modules/bar.js
/home/ry/node_modules/bar.js
/home/node_modules/bar.js
/node_modules/bar.js
This allows programs to localize their dependencies, so that they do not clash.
Whenever we run the npm install command, the packages get stored into the node_modules folder inside the directory in which the command was issued.
Each module might have its own set of dependencies, which are then installed inside node_modules folder of that module. So in effect we obtain a dependency tree with each module having its dependencies installed in its own folder. Imagine two modules, on which your code is dependent, uses a different version of a third module. Having dependencies installed in their own folders, and the fact that require will look into the innermost node_modules folder first, affords a kind of safety that very few platforms are able to provide. Each module can have its own version of the same dependency. Thus node.js tactfully avoids dependency-hell which most of its peers haven't been able to do so far.
We have finished creating the simplemath node module Version 0.0.1 (denoted as [email protected]) and we feel quite accomplished with it. Next thing is, we want to share it with the world. We want to make it available to anyone who could benefit from it, just as we could easily obtain and use mathutils package. Our package should be downloadable by anyone with a valid node.js installation by issuing npm install simplemath command.
If this is the first time you are publishing a package, you will need to register yourself on the npm registry first. Here too, npm command line will be of great help to us. Whether or not you are already registered on npmjs, the steps to login through command line is pretty much the same.
$ npm login Username:
If you are already logged in or if you have logged into npm from your computer before, npm may suggest the username. Choose a unique username for yourself.
Password:
Enter a password and be sure about it, because the command-line utility doesn't ask you to confirm the password.
Email:
Enter your e-mail ID and hit return. If everything went well you will see the following logs on your console.
npm http PUT https://registry.npmjs.org/-\/user/org.couchdb.user:username npm http 201 https://registry.npmjs.org/-\/user/org.couchdb.user:username
Npm just called a REST APIs provided by the npm registry and registered you as a user. You can login with the same credentials on https://npmjs.org/login. If you are already a registered user you will see a slightly different log. Anyhow you should be successfully logged into the npm registry through npm command-line. Just make sure by doing the following:
$ npm whoami
And you should see your username printed on the console. Finally, lets proceed to publish our code
Before proceeding to publish our module I want to share a word of caution. npm registry is a free service that hosts node modules and thousands of developers use it. Namespace is a big problem on npmjs. Only one node module can be published under a name. It is not helpful to publish packages that will not be useful to a fair number of developers. It is just wasting away the namespace. We would rather see the simplemath namespace to be taken by some library that can provide non-trivial code to the users.
Although we will publish our package to learn the process, we will make sure that we don't take up a namespace that could accommodate a more robust package with which some non-trivial programs could be built. So rename your package such that it indicates that this is a trivial module. In package.json precede the package name with x- followed by your username and a hyphen. So it should look like this
{
name: x-username-simplemath
…
}now issue npm publish from inside the simplemath folder.
$ npm publish npm http PUT https://registry.npmjs.org/x-username-simplemath npm http 201 https://registry.npmjs.org/x-username-simplemath npm http GET https://registry.npmjs.org/x-username-simplemath npm http 200 https://registry.npmjs.org/x-username-simplemath npm http PUT https://registry.npmjs.org/x-username-simplemath/0.0.1/-\tag/latest npm http 201 https://registry.npmjs.org/x-username-simplemath/0.0.1/-\tag/latest npm http GET https://registry.npmjs.org/x-username-simplemath npm http 200 https://registry.npmjs.org/x-username-simplemath npm http PUT https://registry.npmjs.org/x-username-simplemath/-/x-username-simplemath-0.0.1.tgz/-rev/2-0c70b5d837c74d36d4e568b803c2896f npm http 201 https://registry.npmjs.org/x-username-simplemath/-/x-username-simplemath-0.0.1.tgz/-rev/2-0c70b5d837c74d36d4e568b803c2896f + [email protected]
You have successfully published your first node module. To test it, go to a random folder in your computer, or on a different computer altogether, which has a valid node.js installation and run:
$ npm install x-username-simplemath
And you shall see your module get downloaded in node_modules folder in that directory.
Now, since we know that our package is not of use to many developers, we shall unpublish the module.
$ npm unpublish x-username-simplemath ... - x-username-simplemath
And that completes our node module development flow.