Book Image

Instant Node Package Manager

By : Juzer Ali
Book Image

Instant Node Package Manager

By: Juzer Ali

Overview of this book

Node.js is the new buzz in town and has a vibrant community.It ships with npm, which is a tool that helps automate various development, deployment, and installation of node modules, and makes the process much easier. Instant Node Package Module is a practical guide to authoring and distributing node modules, and also for creating and managing private, standalone node.js projects. Starting with getting node.js installed and running on a variety of platforms, we then proceed to learn about code modularization and achieving it in node.js. Picking up a practical example, we will then explore the whole process and life cycle of conceiving, writing, and publishing a node module. We will also learn how to exploit a toolsincluded with npm to automate various development and deployment steps. Later on, we will discover the power of npm and how its different features and configurations can save large amounts of time by providing a helping hand for automating repetitive tasks between writing and publishing your modules.
Table of Contents (8 chapters)

Top 10 features you need to know about


We created and published our very first node module in the previous section. In this section we will learn tips and tricks to use features provided by npm to our advantage. The npm command-line utility offers a range of functionalities to help smoothen the development and deployment of node modules. Let's find out how can we exploit them. Here are the 10 things you should know.

Before we proceed, remember the glossary term prefix.

Note

Prefix: {prefix} is the full path where node.js installation is kept on the system. On most of the systems, this is /usr/local. On Windows systems it is usually C:/Users/Username . So wherever you see the word prefix inside curly braces, think of this definition.

Tag

As we know that npm keeps multiple versions of a node module, we can associate a tag, which is a short name, to a particular version. For example, many popular node modules usually keep their latest stable version under a stable tag. npm automatically tags the most recent version of a node module as latest. Command for tagging is:

$ npm tag <package-name>@<version> [tag]

Remember we are tagging a specific version of a package with the tag name so specifying both package-name and version are mandatory. The specified version should have been published before tagging it on npm registry. Providing a tag name is optional, if not, present the npm defaults to latest.

Install

We have already seen npm install in action in the previous section. Let's revisit this command and see what we missed.

$ npm install

From inside a module directory which contains a valid package.json, installs all the packages listed as dependencies and devDependencies inside package.json, into the ./node_modules folder. If no package.json exists at current path, npm will exit with an error. The version of the package that will be installed, depends on as specified in package.json. The dependency package's names are specified as keys of dependencies, devDependencies, and optionalDependencies property and their values indicate version. The version follows semver semantics. * means latest available version. Some more are:

  • > or >=, that is, greater than or greater than or equal to the specified version

  • < or <=, less than or less than or equal to the specified version

  • A version range, for example, >=0.5 <=1.0

  • An exact version string, for example, 1.2.3-beta.

Take a look at the sample package.json:

{…
  name: "pacakgename",
  "version": "0.0.1",
  dependencies: {
    "oauth": "0.9.8",
    "express": "*",
    "mongoskin": ">=0.2.0",
    "lodash": "< 1.0.0-rc.3",
    "passport": ">0.1.0 <=0.1.15"
    ...
  }
  ...
}

In this project, exact Version 0.9.8 of the oauth package will be installed. Latest available express package will be installed. Latest mongoskin available after 0.2.0 (inclusive) will be installed. Lodash Version less than 1.0.0-rc.3 will be installed. Any passport version available between 0.1.0 (not including 0.1.0 itself) and 0.1.15 (inclusive) will be installed with recent version taking precedence over earlier versions.

To understand semver (semantic versioning) semantics, that is exactly how package versions are sorted or which version pattern is considered recent than another, checkout http://semver.org/ and https://npmjs.org/doc/semver.html.

Note: npm version semantics differ slightly from semver:

$ npm install <folder | tarball-file | tarball-url>

Pass to the argument either a folder name or a tarball name on filesystem containing a valid node module or a URL that serves such a tarball. This and subsequent commands accepts space separated list of dependencies to install in any combination:

$ npm install <package-name> [--save, --save-dev, --save-optional]

Looks up the latest version of <package-name> in the npm registry and installs it into the local node_modules directory.

Optionally provide --save, --save-dev, --save-optional switches to automatically update corresponding section in package.json. These switches are valid for subsequent install commands mentioned. devDependencies are dependencies which are required during development of the module but not while using it in production, for example, a testing framework, or assertion library which are required during development but are never used while actually running the code in production.

An optionalDependency is one without which a module should be able to run successfully. For example, the redis module comes with hiredis specified as an optional dependency. The hiredis module is a native module written in C++ that parses response from the redis server. If somehow hiredis could not be installed, redis uses JavaScript parser instead. So the redis module is able to function properly even without its optional dependency.

$ npm install <package-name>@<version | tag | version-range>

The preceding command installs the package with a given version number, tag, or version range. You can also specify version ranges, for example, $ npm install mocha@">0.2.0 <=0.3.0". As discussed previously, they will be resolved according to semver semantics.

$ npm install <git-url>

For users who use git for version control, which is very likely since most node.js developers use git, you can directly install a package by cloning its git repository. By default, latest commit to master branch is fetched. You can specify a particular commit hashtag or git tag by appending #<commit-hashtag | git-tag> in the URL. One reason you would want to use a git URL directly is when the code is not published on npm, for example, when it is not an open source module and is not available on npm registry. Although you can have your own private npm registry, that will come at a cost of some technical overhead. This is the simplest way of obtaining private code at the moment. The downside of using a git URL directly is that you lose semver versioning, and installation is usually slower than installing from npm repository directly. While using a git URL, it is advisable to use a tag rather than simply a git URL, as it gives some control over the code that comes in. If you simply save a git URL in your package.json, whatever committed latest to the branch specified (master by default) will be installed and that could be something broken. Tags are usually stable releases, which can be relied upon.

Related commands

The next command is issued from inside a project. It updates all the dependencies inside the project to the latest. Providing a list of package names or unique identifiers, it only updates those packages. Provide a --global or -g flag to update globally installed packages:

$ npm update [pkg...]

The next command uninstalls all the packages, list of package names, or package identifiers, provided:

$ npm uninstall [pkg..]

Binaries and global installs

A lot of software programs provide a command-line interface to use them. For example, npm itself is an excellent command-line program. Similarly, express and tower.js provide command-line helpers that create scaffoldings for web applications. The files containing the programs that run on a command from the command line are loosely referred to as binaries or executables.

To use binaries provided by external modules, express, for example, install them globally. Any package can be installed from npm registry globally by passing --global or -g command-line flag to the install command. For example:

$ npm install -g express

The previous command installs express globally. The executable will be available on the command line.

$ express myapp

The previous command will create a new folder myapp and create express scaffold for a minimal express web application.

To include an executable in a node module, it is required to include a file that can parse the command-line arguments and provide a command-line entry point to the application. The binary will probably call the programmatic API of the node module.

It is required to register the binary entry point in package.json. Take a look at express's reduced package.json.

{
  "name": "express",
  "description": "Sinatra inspired web development framework",
  "version": "3.0.1",
  …
  "bin": { "express": "./bin/express" },
  …
}

The package.json tells us that express keeps its binary in express/bin folder with the name express and if we care to look inside express's directory structure we will find that to be the case. Let's try to create a binary ourselves. Let's extend our simplemath library and provide a command-line API for just adding two numbers.

$ simplemath-add 2 3
5

First make package.json aware of a binary:

{
  "name": "simplemath",
  "version": "0.0.1",
  "description": "A simple math library",
  "main": "index.js",
  "dependencies": {
    "mathutils": "~0.0.1"
  },
  "devDependencies": {},
  "scripts": {
    "test": "node tests.js"
  },
  "repository": "",
  "keywords": [
    "math",
    "mathematics",
    "simple"
  ],
  "bin": {"simplemath-add": "./bin/simplemath"},
  "author": "yourname <[email protected]>",
  "license": "BSD"
}

The highlighted code in the preceding code snippet, as we will see, will expose the ./bin/simplemath-add file (binary) to the command line from the system's binary folder.

Now add a file ./bin/simplemath-add to the project with the following contents.

#!/usr/bin/env node

var simplemath = require("../index");	//Obtain simplemath object
var result;

// Parse the 2nd and 3rd arguments into integer values since that iswhat our
// program expects. Preceding an evaluation with "+" sign is a shorthand method
// of parsing the evaluation to a number.
var num1 = +process.argv[3];
var num2 = +process.argv[4];

// Do the math and print result
result = simplemath.sum(num1, num2);
console.log( result );

The first line in the previous program is called a shebang. In *nix environments, it is used to inform the command line to treat the rest of the file as binary of the given program. We tell it to treat our file as a node.js binary. Let us run the program and see. First provide execute permission to the file if you are on a *nix machine:

$ chmod u+x ./bin/simplemath-add
$ ./bin/simplemath-add sum 4 5
9

And it works perfectly well. But wait, why do we need to give the full path of our binary? We want to be able to access the command-line API globally, from any folder, without needing to type the full path. We need to install our package globally. After we publish this code to npm registry and then install simplemath from registry with a --global or simply -g flag, we will be able to do so. You can try it if you want.

You do not necessarily have to publish and install/update your package globally from npm registry, there is another way to do this in development environment. That is through linking, which is our next topic.

Related commands

The next command shows the folder in which globally installed module's binaries are kept:

$ npm --global bin

The next command will show path to binaries kept inside a local install:

$ npm bin

The next command updates all globally installed packages. If a list of package names or identifiers are mentioned, updates only them:

$ npm update --global [pkg...]

The next command uninstalls a globally installed package:

$ npm uninstall --global pkg

Linking

Often you want to use the latest, bleeding edge version of your node module, or someone else's module, which hasn't been published yet. One option is to keep the latest project folder into node_modules folder of another project. But that can quickly become taxing with deeply nested dependencies, or impossible to maintain if the node module is a circular dependency. Also imagine a case where we want to use the bleeding edge version of our node module in three different projects. Every time we make changes to our module, we will have to copy paste it in all three. npm has a better solution:

npm link
npm link <package-name>

Linking allows you to bootstrap a project as a dependency in another, so that the changes in one immediately can be used or tested in the other. To understand this let us again take the example of our simplemath node module. Imagine we are building another node module called advancedmath, that internally uses simplemath as a dependency. Perhaps we are building both modules simultaneously and we always want to use the latest version of simplemath in advancedmath.

So advancedmath will perhaps, have following folder structure:

- advancedmath
  + lib
     pacakge.json
     index.json
  - node_modules
  + simplemath

package.json of advancedmath should look like the following code snippet:

{
  "name": "advancedmath",
  "version": "0.0.0",
  "dependencies": {
    "simplemath": "*"
  }
}

As we have already discussed, we won't simply keep simplemath folder inside advancedmath/node_modules folder. There are better ways to keep ourselves sane. We will link the simplemath into advancedmath. You will require admin privileges on the computer, to do this, refer to the following commands:

$ cd path/to/simplemath
$ sudo npm link

The previous command links simplemath into a global installation and creates the simplemath binary that we have created in previous section, available globally. So we can do the following without publishing and installing our module globally from registry.

$ cd /some/random/folder
$ simplemath-add sum 1 3
4

Now let us bootstrap it to advancedmath, so that changes in simplemath are immediately reflected in advancedmath dependency of it.

$ cd path/to/advancedmath
$ npm link simplemath

The previous command has bootstrapped simplemath into advancedmath. It should be noted that any changes in binary would also affect the executable we are accessing from the command line.

If you are familiar with UNIX symlinks, it will probably interest you that a symlink to the globally installed simplemath folder is created inside advancedmath/node_modules directory. Hence any change is immediately reflected. Now we can independently develop the two modules without the advancedmath developer worrying about keeping abreast with simplemath, except for API changes of course. Notice that we didn't have to give the full path of simplemath while linking it inside advancedmath folder, although we may do so if we want, but the result will be same, simplemath will be installed globally before being linked to advancedmath.

Any folder installed globally, either by linking or by installing with -g flag can be linked to other projects as dependencies.

We can remove the bootstrapping anytime by executing npm rm simplemath or npm uninstall simplemath from inside advancedmath folder or by simply deleting the simplemath symlink from advancedmath/node_modules.

To remove global installation of the package, use the same command that is used to remove global installation of external packages.

$ npm uninstall -g simplemath

As seen in the previous section, linking also symlinks the binaries included in the module to the system's global binary folder. On the Linux environments those may be included in /usr/bin, usr/sbin, /usr/local/bin, usr/local/sbin, and so on.

Bear in mind that linking does not work on Windows without cygwin.

.npmignore

If you have used git and know what .gitignore does, .npmignore does pretty much the same thing. It is used to provide lists of files and folders to exclude while publishing the package. For example, our module might be generating a lot of log information in a log file while being tested. We don't want to include those in deployable builds since they are irrelevant to other users and unnecessarily increase package size.

We also might have some configuration files containing sensitive information like usernames and passwords. We don't want to have to remember every time we deploy to delete or move those files out of the folder before publishing the module. .npmignore is a newline delimited list of file names or patterns. The following patterns are recognized:

  • filename: File or folder with this name anywhere in the module will not be included in the bundle. For example, we don't want to include test files in the production bundle so we can add tests.js. We also don't want to include the node_modules folder because we expect npm to install all the dependencies recursively for our module users, so we might also include node_modules.

  • *.ext: File ending in .ext. For example *.log will ignore all files ending in .log and hence all of the log files.

  • folder/<filename>: Ignore file only in particular folder.

  • folder/*.ext: File ending in .ext in a particular folder.

  • folder/*: All files inside a particular folder.

  • folder/**/filename: any number of folders between folder and file.

  • directory/: Only matches a directory named directory and not files named directory.

Remember, lines starting with hash (#) are treated as comments. To override that behavior, escape # with a backslash (\). For a comprehensive list of patterns visit http://git-scm.com/docs/gitignore. The page is for .gitignore but same rules applies to .npmignore as well.

A common .npmignore file has the following content:

node_modules/
npm-debug.log

# other patterns

The above two patterns are ignored by default during npm publishing and don't need to be mentioned in the .npmignore file explicitly.

Global .npmignore

A common npmignore for all packages for all the users can be defined at {prefix}/etc/npmignore. This default can be overridden by specifying a file path in globalignoreconfig configuration. To know more about npm configuration please see the Config section.

Scripts

npm allows you to run arbitrary scripts on certain events, for example, before a module gets downloaded, or before the module gets published. These scripts could be anything, for example, to clean up files or to clean up some system state. Following scripts are available from npm scripts official documentation:

  • prepublish: Run before the package is published. (Also run on local npm install without any arguments.)

  • publish, postpublish: Run after the package is published.

  • preinstall: Run before the package is installed.

  • install, postinstall: Run after the package is installed.

  • preuninstall, uninstall: Run BEFORE the package is uninstalled.

  • postuninstall: Run after the package is uninstalled.

  • preupdate: Run before the package is updated with the update command.

  • update, postupdate: Run after the package is updated with the update command.

  • pretest, test, posttest: Run by the npm test command.

  • prestop, stop, poststop: Run by the npm stop command.

  • prestart, start, poststart: Run by the npm start command. If there is a server.js file present in the root of the package this command will default to node server.js.

  • prerestart, restart, postrestart: Run by the npm restart command.

Note

npm restart will run the stop and start scripts if no restart script is provided.

Test, start, stop, and restart scripts are run when the commands are called directly from the command line. Remember we had delegated test scripts to npm in previous sections by providing node tests.js to test property of the scripts object inside our project's package.json. Similarly, we can provide scripts to be run through the command line for other events mentioned previously as well.

{…
  "scripts": {
          "update": "echo 'Updating'",
          "start": "echo 'starting'",
          "stop": "echo 'stopping'",
          "prepublish": "echo 'About to publish'",
          "postpublish": "echo 'Published already!'"
       },
  ...
}

An interesting bit to know is that these npm scripts are aware of binaries inside ./node_modules/.bin even if they aren't installed globally.

Config

npm configuration are lists of key-value pairs comprising convenient defaults, information of user and computer, SSL certificates, and so on. For example, npm config stores the URL of standard registry; or npmjs username of a user who is currently logged in and its session information; or number of retries npm should make if a package installation fails for any reason. Let us check out what the registry URL is in npm config:

$ npm config get registry
https://registry.npmjs.org/

If the user wishes, he/she can change the default and provide a URL of a custom registry maintained by someone else or by themselves. The command for printing a config value on console is:

$ npm config get <key>

Or simply:

$ npm get <key>

Command for changing a config setting is:

$ npm config set <key> <value>

Or simply:

$ npm set <key> <value>

npm reads these values from a number of sources. The following are the sources from where npm picks up configs in the order of their precedence.

Command-line flag

Config information provided in the command line is of highest precedence. For example, to install a package from an alternate registry:

$ npm install <pacakge-identifier> --registry <registry-url>

The previous command doesn't persist the config value. To provide any config information in a command, include a flag with the config key followed by the value you want to set for that command: --<key> <value>.

Environment variables

Next, npm looks for a config value in the environment variable. The name of an environment variable corresponding to a config should be the config key preceded by npm_config_. So for example, to provide the configuration for loglevel in an environment variable, the variable should be named npm_config_loglevel.

User config file

Usually the file placed in $HOME/.npmrc, where $HOME is the logged-in user's home on the computer (/home/username on Unix, C:/Users/Username on Windows), or the userconfig configuration value set in either an environment variable or in the command line.

This file is an INI file, a formatted list of key-value pairs. We can use environment variables in this file by replacing with ${ENVIRONMENT_VARIABLE}. So for example, to provide configuration for maximum number of retries, npm should make for installing a package, we would write following in the config file:

fetch-retries = 3

Global config file

Global config file is similar to user config file. But global config file is for all the users on the computer. This file should be placed at {prefix}/etc/npmrc.

Defaults

Defaults refer to the defaults from npm or node.js source. This is not meant to be changed by npm users but by distributors who maintain their own node.js or npm fork.

So these are the places from where npm reads the configuration. Following the command is to get a list of configs defined on a computer:

$ npm config list

Provide a --long or -l switch to get the full list. To edit the user config file, execute the following command:

$ npm config edit

This will open the user config file ($HOME/.npmrc) on the system's default text editor. On Linux this defaults to vi, for Windows, notepad is the default. To read man pages for config:

$ npm help config

Shrinkwrap

npm has this way of fetching the latest package available, according to the semver version-range we give it. Whether we are publishing our package to registry or using it internally within our organization, sometimes it is not a good idea to publish a module which will probably be used by many in production, with loosely packed dependencies. You might never know what hell might break loose with updates to any of your dependencies or your dependencies' dependencies and so on.

Sometimes it is useful to deploy a node.js application with dependencies locked to exact versions against which it was tested so that we can have predictable results in the end. Although we can manage to provide exact versions of each of our dependencies, we cannot expect the same from every author. Our dependencies and their dependencies might have dependencies loosely defined.

Shrinkwrap command, given it was run inside a valid npm package root, creates a file npm-shrinkwrap.json in the project root. It is a .json file similar to package.json except that its dependencies property recursively defines the exact version of each and every dependency that our project uses. The prerequisites of executing the shrinkwrap command are:

  • All the dependencies indicated in package.json should be installed, no packages should be missing

  • No extraneous packages, that is other than those mentioned in package.json dependencies should be installed

Pruning

npm prune removes all the extraneous packages, that is, those packages not mentioned in package.json.

Running npm install inside a project or a package with npm-shrinkwrap.json file will pick up dependency versions from the npm-shrinkwrap.json file instead of package.json.

To shrinkwrap a project, run the following command inside the project root.

$ npm shrinkwrap
wrote npm-shrinkwrap.json

You will find the mentioned file. Take a look at reduced npm-shrinkwrap.json obtained on running shrinkwrap on mocha.

{
  "name": "mocha",
  "version": "1.8.1",
  "dependencies": {
    "commander": {
      "version": "0.6.1",
      "from": "[email protected]"
    },
    "growl": {
      "version": "1.7.0",
      "from": "[email protected]"
    },
    "jade": {
      "version": "0.26.3",
      "from": "[email protected]",
      "dependencies": {
        "mkdirp": {
          "version": "0.3.0",
          "from": "[email protected]"
        }
      }
    },
  // Other dependencies

Take a look at jade and its dependency mkdirp, the version of both the packages have recursively been locked down.

Tip

Run npm ls in a project root and see the how npm beautifully prints the dependency tree on the console. It even indicates if there are unmet or extraneous dependencies in the project.

Publishing

We used publish command in quick start, lets revisit and see what all things we can do with our packages on npm registry. Inside a project root npm publish publishes the node module. We can also do the following:

$ npm publish <folder | tarball>

From outside the project root, we can either specify the path to folder or the path to tarball or a URL to the tarball to publish. A new version should be mentioned in package.json every time we publish a module. Trying to publish a module with the same version twice will result in failure. But adding a --force flag will overwrite the version.

To unpublish a version use the following command:

$ npm unpublish <package-name>[@version]

The previous command removes the provided version of the package. To remove entire package and free the namespace on npm, drop the version part and provide --force flag.

Tip

npm pack from inside a package root creates a tarball of the package without node_modules folder. Npm uses this command for creating a tarball to send to registry for publishing. It comes handy in inspecting the final contents that will go on npm publish. Check out other variants of pack command from man pages. npm help pack

And finally you can add/remove module owners to the packages that you own and that are published on registry.

To add an owner, follow the given command:

$ npm owner add <username> <packag-name>

In the previous command <username> is the username of the user registered on http://npmjs.org and <package-name> is the name of the published package owned by you. This command adds an owner, which has full rights to publish, upgrade, unpublish, or completely remove the package from npm. The added owner can even remove other owners, including the one who added him in the first place, so be careful with this command and add owners only that you completely trust. This might change, and npm suggests providing more fine-grained access control in future.

Similarly, to remove an owner use the following command:

$ npm owner rm <username> <packag-name>

And to list owners of a package use the following command:

$ npm owner ls <packag-name>

To run the previous command you don't need to be the owner of the mentioned package.

Help

Lastly, the best way to help yourself is to use help provided through npm man pages. Here are few commands to get quick help:

$ npm

Just running npm command provides a brief, non-exhaustive list of npm commands. Providing an -l switch will provide a more exhaustive list.

$ npm faq

The previous command opens man page for frequently asked npm questions.

$ npm <command> -h

The previous command provides quick help on the command.

$ npm help <term>

The previous command opens man page for the term and/or associated command, and gives a detailed description. For example, npm help publish or npm help star.

$ npm help npm

The previous command opens a detailed document of npm, its purpose and the way it works.

You can also text search through npm man pages using the help-search command.

$ npm help-search <text>

The help-search command will search for text and display the search results and their locations in man pages on command line.