The WWW (World Wide Web) makes it possible for hypermedia objects on the Internet to interconnect, communicating through a standard set of Internet protocols, commonly HTTP (Hyper Text Transfer Protocol). The growth in the complexity, number, and type of web applications delivering curated collections of these objects through the browser has increased interest in technologies that aid in the construction and management of intricate networked applications. Node is one such technology. By mastering Node you are learning how to build the next generation of software.

The hold that any one person has on information is tenuous. Complexity follows scale; confusion follows complexity. As resolution blurs, errors happen.

Similarly, the activity graph describing all expected I/O (Input/Output) interactions an application may potentially form between clients and providers must be carefully planned and managed, lest the capacity of both the system and its creator be overwhelmed. This involves controlling two dimensions of information: volume and shape.

As a network application scales, the volume of information it must recognize, organize, and maintain increases. This volume, in terms of I/O streams, memory usage, and CPU (Central Processing Unit) load, expands as more clients connect, and even as they leave (in terms of persisting user-specific data).

This expansion of information volume also burdens the application developer, or team of developers. Scaling issues begin to present themselves, usually demonstrating a failure to accurately predict the behavior of large systems from the behavior of small systems. Can a data layer designed for storing a few thousand records accommodate a few million? Are the algorithms used to search a handful of records efficient enough to search many more? Can this server handle 10,000 simultaneous client connections? The edge of innovation is sharp and cuts quickly, presenting less time for deliberation precisely when the cost of error is being magnified. The shape of objects comprising the whole of an application becomes amorphous and difficult to understand, particularly as ad hoc modifications are made, reactively, in response to dynamic tension in the system. What is described in a specification as a small subsystem may have been patched into so many other systems that its actual boundaries are misunderstood. It becomes impossible to accurately trace the outline of the composite parts of the whole.

Eventually an application becomes unpredictable. It is dangerous when one cannot predict all future states of an application, or the side effects of change. Any number of servers, programming languages, hardware architectures, management styles, and so on, have attempted to subdue the intractable problem of risk following growth, of failure menacing success. Oftentimes systems of even greater complexity are sold as the cure.

Node chose clarity and simplicity instead. There is one thread, bound to an event loop. Deferred tasks are encapsulated, entering and exiting the execution context via callbacks. I/O operations generate evented data streams, these piped through a single stack. Concurrency is managed by the system, abstracting away thread pools and simplifying memory management. Dependencies and libraries are introduced through a package management system, neatly encapsulated, and easy to distribute, install, and invoke.

Experienced developers have all struggled with the problems that Node aims to solve:

As each year passes, we see collaborative applications and software responsible for managing levels of concurrency that would have been considered rare just a few years ago. Managing concurrency, both in terms of connection handling and application design, is the key to building scalable web architectures.

In this book we will study the techniques professional Node developers use to tackle these problems. In this chapter, we will explore how a Node application is designed, the shape and texture of its footprint on a server, and the powerful base set of tools and features Node provides for developers. Throughout we will examine progressively more intricate examples demonstrating how Node's simple, comprehensive, and consistent architecture solves many difficult problems well.