Book Image

Distributed Computing with Go

By : V.N. Nikhil Anurag
Book Image

Distributed Computing with Go

By: V.N. Nikhil Anurag

Overview of this book

Distributed Computing with Go gives developers with a good idea how basic Go development works the tools to fulfill the true potential of Golang development in a world of concurrent web and cloud applications. Nikhil starts out by setting up a professional Go development environment. Then you’ll learn the basic concepts and practices of Golang concurrent and parallel development. You’ll find out in the new few chapters how to balance resources and data with REST and standard web approaches while keeping concurrency in mind. Most Go applications these days will run in a data center or on the cloud, which is a condition upon which the next chapter depends. There, you’ll expand your skills considerably by writing a distributed document indexing system during the next two chapters. This system has to balance a large corpus of documents with considerable analytical demands. Another use case is the way in which a web application written in Go can be consciously redesigned to take distributed features into account. The chapter is rather interesting for Go developers who have to migrate existing Go applications to computationally and memory-intensive environments. The final chapter relates to the rather onerous task of testing parallel and distributed applications, something that is not usually taught in standard computer science curricula.
Table of Contents (11 chapters)


Programmers tend to work on many projects and it is good practice to have the source code separate from nonprogramming-related files. It is a common practice to have the source code in a separate location or workspace. Every programming language has its own conventions on how the language-related projects should be set up and Go is no exception to this.

GOPATH is the most important environment variable the developer has to set. It tells the Go compiler where to find the source code for the project and its dependencies. There are conventions within the GOPATH that need to be followed, and they have to deal with folder hierarchies.


This is the directory that will contain the source code of our projects and their dependencies. In general, we want our source code to have version control and be hosted on the cloud. It would also be great if we or anyone else could easily use our project. This requires a little extra setup on our part.

Let's imagine that our project is hosted at We want to clone and build this project on our local system. To make it properly work, we need to clone it to $GOPATH/src/ When we build a Go project with dependencies for the first time, we will see that the src/ folder has many directories and subdirectories that contain the dependencies of our project.


Go is a compiled programming language; we have the source code and code for the dependencies that we want to use in our project. In general, every time we build a binary, the compiler has to read the source code of our project and dependencies and then compile it to machine code. Compiling unchanged dependencies every time we compile our main program would lead to a very slow build process. This is the reason that object files exist; they allow us to compile dependencies into reusable machine code that can be readily included in our Go binary.

These object files are stored in $GOPATH/pkg; they follow a directory structure similar to that of src/, except that they are within a subdirectory. These directories tend to follow the naming pattern of <OS>_<CPU-Architecture>, because we can build executable binaries for multiple systems:

$ tree $GOPATH/pkg
└── linux_amd64
    │   ├── abbot
    │   │   └── go-http-auth.a
    │   ├── dimfeld
    │   │   └── httppath.a
    │   ├── oklog
    │   │   └── ulid.a
    │   ├── rcrowley
    │   │   └── go-metrics.a
    │   ├── sirupsen
    │   │   └── logrus.a
    │   ├── sony
    │   │   └── gobreaker.a
        └── x
            ├── crypto
            │   ├── bcrypt.a
            │   ├── blowfish.a
            │   └── ssh
            │       └── terminal.a
            ├── net
            │   └── context.a
            └── sys  


Go compiles and builds our projects into executable binaries and places them in this directory. Depending on the build specs, they might be executable on your current system or other systems. In order to use the binaries that are available in the bin/ directory, we need to set the corresponding GOBIN=$GOPATH/bin environment variable.