Book Image

Hands-On Software Engineering with Golang

By : Achilleas Anagnostopoulos
Book Image

Hands-On Software Engineering with Golang

By: Achilleas Anagnostopoulos

Overview of this book

Over the last few years, Go has become one of the favorite languages for building scalable and distributed systems. Its opinionated design and built-in concurrency features make it easy for engineers to author code that efficiently utilizes all available CPU cores. This Golang book distills industry best practices for writing lean Go code that is easy to test and maintain, and helps you to explore its practical implementation by creating a multi-tier application called Links ‘R’ Us from scratch. You’ll be guided through all the steps involved in designing, implementing, testing, deploying, and scaling an application. Starting with a monolithic architecture, you’ll iteratively transform the project into a service-oriented architecture (SOA) that supports the efficient out-of-core processing of large link graphs. You’ll learn about various cutting-edge and advanced software engineering techniques such as building extensible data processing pipelines, designing APIs using gRPC, and running distributed graph processing algorithms at scale. Finally, you’ll learn how to compile and package your Go services using Docker and automate their deployment to a Kubernetes cluster. By the end of this book, you’ll know how to think like a professional software developer or engineer and write lean and efficient Go code.
Table of Contents (21 chapters)
1
Section 1: Software Engineering and the Software Development Life Cycle
3
Section 2: Best Practices for Maintainable and Testable Go Code
7
Section 3: Designing and Building a Multi-Tier System from Scratch
14
Section 4: Scaling Out to Handle a Growing Number of Users
18
Epilogue

Building a generic data-processing pipeline in Go

The following figure illustrates the high-level design of the pipeline that we will be building throughout the first half of this chapter:

Figure 1: A generic, multistage pipeline

Keep in mind that this is definitely not the only, or necessarily the best, way to go about implementing a data-processing pipeline. Pipelines are inherently application specific, so there is not really a one-size-fits-all guide for constructing efficient pipelines.

Having said that, the proposed design is applicable to a wide variety of use cases, including, but not limited to, the crawler component for the Links 'R' Us project. Let's examine the preceding figure in a bit more detail and identify the basic components that the pipeline comprises:

  • The input source: Inputs essentially function as data-sources that pump data into the pipeline...