CSV is a common format that is used to manipulate data. It's common, for example, to import or export a CSV file into Excel. The Go CSV package operates on data interfaces, and as a result, it's easy to write data to a buffer, stdout, a file, or a socket. The examples in this section will show some common ways to get data into and out of the CSV format.
How to do it...
These steps cover how to write and run your application:
- From your Terminal or console application, create a new directory called ~/projects/go-programming-cookbook/chapter1/csvformat.
- Navigate to this directory.
- Run the following command:
$ go mod init github.com/PacktPublishing/Go-Programming-Cookbook-Second-Edition/chapter1/csvformat
You should see a file called go.mod that contains the following contents:
module github.com/PacktPublishing/Go-Programming-Cookbook-Second-Edition/chapter1/csvformat
- Copy the tests from~/projects/go-programming-cookbook-original/chapter1/csvformat or use this as an exercise to write some of your own code!
- Create a file called read_csv.gowith the following contents:
package csvformat
import (
"bytes"
"encoding/csv"
"fmt"
"io"
"strconv"
)
// Movie will hold our parsed CSV
type Movie struct {
Title string
Director string
Year int
}
// ReadCSV gives shows some examples of processing CSV
// that is passed in as an io.Reader
func ReadCSV(b io.Reader) ([]Movie, error) {
r := csv.NewReader(b)
// These are some optional configuration options
r.Comma = ';'
r.Comment = '-'
var movies []Movie
// grab and ignore the header for now
// we may also want to use this for a dictionary key or
// some other form of lookup
_, err := r.Read()
if err != nil && err != io.EOF {
return nil, err
}
// loop until it's all processed
for {
record, err := r.Read()
if err == io.EOF {
break
} else if err != nil {
return nil, err
}
year, err := strconv.ParseInt(record[2], 10,
64)
if err != nil {
return nil, err
}
m := Movie{record[0], record[1], int(year)}
movies = append(movies, m)
}
return movies, nil
}
- Add this additional function to read_csv.go, as follows:
// AddMoviesFromText uses the CSV parser with a string
func AddMoviesFromText() error {
// this is an example of us taking a string, converting
// it into a buffer, and reading it
// with the csv package
in := `
- first our headers
movie title;director;year released
- then some data
Guardians of the Galaxy Vol. 2;James Gunn;2017
Star Wars: Episode VIII;Rian Johnson;2017
`
b := bytes.NewBufferString(in)
m, err := ReadCSV(b)
if err != nil {
return err
}
fmt.Printf("%#vn", m)
return nil
}
- Create a file called write_csv.gowith thefollowing contents:
package csvformat
import (
"bytes"
"encoding/csv"
"io"
"os"
)
// A Book has an Author and Title
type Book struct {
Author string
Title string
}
// Books is a named type for an array of books
type Books []Book
// ToCSV takes a set of Books and writes to an io.Writer
// it returns any errors
func (books *Books) ToCSV(w io.Writer) error {
n := csv.NewWriter(w)
err := n.Write([]string{"Author", "Title"})
if err != nil {
return err
}
for _, book := range *books {
err := n.Write([]string{book.Author,
book.Title})
if err != nil {
return err
}
}
n.Flush()
return n.Error()
}
- Add these additional functions to write_csv.go, as follows:
// WriteCSVOutput initializes a set of books
// and writes them to os.Stdout
func WriteCSVOutput() error {
b := Books{
Book{
Author: "F Scott Fitzgerald",
Title: "The Great Gatsby",
},
Book{
Author: "J D Salinger",
Title: "The Catcher in the Rye",
},
}
return b.ToCSV(os.Stdout)
}
// WriteCSVBuffer returns a buffer CSV for
// a set of books
func WriteCSVBuffer() (*bytes.Buffer, error) {
b := Books{
Book{
Author: "F Scott Fitzgerald",
Title: "The Great Gatsby",
},
Book{
Author: "J D Salinger",
Title: "The Catcher in the Rye",
},
}
w := &bytes.Buffer{}
err := b.ToCSV(w)
return w, err
}
- Create a new directory named example and navigate to it.
- Create a main.go file with the following contents:
package main
import (
"fmt"
"github.com/PacktPublishing/
Go-Programming-Cookbook-Second-Edition/
chapter1/csvformat"
)
func main() {
if err := csvformat.AddMoviesFromText(); err != nil {
panic(err)
}
if err := csvformat.WriteCSVOutput(); err != nil {
panic(err)
}
buffer, err := csvformat.WriteCSVBuffer()
if err != nil {
panic(err)
}
fmt.Println("Buffer = ", buffer.String())
}
- Rungo run ..
- You may also run the following:
$ go build
$ ./example
You should see the following output:
$ go run .
[]csvformat.Movie{csvformat.Movie{Title:"Guardians of the
Galaxy Vol. 2", Director:"James Gunn", Year:2017},
csvformat.Movie{Title:"Star Wars: Episode VIII", Director:"Rian
Johnson", Year:2017}}
Author,Title
F Scott Fitzgerald,The Great Gatsby
J D Salinger,The Catcher in the Rye
Buffer = Author,Title
F Scott Fitzgerald,The Great Gatsby
J D Salinger,The Catcher in the Rye
- If you copied or wrote your own tests, go up one directory and run go test, and ensure that all tests pass.
How it works...
In order to learn how to read a CSV format, we first represent our data as a structure. In Go, it's very useful to format data as a structure, as it makes things such as marshaling and encoding relatively simple. Our read example uses movies as our data type. The function takes anio.Readerinterface that holds our CSV data as an input. This could be a file or a buffer. We then use that data to create and populate aMoviestructure, including converting the year into an integer. We also add options to the CSV parser to use; (semi-colon) as the separator and- (hyphen) as a comment line.
Next, we explore the same idea, but in reverse. Novels are represented with a title and an author. We initialize an array of novels and then write specific novels in the CSV format to an io.Writer interface. Once again, this can be a file, stdout, or a buffer.
The CSV package is an excellent example of why you'd want to think of data flows in Go as implementing common interfaces. It's easy to change the source and destination of our data with small one-line tweaks, and we can easily manipulate CSV data without using an excessive amount of memory or time. For example, it would be possible to read from a stream of data one record at a time and write to a separate stream in a modified format one record at a time. Doing this would not incur significant memory or processor usage.
Later, when we explore data pipelines and worker pools, you'll see how these ideas can be combined and how to handle these streams in parallel.