One of the tasks many developers will take on is the building of a logfile processor. A logfile can be very large and many megabytes long. Any single program working on a very large file can easily run into memory problems or simply run much too slowly. It makes sense to process a large file in pieces. We'll build a simple log processor that breaks up a big file into pieces and assigns one to each of several child workers, running them in parallel.
The entire code for this example can be found in the logproc folder of the code bundle. We will focus on the main routines:
- Determining the number of lines in the logfile
- Breaking those up into equal chunks
- Creating one child for each chunk and passing it parse instructions
- Assembling and displaying the results
To get the word count of our file, we use the wc command with child.exec, as...