Book Image

R High Performance Programming

Book Image

R High Performance Programming

Overview of this book

Table of Contents (17 chapters)
R High Performance Programming
Credits
About the Authors
About the Reviewers
www.PacktPub.com
Preface
Index

Chapter 7. Processing Large Datasets with Limited RAM

In the previous chapter, we learned how to optimize the memory consumption of R programs by reducing the copying of data and by removing temporary data. Sometimes, that is still not enough. We might have data that is too large to even fit into memory, let alone perform any computations on them, or even if the data can fit into memory, there is not much free memory left for the analyses that we need to perform.

In this chapter, we will learn advanced techniques to overcome memory limitations and process large datasets.

This chapter covers:

  • Using memory-efficient data structures

  • Using memory-mapped files and processing data in chunks