The ptrace() command is the system call that is most commonly used for memory analysis of a userland. In fact, if you are designing forensics software that runs in userland, the only way it can access other processes memory is through the ptrace system call, or by reading the proc filesystem (unless, of course, the program has some type of explicit shared memory IPC setup).
Note
One may attach to a process and then open/lseek/read/write /proc/<pid>/mem as an alternative to ptrace read/write semantics.
In 2011, I was awarded a contract by the DARPA CFT (Cyber Fast Track) program to design something called Linux VMA Monitor. The purpose of this software is to detect a wide range of known and unknown process memory infections, such as rootkits and memory-resident viruses.
It essentially performs automated intelligent memory forensic analysis on every single process address space using special heuristics that understands ELF execution. It can spot anomalies or...