When we've created an analysis that HQ finds useful, we'll often have to scale this up to work on a larger supply of log files. This will involve acquiring and downloading files from servers without manually clicking a link to download and save each file.
We'll provide a sample of how we might use Python's ftplib to acquire files in bulk for analysis. Once we have the files locally, we can process them using our local_gzip() or local_text() functions.
Here's a function that performs a complex of FTP interaction:
import ftplib
def download( host, path, username=None ):
with ftplib.FTP(host, timeout=10) as ftp:
if username:
password = getpass.getpass("Password: ")
ftp.login(user=username,passwd=password)
else:
ftp.login()
ftp.cwd(path)
for name, facts in ftp.mlsd(".", ["type","size"]):
if name.startswith("."): continue
if facts['type'] == 'dir': continue
...