Retrieving files from an FTP server for processing is a very common operation for GIS programmers and can be automated with a Python script.
Connecting to an FTP server and downloading a file is accomplished through the ftplib module. A connection to an FTP server is created through the FTP object, which accepts a host, username, and password to create the connection. Once a connection has been opened, you can then search for and download files.
In this recipe, you will connect to the National Interagency Fire Center Incident FTP site and download a PDF file for a wildfire in Colorado. Before you run the following script, you will need to create a username/password through http://gis.nwcg.gov/data_nifcftp.html.
Follow these steps to create a script that connects to an FTP server and downloads a file:
Open IDLE and create a file called
C:\ArcpyBook\Appendix2\ftp.py.We'll be connecting to an FTP server at the NIFC. Visit their website at http://gis.nwcg.gov/data_nifcftp.html for more information.
Import the
ftplib,os, andsocketmodules:import ftplib import os import socket
Add the following variables that define the URL, directory, and filename:
HOST = 'ftp.nifc.gov' USER = '<your username here>' PASSW = '<your password here>' DIRN = '/Incident_Specific_Data/2012 HISTORIC/ROCKY_MTN/Arapaho/GIS/20120629' FILE = '20120629_0600_Arapaho_PIO_0629_8x11_land.pdf'
Add the following code block to create a connection. If there is a connection error, a message will be generated. If the connection was successful, a success message will be printed:
try: f = ftplib.FTP(HOST,USER,PASS) except (socket.error, socket.gaierror), e: print('ERROR: cannot reach "%s"' % HOST) print('*** Connected to host "%s"' % HOST)Add the following code block to anonymously log in to the server:
try: f.login() except ftplib.error_perm: print('ERROR: cannot login') f.quit() print('*** Logged in ')Add the following code block to change to the directory specified in our
DIRNvariable:try: f.cwd(DIRN) except ftplib.error_perm: print('ERROR: cannot CD to "%s"' % DIRN) f.quit() print('*** Changed to "%s" folder' % DIRN)Use the
FTP.retrbinary()function to retrieve the PDF file:try: f.retrbinary('RETR %s' % FILE, open(FILE, 'wb').write) except ftplib.error_perm: print('ERROR: cannot read file "%s"' % FILE) os.unlink(FILE) else: print('*** Downloaded "%s" to CWD' % FILE)Make sure you disconnect from the server:
f.quit()
The entire script should appear as follows:
import ftplib import os import socket HOST = 'ftp.nifc.gov' USER = '<your username here>' PASSW = '<your password here>' DIRN = '/Incident_Specific_Data/2012 HISTORIC/ROCKY_MTN/Arapaho/GIS/20120629' FILE = '20120629_0600_Arapaho_PIO_0629_8x11_land.pdf' try: f = ftplib.FTP(HOST,USER,PASSW) except (socket.error, socket.gaierror), e: print('ERROR: cannot reach "%s"' % HOST) print('*** Connected to host "%s"' % HOST) try: f.login() except ftplib.error_perm: print('ERROR: cannot login') f.quit() print('*** Logged in ') try: f.cwd(DIRN) except ftplib.error_perm: print('ERROR: cannot CD to "%s"' % DIRN) f.quit() print('*** Changed to "%s" folder' % DIRN) try: f.retrbinary('RETR %s' % FILE, open(FILE, 'wb').write) except ftplib.error_perm: print('ERROR: cannot read file "%s"' % FILE) os.unlink(FILE) else: print('*** Downloaded "%s" to CWD' % FILE) f.quit()You can check your work by examining the
C:\ArcpyBook\code\Appendix2\ftp.py solution file.Save and run the script. If everything is successful, you should see the following output:
*** Connected to host "ftp.nifc.gov" *** Logged in as "anonymous" *** Changed to "'/Incident_Specific_Data/2012 HISTORIC/ROCKY_MTN/Arapaho/GIS/20120629'" folder *** Downloaded "'20120629_0600_Arapaho_PIO_0629_8x11_land.pdf " to CWD
Check your
C:\ArcpyBook\Appendix2directory for the file. By default, FTP will download files to the current working directory.
To connect to an FTP server, you need to know the URL. You also need to know the directory and filename for the file that will be downloaded. In this script, we have hardcoded this information, so that you can focus on implementing the FTP-specific functionality. Using this information, we then created a connection to the NIFC FTP server. This is done through the ftplib.FTP() function, which accepts a URL to the host.
Keep in mind that you'll need to obtain a username/password to log in and download the data. Once logged in, the script then changes directories from the root of the FTP server to the path defined in the DIRN variable. This was accomplished with the cwd(<path>) function. The PDF file was retrieved by using the retrbinary() function. Finally, you will want to close your connection to the FTP server when you're done. This is done with the quit() method.
There are a number of additional FTP-related methods that you can use to perform various actions. Generally, these can be divided into directory-level operations and file-level operations. Directory level methods include the dir() method to obtain a list of files in a directory, mkd() to create a new directory, pwd() to get the current working directory, and cwd() to change the current directory. Keep in mind that the actions you attempt to perform through your script will be governed by the privileges assigned to your account, so you may not be able to successfully execute every method that I mention.
The ftplib module also includes various methods to work with files. You can upload and download files in a binary or plain text format. The retrbinary() and storbinary() methods are used to retrieve and store binary files, respectively. Plain text files can be retrieved and stored using retrlines() and storlines().
There are several others methods on the FTP class that you should be aware of. Deleting a file can be done with the delete() method, while renaming a file can be accomplished with rename(). You can also send commands to the FTP server through the sendcmd() method.