Book Image

Web Scraping with Python

By : Richard Penman
Book Image

Web Scraping with Python

By: Richard Penman

Overview of this book

Table of Contents (16 chapters)

Disk cache

To cache downloads, we will first try the obvious solution and save web pages to the filesystem. To do this, we will need a way to map URLs to a safe cross-platform filename. The following table lists the limitations for some popular filesystems:

Operating system

File system

Invalid filename characters

Maximum filename length



/ and \0

255 bytes


HFS Plus

: and \0

255 UTF-16 code units



\, /, ?, :, *, ", >, <, and |

255 characters

To keep our file path safe across these filesystems, it needs to be restricted to numbers, letters, basic punctuation, and replace all other characters with an underscore, as shown in the following code:

>>> import re
>>> url = ''
>>> re.sub('[^/0-9a-zA-Z\-.,;_ ]', '_', url)

Additionally, the filename and the parent directories need to be restricted to 255 characters...