Book Image

Web Scraping with Python

By : Richard Penman
Book Image

Web Scraping with Python

By: Richard Penman

Overview of this book

Table of Contents (16 chapters)

Extending the login script to update content

Now that the login automation is working, we can make the script more interesting by extending it to interact with the website and update the country data. The code used in this section is available at You may have noticed an Edit link at the bottom of each country:

When logged in, this leads to another page where each property of a country can be edited:

We will make a script to increase the population of a country by one person each time it is run. The first step is to extract the current values of the country by reusing the parse_form() function:

>>> import login
>>> COUNTRY_URL = ''
>>> opener = login.login_cookies()
>>> country_html =
>>> data = parse_form(country_html)
>>> pprint.pprint(data)
{'_formkey': '4cf0294d-ea71-4cd8-ae2a-43d4ca0d46dd',...