Book Image

R Web Scraping Quick Start Guide

By : Olgun Aydin
Book Image

R Web Scraping Quick Start Guide

By: Olgun Aydin

Overview of this book

Web scraping is a technique to extract data from websites. It simulates the behavior of a website user to turn the website itself into a web service to retrieve or introduce new data. This book gives you all you need to get started with scraping web pages using R programming. You will learn about the rules of RegEx and Xpath, key components for scraping website data. We will show you web scraping techniques, methodologies, and frameworks. With this book's guidance, you will become comfortable with the tools to write and test RegEx and XPath rules. We will focus on examples of dynamic websites for scraping data and how to implement the techniques learned. You will learn how to collect URLs and then create XPath rules for your first web scraping script using rvest library. From the data you collect, you will be able to calculate the statistics and create R plots to visualize them. Finally, you will discover how to use Selenium drivers with R for more sophisticated scraping. You will create AWS instances and use R to connect a PostgreSQL database hosted on AWS. By the end of the book, you will be sufficiently confident to create end-to-end web scraping systems using R.
Table of Contents (7 chapters)

Introduction to Web Scraping

Web scraping is the process of extracting a structural representation of data from a website. The formatting language used to configure data on web pages may display HTML variability, because existing techniques for web scraping are based on markup. A change in HTML can lead to the removal of incorrect data.

Throughout this book, we will be using R to help us scrape data from web pages. R is an open source programming language and it's one of the most preferred programming languages among data scientists and researchers. R not only provides algorithms for statistical models and machine learning methods, but also provides a web scraping environment for researchers. The data collected from websites should also be stored somewhere. For this, we will learn to store the data in PostgreSQL databases, which we will do by using R.

As an example, a company may want to autonomously track product prices for its competitors. If the information does not provide a proprietary API, the solution is to write a program that targets the marking of the web page. A common approach is to parse the web page into a tree representation and resolve it with XPath expressions. If you have any questions like, Okay how can we make scripts run automatically? You will find the answer in this book.

The aim of this book is to offer a quick guide on web Scraping techniques and software that can be used to extract data from websites.

In this chapter, we will learn about the following topics:

  • Data on the internet
  • Introduction to XPath (XML Path)
  • Data extraction systems
  • Web scraping techniques