We start by going to the Yelp website (https://yelp.com/) and searching for all the restaurants in the city of London. When you do that, you'll get a list of all the restaurants in London. Observe the URL that displays the search criteria. It is https://www.yelp.com/search?find_desc=Restaurants&find_loc=London. See the following screenshot for reference:

Now, if you click on any of the restaurants' link that shows up in the search results, we should get the details that Ryan needs. See the following screenshot, where we get the details of Ffiona's Restaurant. Note how every restaurant has a dedicated URL; in this case, it is https://www.yelp.com/biz/ffionas-restaurant-london?osq=Restaurants. Also note that on this page, we have the name of the restaurant, the street address, and even the contact number. All the details that Ryan needs for his campaign; that's cool!

OK nice, so we now know how to get the list of restaurants and also fetch the relevant details for a restaurant. But how do we achieve this in an automated way? As we saw in the web scraping example, we need to look for the HTML elements on the web pages from where we can collect this data.

Let's start with the search page. Open the search page (https://www.yelp.com/search?find_desc=Restaurants&find_loc=London) on your Chrome browser. Now, right-click on the first restaurant's URL and click on Inspect to get the HTML elements. If you notice, in the following screenshot, all the restaurants that are listed on the search page have a common CSS class name, biz-name, which indicates the name of the restaurant. It also contains the href tag, which points to the dedicated URL of the restaurant. In our screenshot, we get the name, Ffiona's Restaurant, and the href points to the restaurant's URL, https://yelp.com/biz/ffionas-restaurant-london?osq=Resturants.

Now, let's look at the dedicated page of the restaurant to see how we collect the street address and the contact number of the restaurant with the HTML elements. We perform the same operation, right-click, and Inspect to get the HTML elements of street address and contact number. See the following screenshot for reference. Note that for the street address, we have a separate CSS class, street-address, and the contact number is available under a span with the class name, biz-phone.

Awesome! So, we now have all the HTML elements that can be used to scrape the data in an automated way. Let's now look at the implementation. The following Python code performs these operations in an automated way:

        from bs4 import BeautifulSoup 
        from threading import Thread 
        import urllib 
 
        #Location of restaurants 
        home_url = "https://www.yelp.com" 
        find_what = "Restaurants" 
        location = "London" 
 
        #Get all restaurants that match the search criteria 
        search_url = "https://www.yelp.com/search?find_desc=" +
        find_what + "&find_loc=" + location 
        s_html = urllib.urlopen(search_url).read() 
        soup_s = BeautifulSoup(s_html, "lxml") 
 
        #Get URLs of top 10 Restaurants in London 
        s_urls = soup_s.select('.biz-name')[:10] 
        url = [] 
        for u in range(len(s_urls)): 
        url.append(home_url + s_urls[u]['href']) 
 
 
        #Function that will do actual scraping job 
        def scrape(ur): 
                html = urllib.urlopen(ur).read() 
                soup = BeautifulSoup(html, "lxml") 
 
                title = soup.select('.biz-page-title') 
                saddress = soup.select('.street-address') 
                phone = soup.select('.biz-phone') 
 
                if title: 
                     print "Title: ", title[0].getText().strip() 
                if saddress: 
                     print "Street Address: ",
        saddress[0].getText().strip() 
                if phone: 
                     print "Phone Number: ", phone[0].getText().strip() 
                print "-------------------" 
 
        threadlist = [] 
        i=0 
        #Making threads to perform scraping 
        while i<len(url): 
                  t = Thread(target=scrape,args=(url[i],)) 
                  t.start() 
                  threadlist.append(t) 
                  i=i+1 
 
        for t in threadlist: 
                  t.join() 
 

OK, great! Now, if we run the preceding Python code, we get the details of the top 10 restaurants in London, along with their names, street addresses and contact numbers. Refer to the following screenshot:

In the preceding screenshot, we get the records of 10 restaurants in London provided by Yelp. Title is the name of the restaurant and Street Address and Phone Number are self-explanatory. Awesome! We did it for Ryan.