Sign In Start Free Trial
Account

Add to playlist

Create a Playlist

Modal Close icon
You need to login to use this feature.
  • Book Overview & Buying Java Data Science Cookbook
  • Table Of Contents Toc
  • Feedback & Rating feedback
Java Data Science Cookbook

Java Data Science Cookbook

By : Shams
close
close
Java Data Science Cookbook

Java Data Science Cookbook

By: Shams

Overview of this book

If you are looking to build data science models that are good for production, Java has come to the rescue. With the aid of strong libraries such as MLlib, Weka, DL4j, and more, you can efficiently perform all the data science tasks you need to. This unique book provides modern recipes to solve your common and not-so-common data science-related problems. We start with recipes to help you obtain, clean, index, and search data. Then you will learn a variety of techniques to analyze, learn from, and retrieve information from data. You will also understand how to handle big data, learn deeply from data, and visualize data. Finally, you will work through unique recipes that solve your problems while taking data science to production, writing distributed data science applications, and much more - things that will come in handy at work.
Table of Contents (10 chapters)
close
close

Extracting web data from a website using Selenium Webdriver

Selenium is a Java-based tool to help automating software testing or quality assurance. Interestingly enough, Selenium can be used to automatically retrieve and utilize web data. This recipe shows you how.

Getting ready

In order to perform this recipe, we will require the following:

  1. Download selenium-server-standalone-2.53.1.jar and selenium-java-2.53.1.zip from http://selenium-release.storage.googleapis.com/index.html?path=2.53/. From the latter, extract the selenium-java-2.53.1.jar file. Include these two JAR files in your eclipse project an external Java library.
  2. Download and install Firefox 47.0.1 from https://ftp.mozilla.org/pub/firefox/releases/47.0.1/ by selecting the version appropriate for your operating system.

Tip

Because of the version conflict issues between Selenium and Firefox, once you run code with a particular version, turn off the automatic update download and installation option in Firefox.

How to do it...

  1. Create a method named extractDataWithSelenium(String) that takes a String as a parameter, which eventually is the URL from where we are going to extract data. There can be many different types of data that we can extract from URLs, such as the title, the headers, and the values in a selection drop-down box. This recipe only concentrates on extracting the text part of the webpage:
            public String extractDataWithSelenium(String url){ 
    
  2. Next, create a Firefox web driver using the following code:
            WebDriver driver = new FirefoxDriver(); 
    
  3. Use the get() method of the WebDriver object by passing the URL:
            driver.get("http://cogenglab.csd.uwo.ca/rushdi.htm"); 
    
  4. The text of the webpage can be found using xpath, where the value of id is content:

    How to do it...

    How to do it...

  5. Find this particular element with the findElement() method. This method returns a WebElement object. Create a WebElement object named webElement to hold the returned value:
            WebElement webElement = driver.findElement(By.xpath("//* 
              [@id='content']")); 
    
  6. The WebElement object has a method named getText(). Call this method to retrieve the text of the web page, and put the text into a String variable as follows:
            String text = (webElement.getText()); 
    
  7. Finally, return the String variable and close the method:
            }

    The complete code segment with the driver main() method for the recipe looks like the following:

    import org.openqa.selenium.By; 
    import org.openqa.selenium.WebDriver; 
    import org.openqa.selenium.WebElement; 
    import org.openqa.selenium.firefox.FirefoxDriver; 
     
    public class TestSelenium { 
       public String extractDataWithSelenium(String url) { 
          WebDriver driver = new FirefoxDriver(); 
          driver.get("http://cogenglab.csd.uwo.ca/rushdi.htm"); 
          WebElement webElement = driver.findElement(By.xpath("//*
            [@id='content']")); 
          System.out.println(webElement.getText()); 
          return url;    
       } 
     
       public static void main(String[] args){ 
          TestSelenium test = new TestSelenium(); 
          String webData = test.extractDataWithSelenium
            ("http://cogenglab.csd.uwo.ca/rushdi.htm"); 
          //process webData 
       } 
    } 
    

Note

Selenium and Firefox have compatibility issues. Some Selenium versions do not work with some Firefox versions. The recipe provided here works fine with the versions mentioned in the recipe. But it does not have any guarantee that it will work with other Selenium or Firefox versions.

Because of the version conflict issues between Selenium and Firefox, once you run a code with a particular version of the both, turn off the automatic update download and installation option in Firefox.

Visually different images
CONTINUE READING
83
Tech Concepts
36
Programming languages
73
Tech Tools
Icon Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.
Icon Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.
Icon 50+ new titles added per month and exclusive early access to books as they are being written.
Java Data Science Cookbook
notes
bookmark Notes and Bookmarks search Search in title playlist Add to playlist download Download options font-size Font size

Change the font size

margin-width Margin width

Change margin width

day-mode Day/Sepia/Night Modes

Change background colour

Close icon Search
Country selected

Close icon Your notes and bookmarks

Confirmation

Modal Close icon
claim successful

Buy this book with your credits?

Modal Close icon
Are you sure you want to buy this book with one of your credits?
Close
YES, BUY

Submit Your Feedback

Modal Close icon
Modal Close icon
Modal Close icon