-
Book Overview & Buying
-
Table Of Contents
-
Feedback & Rating
Java Data Science Cookbook
By :
Selenium is a Java-based tool to help automating software testing or quality assurance. Interestingly enough, Selenium can be used to automatically retrieve and utilize web data. This recipe shows you how.
In order to perform this recipe, we will require the following:
selenium-server-standalone-2.53.1.jar and selenium-java-2.53.1.zip from http://selenium-release.storage.googleapis.com/index.html?path=2.53/. From the latter, extract the selenium-java-2.53.1.jar file. Include these two JAR files in your eclipse project an external Java library.Because of the version conflict issues between Selenium and Firefox, once you run code with a particular version, turn off the automatic update download and installation option in Firefox.
extractDataWithSelenium(String) that takes a String as a parameter, which eventually is the URL from where we are going to extract data. There can be many different types of data that we can extract from URLs, such as the title, the headers, and the values in a selection drop-down box. This recipe only concentrates on extracting the text part of the webpage: public String extractDataWithSelenium(String url){
WebDriver driver = new FirefoxDriver();
get() method of the WebDriver object by passing the URL: driver.get("http://cogenglab.csd.uwo.ca/rushdi.htm");
xpath, where the value of id is content:


findElement() method. This method returns a WebElement object. Create a WebElement object named webElement to hold the returned value: WebElement webElement = driver.findElement(By.xpath("//*
[@id='content']"));
WebElement object has a method named getText(). Call this method to retrieve the text of the web page, and put the text into a String variable as follows:String text = (webElement.getText());
}
The complete code segment with the driver main() method for the recipe looks like the following:
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.firefox.FirefoxDriver;
public class TestSelenium {
public String extractDataWithSelenium(String url) {
WebDriver driver = new FirefoxDriver();
driver.get("http://cogenglab.csd.uwo.ca/rushdi.htm");
WebElement webElement = driver.findElement(By.xpath("//*
[@id='content']"));
System.out.println(webElement.getText());
return url;
}
public static void main(String[] args){
TestSelenium test = new TestSelenium();
String webData = test.extractDataWithSelenium
("http://cogenglab.csd.uwo.ca/rushdi.htm");
//process webData
}
}
Selenium and Firefox have compatibility issues. Some Selenium versions do not work with some Firefox versions. The recipe provided here works fine with the versions mentioned in the recipe. But it does not have any guarantee that it will work with other Selenium or Firefox versions.
Because of the version conflict issues between Selenium and Firefox, once you run a code with a particular version of the both, turn off the automatic update download and installation option in Firefox.