Miscellaneous Jsoup options (Should know)
Usually, developers only work on Jsoup with default options, unaware that it provides various useful options. This recipe will acquaint you with some common-use options.
How to do it...
How to work with connection objects:
Setting
userAgent: It is very important to always specifyuserAgentwhen sending HTTP requests. What if the web page displays some information differently on different browsers? The result of parsing might be different.Document doc = Jsoup.connect(url).userAgent("Mozilla/5.0 (Windows NT 6.1)").get();
Especially when using Jsoup in Android, you must always specify a user agent; otherwise, it won't work properly.
When forced to work with different content types:
Document doc = Jsoup.connect(url).ignoreContentType(true).get();
By default, Jsoup only allows working with HTML and XML content type and throws exceptions for others. So, you will need to specify this properly in order to work with other content types, such as RSS, Atom, and so on.
Configure a connection timeout:
Document doc = Jsoup.connect(url).timeout(5000).get();
The default timeout for Jsoup is 3000 milliseconds (three seconds). Zero indicates an infinite timeout.
Add a parameter request to the connection:
Document doc = Jsoup.connect(url).data("author", "Pete Houston").get();
In dynamic web, you need to specify a parameter to make a request; the
data()method works for this purpose.Note
Please refer to the following link for more information:
http://jsoup.org/apidocs/org/jsoup/Connection.html#data(java.lang.String, java.lang.String)
Sometimes, the request is
post.Document doc = Jsoup.connect(url).data("author", "Pete Houston").post();
Setting the HTML output of the
Documentclass.This option works through the
Document.OutputSettingsclass.Note
Please refer to the following link for more information:
http://jsoup.org/apidocs/org/jsoup/nodes/Document.OutputSettings.html
This class outputs HTML text in a neat format with the following options:
Character set: Get/set document charset
Escape mode: Get/set escape mode of HTML output
Indentation: Get/set indent amount for pretty printing (by space count)
Outline: Enable/disable HTML outline mode
Pretty print: Enable/disable pretty printing mode
For example, display the HTML output with; charset as
utf-8and the indentation amount as four spaces, enable the HTMLoutlinemode, and enable pretty printing:Document.OutputSettings settings = new Document.OutputSettings(); settings.charset("utf-8").indentAmount(4).outline(true).prettyPrint(true); Document doc = …// create DOM object somewhere.doc.outputSettings(settings); System.out.println(doc.html());After setting the output format to
Document, the content ofDocumentis processed into the according format; call theDocument.html()method for output result.Configure the parser type.
Jsoup provides two parser types: HTML parser and XML parser.
By default, it uses HTML parser. However, if you are going to parse XML such as RSS or Atom, you should change the parser type to XML parser or it will not work properly.
Document doc = Jsoup.connect(url).parser(Parser.xmlParser()).get();
There's more...
The previously mentioned options in Jsoup are important ones that the developers should know and make use of.
However, there are several more that you can try:
DataUtil: This provides methods to load a file, or stream and transform into aDocumentobject. To know more about this option, go to the following location:StringUtil: This provides methods to handle strings; for example, to search in array, join string array, or test string. To know more about this option, go to the following location:Validate: This provides methods to test objects, such as test empty, test null, and so on. To know more about this option, go to the following location:Entities: This provides methods to test or get HTML entities. To know more about this option, go to the following location:Parser: This provides methods to parse HTML intoDocument. To know more about this option, go to the following location: