In this section, you will learn how to interact with the Web through HTTP requests, both for getting data and posting data to the Web. You will learn about sending and getting requests to and from websites and also analyzing those responses.
Start by downloading and installing the Requests.jl package of Julia, which is available at Pkg.add("Requests").
Make sure that you have an active Internet connection while reading and using the code in the recipe, as it deals with interacting with live websites on the Web. You can experiment with this recipe on the website http://httpbin.org , as it is designed especially for such experiments and tutorials.
This is how you use the Requests.jl package and import the required modules:
Start by importing the package:
Pkg.add("Requests")Next, import the necessary modules from the package for quick use. The modules that will be used in this recipe are
get,post,put, anddelete. So, this is how to import the modules:import Requests: get, post
Here, you will learn how to interact with the Web through the HTTP protocol and requests. You will also learn how to send and receive data, and autofill forms on the Internet, through HTTP requests.
The
GETrequest is used to request data from a specified web resource. So, this is how we send theGETrequest to a website:get("url of the website")To get requests from a specific web page inside the website, the query parameter of the
GETcommand can be used to specify the web page. This is how you do it:get("url of the website"; query = Dict("title" => "page number/page name"))Timeouts can also be set for the
GETrequests. This would be useful for identifying unresponsive websites/web pages. The timeout parameter in theGETrequest takes a particular numeric value to be set as the timeout threshold; above this, if the server does not return any data, a timeout request will be thrown. This is how you set it:get("url of the website"; timeout = 0.5)Here,
0.5means 50 ms.
Some websites redirect users to different web pages or sometimes to different websites. So, to avoid getting your request repeatedly redirected, you can set the
max_redirectsandallow_redirectsparameters in theGETrequest. This is how they can be set:get("url of the website"; max_redirects = 4)Now, to set the
allow_redirectsparameter preventing the site from redirecting yourGETrequests:get("url of the website"; allow_redirects = false)This would not allow the website to redirect your
GETrequest. If a redirect is triggered, it throws an error.
The
POSTrequest submits data to a specific web resource. So, this is how to send a post request to a website:post("url of the website")Data can be sent to a web resource through the
POSTrequest by adding it into the data parameter in thePOSTrequest statement:post("url of the website"; data = "Data to be sent")Data for filling forms on the Web also can be sent through the
POSTrequest through the same data parameter, but the data should now be sent in the form of a Julia dictionary data structure:post("url of the website"; data = Dict(First_Name => "abc", Last_Name => "xyz" ))Data such as session cookies can also be sent through the
POSTrequest by including the session details inside a Julia Dictionary and including it in thePOSTrequest as the cookies parameter:post("url of the website"; cookies = Dict("sessionkey" => "key"))Files can also be sent to web resources through the
POSTrequests. This can be done by including the files in the files parameter of thePOSTrequest:file = "xyz.jl" post("url of the website"; files = [FileParam(file), "text/julia", "file_name", "file_name.jl"])
There are more HTTP requests with which you can interact with web resources such as the PUT and DELETE requests. All of them can be studied in detail from the documentation for the Requests.jl package, which is available at
https://github.com/JuliaWeb/Requests.jl
.