For the previous recipe, the embedded domain-specific language (EDSL) used for the query gets converted to SPARQL, the query language for many linked data systems. If you squint just right at the query, it looks kind of like a
SPARQL WHERE clause. It's a simple query, but one nevertheless.
And this worked great when we had access to the raw data in our own triple store. However, if we need to access a remote SPARQL end-point directly, it's more complicated.
For this recipe, we'll query DBPedia (http://dbpedia.org) for information about the United Arab Emirates' currency, the dirham. DBPedia extracts structured information from Wikipedia (the summary boxes) and re-publishes it as RDF. Just as Wikipedia is a useful first-stop for humans to get information about something, DBPedia is a good starting point for computer programs gathering data about a domain.
First, we need to make sure the dependencies are listed in our project.clj file:
:dependencies [[org.clojure/clojure "1.4.0"]
[incanter/incanter-core "1.4.1"]
[edu.ucdenver.ccp/kr-sesame-core "1.4.5"]
[org.clojure/tools.logging "0.2.4"]
[org.slf4j/slf4j-simple "1.7.2"]]Then, load the Clojure and Java libraries that we'll use.
(require '(clojure.java [io :as io]))
(require '(clojure [xml :as xml]
[pprint :as pp]
[zip :as zip]))
(use 'incanter.core
'[clojure.set :only (rename-keys)]
'edu.ucdenver.ccp.kr.kb
'edu.ucdenver.ccp.kr.rdf
'edu.ucdenver.ccp.kr.sparql
'edu.ucdenver.ccp.kr.sesame.kb)
(import [java.io File]
[java.net URL URLEncoder]) As we work through this, we'll define a series of functions. Finally, we'll create one function, load-data, to orchestrate everything, and we'll finish by calling it.
We have to create a Sesame triple store and initialize it with the namespaces that we'll use. For both of these we'll use the
kb-memstoreandinit-kbfunctions that we discussed in the previous recipe. We define a function that takes a URI for a subject in the triple store and constructs a SPARQL query that returns at most 200 statements about that. It filters out any statements with non-English strings for objects, but it allows everything else through:(defn make-query "This creates a query that returns all the triples related to a subject URI. It does filter out non-English strings." ([subject kb] (binding [*kb* kb *select-limit* 200] (sparql-select-query (list '(~subject ?/p ?/o) '(:or (:not (:isLiteral ?/o)) (!= (:datatype ?/o) rdf/langString) (= (:lang ?/o) ["en"])))))))Now that we have the query, we'll need to encode it into a URL to retrieve the results:
(defn make-query-uri "This constructs a URI for the query." ([base-uri query] (URL. (str base-uri "?format=" (URLEncoder/encode "text/xml") "&query=" (URLEncoder/encode query)))))Once we get a result, we'll parse the XML file, wrap it in a zipper, and navigate to the first result. All this will be in a function that we'll write in a minute. Right now, the next function will take that first result node and return a list of all of the results:
(defn result-seq "This takes the first result and returns a sequence of this node, plus all the nodes to the right of it." ([first-result] (cons (zip/node first-result) (zip/rights first-result))))The following set of functions takes each result node and returns a key-value pair (
result-to-kv). It usesbinding-strto pull the results out of the XML file. Thenaccum-hashfunction pushes those key-value pairs into a map. Keys that occur more than once have their values accumulated in a vector.(defn binding-str "This takes a binding, pulls out the first tag's content, and concatenates it into a string." ([b] (apply str (:content (first (:content b)))))) (defn result-to-kv "This takes a result node and creates a key-value vector pair from it." ([r] (let [[p o] (:content r)] [(binding-str p) (binding-str o)]))) (defn accum-hash "This takes a map and key-value vector pair and adds the pair to the map. If the key is already in the map, the current value is converted to a vector and the new value is added to it." ([m [k v]] (if-let [current (m k)] (assoc m k (conj current v)) (assoc m k [v]))))For the last utility function, we'll define
rekey. This will convert the keys of a map based on another map:(defn rekey "This just flips the arguments for clojure.set/rename-keys to make it more convenient." ([k-map map] (rename-keys (select-keys map (keys k-map)) k-map)))Now, let's add a function that takes a SPARQL endpoint and a subject, and returns a sequence of result nodes. This will use several of the functions we've just defined.
(defn query-sparql-results "This queries a SPARQL endpoint and returns a sequence of result nodes." ([sparql-uri subject kb] (->> kb ;; Build the URI query string. (make-query subject) (make-query-uri sparql-uri) ;; Get the results, parse the XML, ;; and return the zipper. io/input-stream xml/parse zip/xml-zip ;; Find the first child. zip/down zip/right zip/down ;; Convert all children into a sequence. result-seq)))Finally, we can pull everything together. Here's
load-data:(defn load-data "This loads the data about a currency for the given URI." [sparql-uri subject col-map] (->> ;; Initialize the triple store. (kb-memstore) init-kb ;; Get the results. (query-sparql-results sparql-uri subject) ;; Generate a mapping. (map result-to-kv) (reduce accum-hash {}) ;; Translate the keys in the map. (rekey col-map) ;; And create a dataset. to-dataset))Now let's use it. We can define a set of variables to make it easier to reference the namespaces that we'll use. We'll use them to create a mapping to column names:
(def rdfs "http://www.w3.org/2000/01/rdf-schema#") (def dbpedia "http://dbpedia.org/resource/") (def dbpedia-ont "http://dbpedia.org/ontology/") (def dbpedia-prop "http://dbpedia.org/property/") (def col-map {(str rdfs 'label) :name, (str dbpedia-prop 'usingCountries) :country (str dbpedia-prop 'peggedWith) :pegged-with (str dbpedia-prop 'symbol) :symbol (str dbpedia-prop 'usedBanknotes) :used-banknotes (str dbpedia-prop 'usedCoins) :used-coins (str dbpedia-prop 'inflationRate) :inflation})We call
load-datawith the DBPedia SPARQL endpoint, the resource we want information about (as a symbol), and the column map:user=> (load-data "http://dbpedia.org/sparql" #_=> (symbol (str dbpedia "/United_Arab_Emirates_dirham")) #_=> col-map) [:used-coins :symbol :pegged-with :country :inflation :name :used-banknotes] ["2550" "إ.د" "U.S. dollar = 3.6725 dirhams" "United Arab Emirates" "14" "United Arab Emirates dirham" "9223372036854775807"]
The only part of this recipe that has to do with SPARQL, really, is the function make-query. It uses the function sparql-select-query to generate a SPARQL query string from the query pattern. This pattern has to be interpreted in the context of the triple store that has the namespaces defined. This context is set using the binding command. We can see how this function works by calling it from the REPL by itself:
user=> (println
#_=> (make-query
#_=> (symbol (str dbpedia "/United_Arab_Emirates_dirham"))
#_=> (init-kb (kb-memstore))))
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
SELECT ?p ?o
WHERE { <http://dbpedia.org/resource/United_Arab_Emirates_dirham> ?p ?o .
FILTER ( ( ! isLiteral(?o)
|| ( datatype(?o) != <http://www.w3.org/1999/02/22-rdf-syntax-ns#langString> )
|| ( lang(?o) = "en" ) )
)
} LIMIT 200The rest of the recipe is concerned with parsing the XML format of the results, and in many ways it's similar to the last recipe.