Book Image

Using OpenRefine

Book Image

Using OpenRefine

Overview of this book

Data today is like gold - but how can you manage your most valuable assets? Managing large datasets used to be a task for specialists, but the game has changed - data analysis is an open playing field. Messy data is now in your hands! With OpenRefine the task is a little easier, as it provides you with the necessary tools for cleaning and presenting even the most complex data. Once it's clean, that's when you can start finding value. Using OpenRefine takes you on a practical and actionable through this popular data transformation tool. Packed with cookbook style recipes that will help you properly get to grips with data, this book is an accessible tutorial for anyone that wants to maximize the value of their data. This book will teach you all the necessary skills to handle any large dataset and to turn it into high-quality data for the Web. After you learn how to analyze data and spot issues, we'll see how we can solve them to obtain a clean dataset. Messy and inconsistent data is recovered through advanced techniques such as automated clustering. We'll then show extract links from keyword and full-text fields using reconciliation and named-entity extraction. Using OpenRefine is more than a manual: it's a guide stuffed with tips and tricks to get the best out of your data.
Table of Contents (13 chapters)
Using OpenRefine
Credits
Foreword
About the Authors
About the Reviewers
www.PacktPub.com
Preface
Index

Index

A

  • anchors
    • about / Anchors

C

  • case transformations / Recipe 5 – using simple cell transformations
  • cells
    • clustering / Recipe 3 – clustering similar cells
  • cell values
    • transforming / Recipe 4 – transforming cell values
  • character classes
    • about / Character classes
  • choices
    • about / Choices
  • Clojure / Recipe 4 – transforming cell values
  • clustering / Recipe 3 – clustering similar cells
  • columns
    • manipulating / Recipe 4 – manipulating columns
    • collapsing / Collapsing and expanding columns
    • expanding / Collapsing and expanding columns
    • moving / Moving columns around
    • renaming / Renaming and removing columns
    • removing / Renaming and removing columns
    • transposing / Recipe 7 – transposing rows and columns
    • transposing, into rows / Recipe 7 – transposing rows and columns
  • consecutive whitespace / Recipe 5 – using simple cell transformations
  • customized facets
    • about / Customized facets

D

  • data
    • exploring / Recipe 3 – exploring your data
    • sorting / Recipe 1 – sorting data
    • faceting / Recipe 2 – faceting data
    • splitting, across columns / Recipe 6 – splitting data across columns
  • data cleaning
    • about / Introducing OpenRefine
    / Recipe 6 – removing matching rows
  • data profiling
    • about / Introducing OpenRefine
    / Recipe 6 – removing matching rows
  • DBpedia Spotlight service / Recipe 5 – extracting named entities
  • derived columns
    • adding / Recipe 5 – adding derived columns
  • Digital Enterprise Research Institute (DERI) / Recipe 2 – installing extensions
  • distance function / Recipe 3 – clustering similar cells
  • duplicate facet
    • detecting / Recipe 3 – detecting duplicates

F

  • faceting
    • about / Recipe 2 – faceting data
    • star or flag used / Faceting by star or flag
  • facets
    • about / Recipe 2 – faceting data
    • types / Recipe 2 – faceting data
    • text facets / Text facets
    • numeric facets / Numeric facets
    • customized facets / Customized facets
  • file formats, OpenRefine
    • about / File formats supported by OpenRefine
  • Freebase Gridworks
    • about / Recipe 1 – reconciling values with Freebase

G

  • General Refine Expression Language (GREL)
    • about / Customized facets
    / Recipe 4 – transforming cell values
  • GREL
    • about / General Refine Expression Language (GREL)
    • data, transforming / Transforming data
    • custom facets, creating / Creating custom facets
    • used, in problem solving / Solving problems with GREL
  • groups
    • about / Groups

H

  • HyperText Markup Language (HTML) code / Recipe 5 – using simple cell transformations

I

  • Interactive Data Transformation tools (IDTs) / Introducing OpenRefine

J

  • Jython (Python implemented in Java) / Recipe 4 – transforming cell values

K

  • key collision / Recipe 3 – clustering similar cells
  • keying function / Recipe 3 – clustering similar cells

L

  • Library of Congress Subject Headings (LCSH)
    • about / Recipe 3 – adding a reconciliation service
  • Linked Data
    • reconciling with / Recipe 4 – reconciling with Linked Data
    • principles / Recipe 4 – reconciling with Linked Data
  • Linux
    • OpenRefine, installing / Linux

M

  • Mac
    • OpenRefine, installing / Mac
  • matching rows
    • removing / Recipe 6 – removing matching rows
  • memory, OpenRefine
    • on Windows / Recipe 7 – going for more memory
    • on Mac / Mac
    • on Linux / Linux
  • multi-valued cells
    • handling / Recipe 1 – handling multi-valued cells

N

  • Named-Entity Recognition (NER) extension / Recipe 2 – installing extensions
  • Named-Entity Recognition extension
    • URL / Recipe 5 – extracting named entities
  • named entities, Powerhouse Museum dataset
    • extracting / Recipe 5 – extracting named entities
  • nearest neighbor / Recipe 3 – clustering similar cells
  • numeric facets
    • about / Numeric facets

O

  • OpenRefine
    • about / Introducing OpenRefine
    • installing / Recipe 1 – installing OpenRefine
    • downloading / Recipe 1 – installing OpenRefine
    • installing, on Windows / Windows
    • installing, on Mac / Mac
    • installing, on Linux / Linux
    • project, creating / Recipe 2 – creating a new project
    • supported file formats / File formats supported by OpenRefine
    • data, exploring / Recipe 3 – exploring your data
    • columns, manipulating / Recipe 4 – manipulating columns
    • project history, using / Recipe 5 – using the project history
    • project, exporting / Recipe 6 – exporting a project
    • memory, allocating / Recipe 7 – going for more memory
    • data, sorting / Recipe 1 – sorting data
    • data, faceting / Recipe 2 – faceting data
    • duplicate facet, detecting / Recipe 3 – detecting duplicates
    • text filter, applying / Recipe 4 – applying a text filter
    • simple cell transformations, using / Recipe 5 – using simple cell transformations
    • matching rows, removing / Recipe 6 – removing matching rows
    • multi-valued cells, handling / Recipe 1 – handling multi-valued cells
    • rows and records mode, alternating / Recipe 2 – alternating between rows and records mode
    • similar cells, clustering / Recipe 3 – clustering similar cells
    • cell values, transforming / Recipe 4 – transforming cell values
    • derived columns, adding / Recipe 5 – adding derived columns
    • data, splitting across columns / Recipe 6 – splitting data across columns
    • rows, transposing / Recipe 7 – transposing rows and columns
    • columns, transposing / Recipe 7 – transposing rows and columns
    • columns, transposing into rows / Recipe 7 – transposing rows and columns
  • OpenRefine extensions
    • installing / Recipe 2 – installing extensions
    • for Windows / Recipe 2 – installing extensions
    • for Mac OS X / Recipe 2 – installing extensions
    • for Linux / Recipe 2 – installing extensions

P

  • Powerhouse collection / Text facets
  • Powerhouse Museum dataset
    • values, reconciling with Freebase / Recipe 1 – reconciling values with Freebase
    • reconciliation service, adding / Recipe 3 – adding a reconciliation service
    • Linked Data, reconciling with / Recipe 4 – reconciling with Linked Data
    • named entities, extracting / Recipe 5 – extracting named entities
  • project
    • creating / Recipe 2 – creating a new project
    • exporting / Recipe 6 – exporting a project
  • project history
    • accessing / Recipe 5 – using the project history

Q

  • quantifiers
    • about / Quantifiers
    • plus sign + / Quantifiers
    • asterisk * / Quantifiers
    • question mark ? / Quantifiers

R

  • RDF
    • about / Recipe 3 – adding a reconciliation service
  • RDF extension
    • installing / Recipe 2 – installing extensions
    • downloading / Recipe 2 – installing extensions
  • reconciliation service
    • adding / Recipe 3 – adding a reconciliation service
  • record
    • about / Recipe 2 – alternating between rows and records mode
  • records mode
    • alternating, with rows mode / Recipe 2 – alternating between rows and records mode
  • regular expressions
    • overview / Overview
  • regular expressions, for text patterns
    • about / Regular expressions for text patterns
    • character classes / Character classes
    • quantifiers / Quantifiers
    • anchors / Anchors
    • choices / Choices
    • groups / Groups
  • row
    • about / Recipe 2 – alternating between rows and records mode
    • transposing / Recipe 7 – transposing rows and columns
  • row mode
    • alternating, with records mode / Recipe 2 – alternating between rows and records mode
  • rows
    • reordering / Reordering rows

S

  • Semantic Web / Recipe 4 – reconciling with Linked Data
  • shorthand character classes
    • about / Character classes
  • simple cell transformations
    • using / Recipe 5 – using simple cell transformations
  • SPARQL
    • about / Recipe 3 – adding a reconciliation service

T

  • tab separated values (TSV) / Text facets
  • text facets
    • about / Text facets
  • text filter
    • applying / Recipe 4 – applying a text filter
  • timeline facet / Numeric facets

U

  • Unicode char-code facet
    • about / Customized facets

V

  • values, Powerhouse Museum dataset
    • reconciling, Freebase used / Recipe 1 – reconciling values with Freebase

W

  • whitespace
    • trimming / Recipe 5 – using simple cell transformations
  • Windows
    • OpenRefine, installing / Windows