The combination of math, computer programming, and domain knowledge is what makes data science so powerful. Often, it is difficult for a single person to master all three of these areas. That's why it's very common for companies to hire teams of data scientists instead of a single person. Let's look at a few powerful examples of data science in action and their outcome.
Social security claims are known to be a major hassle for both the agent reading it and for the person who wrote the claim. Some claims take over 2 years to get resolved in their entirety, and that's absurd! Let's look at what goes into a claim:
Sample social security form
Not bad. It's mostly just text, though. Fill this in, then that, then this, and so on. You can see how it would be difficult for an agent to read these all day, form after form. There must be a better way!
Well, there is. Elder Research Inc. parsed this unorganized data and was able to automate 20% of all disability social security forms. This means that a computer could look at 20% of these written forms and give its opinion on the approval.
Not only that, the third-party company that is hired to rate the approvals of the forms actually gave the machine-graded forms a higher grade than the human forms. So, not only did the computer handle 20% of the load, it, on average, did better than a human.
Before I get a load of angry e-mails claiming that data science is bringing about the end of human workers, keep in mind that the computer was only able to handle 20% of the load. That means it probably performed terribly for 80% of the forms! This is because the computer was probably great at simple forms. The claims that would have taken a human minutes took the computer seconds to compute. But these minutes add up, and before you know it, each human is being saved over an hour a day!
Forms that might be easy for a human to read are also likely easy for the computer. It's when the form becomes very terse or when the writer starts deviating from usual grammar that the computer starts to fail. This model is great because it lets the humans spend more time on those difficult claims and gives them more attention without getting distracted by the sheer volume of papers.
A dataset shows the relationship between the money spent in the categories of TV, radio, and newspaper. The goal is to analyze the relationship between the three different marketing mediums and how it affects the sale of a product. Our data is in the form of a row and column structure. Each row represents a sales region and the columns tell us how much money was spent on each medium and the profit achieved in that region.
Note
Usually, the data scientist must ask for units and scale. In this case, I will tell you that TV, radio, and newspaper are measured in "thousands of dollars" and sales in "thousands of widgets sold". This means that in the first region, $230,100 was spent on TV advertising, $37,800 on radio advertising, and $69,200 on newspaper advertising. In the same region, 22,100 items were sold.
Advertising budgets
For example, in the third region, we spent $17,200 on TV advertising and sold 9,300 widgets.
If we plot each variable against sales, we get the following graphs:
import seaborn as sns sns.pairplot(data, x_vars=['TV','Radio','Newspaper'], y_vars='Sales')
Graphs of advertising budgets
Note how none of these variables form a very strong line and, therefore, might not work well to predict sales (on their own). TV comes closest in forming an obvious relationship, but still even that isn't great. In this case, we will have to form a more complex model than the one we used in the spawner-recruiter model and combine all three variables in order to model sales.
This type of problem is very common in data science. In this example, we are attempting to identify key features that are associated with the sales of a product. If we can isolate these key features, then we can exploit these relationships and change how much we spend on advertising in different places with the hopes of increasing our sales.
Looking for a job in data science? Great, let me help. In this case study, I have "scraped" (taken from the Web) 1,000 job descriptions for companies actively hiring data scientists (as of January 2016). The goal here is to look at some of the most common keywords people use in their job descriptions.
An example of data scientist job listings.
(Note the second one asking for core Python libraries; we talk about these later on in this book)
import requests # used to grab data from the web from BeautifulSoup import BeautifulSoup # used to parse HTML from sklearn.feature_extraction.text import CountVectorizer # used to count number of words and phrases (we will be using this module a lot)
The first two imports are used to grab web data from the website, Indeed.com, and the third import is meant to simply count the number of times a word or phrase appears.
texts = []
# hold our job descriptions in this list
for index in range(0,1000,10): # go through 100 pages of indeed
page = 'indeed.com/jobs?q=data+scientist&start='+str(index)
# identify the url of the job listings
web_result = requests.get(page).text
# use requests to actually visit the url
soup BeautifulSoup(web_result)
# parse the html of the resulting page
for listing in soup.findAll('span', {'class':'summary'}:
# for each listing on the page
texts.append(listing.text)
# append the text of the listing to our listOkay, before I lose you, all that this loop is doing is going through 100 pages of job descriptions, and for each page, grabbing each job description. The important variable here is texts, which is a list of over 1,000 job descriptions:
type(texts) # == list vect = CountVectorizer(ngram_range=(1,2), stop_words='english') # Get basic counts of one and two word phrases matrix = vect.fit_transform(texts) # fit and learn to the vocabulary in the corpus print len(vect.get_feature_names()) # how many features are there # There are 11,293 total one and two words phrases in my case!!
I have omitted some code here, but it exists in the GitHub repository for this book. The results are as follows (represented as the phrase, and then the number of of times it occurred):
experience 320 machine 306 learning 305 machine learning 294 techniques 266 statistical 215 team 197 analytics 173 business 167 statistics 159 algorithms 152 datamining 149 software 144 applied 141 programming 132 understanding 127 world 127 research 125 datascience 123 methods 122 join 122 quantitative 122 group 121 real 120 large 120
Notable things:
Machine learning and experience are at the top of the list. Experience comes with practice. A basic idea of machine learning comes with this book.
These words are followed closely by statistical words implying knowledge of math and theory.
The word
teamis very high up, implying that you will need to work with a team of data scientists; you won't be a lone wolf.Computer science words such as
algorithmsandprogrammingare prevalent.The words
techniques,understanding, andmethodsimply a more theoretical approach, ambivalent to any single domain.The word
businessimplies a particular problem domain.
There are many interesting things to note about this case study but the biggest take away is that there are many key words and phrases that make up a data science role. It isn't just math, coding, or domain knowledge; it truly is the combination of these three ideas (whether exemplified in a single person or across a multiperson team) that makes data science possible and powerful.