Practical Application of Data Science and Machine Learning

Using Data Science to Solve a Real-World Problem

By Linwood Creekmore III

Lots of people write or publish data science tutorials. Yet, most of these tutorials -- including those written by folks who hold the title of data scientist -- fail to provide examples where machine learning solves an everyday problem. New comers entering the field will struggle initially because tutorials and education rarely align with the problems (or data) you face in the workplace.  

This real world data science tutorial hopes to address this issue by creating a machine learning pipeline to solve a problem you would face as a data scientist in a real work environment.  We will cover, in detail, the steps to train a model that finds "relevant" or "non-relevant" content based on the requirements of a customer. In the process, you will learn a few tricks on how to pick and tune the best model to achieve optimum performance. And, most importantly, we close out by creating a web service that delivers our machine learning goodness to customers. This is a crucial step because data scientists need the skills to deliver data products to both technical and non-technical audiences.  By following the steps in this tutorial, you will complete the entire cycle of the data science pipeline. 

The Scenario and Goal of this Tutorial

You did it! You landed your first job as a data scientist on a cross functional team that does consulting work for clients. The senior partners of the company were so impressed with your resume, they gave you a lead position on a major account. Our fictitious customer is a sports agency that specializes in representing professional American football quarterbacks (QBs) during contract negotiations. They need a system to track news articles of current or prospective clients, so they hired our firm to develop an application with a tailored feed of news.

As the data scientist, you must build a model that can process the content of a news article and determine whether it is relevant to the customer. To make matters interesting, this client is heavily involved in the development process so has asked to see a prototype of the model in action before it is integrated into the application. Additionally, our developer teammates need an API endpoint to access the output of our model for testing in the application. 

Note: This scenario is representative of a task you would have working as a data scientist in industry. While there are lots of difficult-sounding problems out there, the workplace problems are usually less flashy.

Based on our scenario, this tutorial solves a classification problem. Classification is the process of predicting the category of some data by learning the differences in features between different categories. For example, we could classify days as hot or cold based on the temperature reading at midday. By learning the differences in temperatures associated with hot or cold days, a model would eventually develop enough "intuition" to organize days into two buckets that correspond to hot and cold categories. In a similar fashion, we are going to create a model to organize news articles into two categories.  Our example is trivial but any time you need to separate relevant data from irrelevant data, the tools you learn here will help. Here's a graphic showing what our model plans to do:

Our journey to build and deploy a machine learning model will use real, unclean data;  the data is not overly "dirty" but it will be a welcome change for those who need to expand beyond the Titanic, Iris, or Auto MPG datasets used in most data science tutorials. I did not provide a copy of the data, but I will provide the necessary code so that you can access and prepare the data yourself.

Below are the steps we will cover in this tutorial:

1. Creating the environment.

2. Pulling, cleaning, and loading data (cleaning handled by custom function).

3. Modeling the data.

4. Testing and scoring data against multiple models to pick the best one.

5. Tuning the model to improve performance.

6. Building a web application and API endpoint to expose your model to technical and non-technical audiences.

Create the Environment

Let's start by creating an environment with the tools we will need. But first, let's briefly cover why you need a virtual environment.     

Chances are, this tutorial is one of many tutorials or projects you are completing to improve your data science skills. You also have realized by now that different projects require different tools; one project may require a library that only runs in Python 2 and another may require a library that is only used in Python 3.  By using virtual environments, you can create virtual spaces on a single computer that support both projects, eliminating the potential of version conflicts or dealing with serious errors because you corrupted a system dependency. For more details on virtual environments, read this blog post. 

Note: Using virtual environments has another, more important benefit. Sharing! Outside of creating an isolated environment for testing, we can export our virtual environment into easily shared formats, making it easier for others to recreate our work.   

You will use my exported file to build a mirror environment with all the tools I used to make this tutorial. I recommend using the Anaconda Python distribution platform for set up. Why? It's free, easy to install, and widely used in the data science community.  Moreover, some Python libraries have OS-level dependencies which fail to build when using pip install; this leads to lots of frustration and hand wringing. Avoid these challenges and use Anaconda because it makes set up easy. Visit this page and follow the install instructions for your system. 

Next, clone or download the GitHub project for this tutorial which has the yml file to recreate my environment.  Navigate to the base folder in the command line and execute these commands: 

If you did not receive an error, you have successfully recreated my environment. If you received errors or decided not to use Anaconda, you'll need a bit of luck and determination to get things installed. I recommend using Google and StackOverflow to search for and solve the error messages you receive.

Note: Developing your Google and StackOverflow skills is something you should work on as well. This will be a critical skillset for your data science job. Why? Often, people have already encountered and solved the problem you're facing. Don't reinvent the wheel!

Access and Clean the Data

With environment setup complete, we shift our focus to accessing data.  Let's assume our customer prefers to use ESPN news, so we will get our training data from their website. I built up a corpus of 1100+ news articles from ESPN covering the 2017-2018 football season.  This website provides a snapshot of the ESPN articles used to build the corpus. 

I decided to use the requests library to build my corpus. By changing the month and year in the linked URL above, you could retrieve a list of urls covering news on the NFL for that specific timeframe. Then, you just need to cycle through each URL and download the text to those stories. 

Note: If you are doing a topic outside of NFL news, make sure you have a way to access the data and build a respectable training set. I suggest having at least 500 samples that have a near even split of the desired classes you're trying to predict.

To pull the ESPN articles, I used the multiprocessing library and a custom script to download, clean, and create 100-row parquet files of NFL news stories on local disk. My script works on any web page so feel free to use it if you switch to another topic.  Below is an example where my script pulls data from three urls: 

This will return a Python dictionary of clean, extracted data from the article. We now have key value pairs for title, author, publish date, text, and several other data fragments that would be useful for machine learning. I processed over 1,000 urls and built a pandas dataframe of the output. The first 3 rows look like this: 

Now we're ready to label our data. 

Label the Data: Find Which Articles Discuss Quarterbacks

There are several approaches to labeling this data.  We could label each article manually based on the presence of the word "quarterback" (easy) or we could use other features (such as names) to automatically label our data and hope that our model learns to associate the word "quarterback" to our target class (this is a more useful demonstration of machine learning).  To keep this interesting, we'll opt for the second approach.

The rules are simple; if an article mentions a QB's name,  it receives a label of "1" (positive class). If it doesn't mention a QB name, it receives a "0" label (negative class).  With simple rules like that, all we need is a machine-readable list of QB names.    

Luckily, ESPN has a webpage with QB names in a table. After downloading and parsing this table,  we use pandas and regular expressions (regex) to test whether an article mentioned a QB's name and labeled accordingly. Note, you will need to install some extra pandas dependencies for the pandas.read_html method to work. Review the pandas documentation for help. Here is the code to download the table of QB names:   

And here is the regex to isolate the QB names and filter our dataframe. 

Our rule set should have created a new label column. This column will be our ground truth and organizes the articles into our target categories. The automated approach we used was quick, but it is not representative of what you will face in real project. In an ideal world, we would manually label these articles with a "0" or "1" or have a pre-labeled data set.  Having labeled data is the biggest challenge to using supervised learning methods; it takes considerable time to build up a properly labeled dataset.

Note: If you are using different data for this tutorial, consider labeling it manually to achieve the best results. 

Let's preview the first 3 rows of our new label column:

Picking the Best Model

At this point, we have all the tools and data we need to build our model. We will use scikit-learn to train our model, given that it's Python's most popular machine learning library.  Besides being widely used and having extensive documentation, scikit-learn has many implementations of classification, regression and clustering algorithms. Because of the abundance of algorithms, this library is well suited to solve just about any machine learning problem. But with that variety also comes a curse. With so many models to choose from, which one should you use?  

This decision of which model to use is often influenced by past experiences. Data scientists, biased by formal education, online tutorials, or prior work successes, maintain a "go to" list of algorithms to solve data analytics problems. Why? In simple terms, we use a smaller subset of models because familiarity breeds preference. But sometimes, another algorithm or model may work better for your data. This tutorial introduces a repeatable process (and reusable code) to overcome our model biases and test new algorithms. 

The repeatable process focuses on sampling the performance of multiple algorithms to see which one performs best. To get started, we need a seed of algorithms to try. The flow chart on machine learning from scikit-learn and page on supervised learning methods provide a selection of models to test.  To cycle through and test models easily, we build a custom scikit-learn pipeline to preprocess and convert our documents into the multidimensional arrays scikit-learn uses to train its models. 

We also want to make sure we adhere to best practices while implementing machine learning pipelines. To ensure that our models are generalizable, we include a 10-fold cross-validation step which evaluates our model on different combinations of test/training splits.

Note: Cross validation is useful for avoiding problems such as overfit and gives a more realistic idea for how your model will perform on unseen data. 

Let's take all the guidance above and create a generic function to test our models: 

Next, we only need to make a list of all the algorithms we want to try. Using the f1 score, standard deviation, and time as measures,  we will compare the overall performance of each model.  But, what are we looking for? The perfect model will have a high f1 score, low compute time, and very low standard deviation.

Because we are applying the same function across a list of objects where the results of one operation do not depend on the results of another, this process is considered embarrassingly parallel.

Note: Embarrassingly parallel processes can take advantage of multiprocessing to speed up computation times. 

While testing the multiprocessing workflow, there was a small hiccup which accounts for splitting our estimators into two lists. The Naive Bayes and Deep Learning family of scikit-learn models would not process in parallel so I iterated over these models and appended their scores to the array with all the scores. This code will train 16 algorithms on our data:

Now we build a partial function to run through our list of scikit-learn estimators in parallel. After finishing up with the parallel job, we use the traditional Python list iteration to pass through our list of estimators that cannot be processed in parallel. At the end of this step, we should have scores for all 16 models.

Our score processing is done but we need to compare the performance. The scores are stored in a structured numpy array. The tabular view of a pandas dataframe is an excellent tool for comparison and we can also use function mapping capabilities of pandas to create a custom weighting system. The weighting will reward a model for having a high f1 score and low processing time. At the same time, it will penalize models for having a larger standard deviation. Remember, our ideal model would process the training data quickly (low processing time), perform consistently  (low standard deviation), and have a high f1 score. Here's our code: 

Here is our table of scores: 

The pandas dataframe aligns our scores neatly, enabling a quick comparison. I must note that all scores are based on estimators using default settings. Hyperparameter tuning sometimes leads to substantial performance gains but we use the default configuration to triage and decide which model to spend time tuning. Looking at our table above, how did the models perform?

The BaggingClassifier estimator had a high f1 score but was ranked lower because of the longer compute time (~12 seconds to train the model). The model with the fastest training time, the Multinomial Naive Bayes estimator (and other Bayesian estimators), was the top of the pack despite its lower score and wider standard deviation (our ranking wasn't perfect, but it's good enough). Looking over the results, it appears the Linear Support Vector Classification (Linear SVC) estimator provides the best balance of speed, performance, and consistency. This is the model we will explore going forward.

What does the Linear SVC do?

Since the Linear SVC model was our best performer, let's take some time to understand how it works. Linear SVC is a supervised machine learning method that calculates the optimal separation distance between target classes and a hyperplane; this separation is called the margin. SVC then uses a quadratic function to identify support vectors and iterates over the data set to draw a hyperplane that provides the most distance between the two classes. Support vectors are observations in one class that are closest to an observation in another class.  In short, this model will draw the best line between the classes to separate them. This graphic shows the major components of the Linear SVC calculation:

Read the following for more details on Support Vector Machines:

1. National Instruments Discussion of Support Vector Machines

2. Linear and nonlinear support vector classification theory sections in this paper

3. Brief scikit-learn discussion of Support Vector Machines

Tuning to Improve Model Performance

From our scoring tests above, we see that the LinearSVC model performs well with a ~91% f1 score, but we can likely improve our model's performance by tuning hyperparameters. Hyperparameters are parameters that are not directly learned by our model and tweaking a small subset of these parameters can substantially impact a model's performance. It's best to read the docstring for the scikit-learn estimator you are using to see how different values impact your model's performance. Once you find the models you want to tune, we need to test different values to see how performance changes. Scikit-learn can perform a randomized search of parameters using RandomizedSearchCV to find the perfect combination of values that achieve the highest score. 

Note: I am using the scipy.stats function to specify a distribution over possible values. Any function can be passed that provides a RVS (random variate sample) method to sample a value. See scipy.stats documentation for details.

After looking at the possible LinearSVC parameters, our code to tune is as follows:

Let's execute our randomized search for the best parameters:

Our best parameters and scores were: 

We were able to improve model performance slightly, achieving a ~94% f1 score. This is a good score, but let's expand how we review the performance of our model.

Evaluating Model Performance Visually using yellowbrick

Up until now, we printed the numeric scores to evaluate the performance of our model. But, we can also compare performance visually. The Yellowbrick library provides a suite of visual diagnostic tools that help us pick the best model based on performance in several key metric areas.

Let's review model performance using the classification report visualizer, which is an intuitive display of your model's scores in precision, recall, and the f1-measure. Evaluation is simple; you want a matrix that has all dark colors and high numbers closer to 1.0.  Let's build the function to visualize model performance:

Now we visualize performance for the LinearSVC model we tuned. Earlier, our f1 score reached 94%, but now we can use this tool to understand strengths and weaknesses of the model. Remember, we want to see all dark red blocks and numbers closer to 1.0 in our heat map.  

This visual gives us a better understanding of our model. The good news is that our heatmap has a high concentration of dark colors, which means it is high performing. But, a look at the numbers provides more detail. The model has better precision at finding articles that discuss quarterback. Therefore, you can have more confidence that when the model returns "about a quarterback," the article is indeed discussing a quarterback. But, we see the opposite relationship in the recall metric; the model can correctly classify more articles that are not discussing quarterbacks than models that do. That means we're going to be missing some "about a quarterback" articles. In other words, you are seeing a real life example of the trade-off between precision and recall. Again, this performance is adequate, but we'll go one step farther to see how we can improve our scores.

Ensembling/Stacking to get a "Boost"

RandomizedSearchCV gave us a small uptick in model performance but we will close out our model tuning section by introducing a powerful machine learning approach. Ensembling is the process of combining the outputs of various algorithms on the same data to create a stronger learner. When solving supervised learning problems, the model can adjust the various models with a special emphasis on improving the performance on misclassified outputs. The approach will often outperform individual models and result in a higher performing model. What does this mean for our tutorial? After we have found a high performing model, we can then use this estimator to boost performance. For more reading on boosting:

• Adaboost discussion

• Boosting Algorithms

• Improving Machine Learning with Ensemble methods

Scikit-learn has a method for the popular boosting algorithm, AdaBoost. Let's try this estimator and get the visual inspection as well: 

Wow! Note the change in numbers. Averaging scores across rows or columns, the score is consistently ~96%. Our tinkering has been successful because we boosted our model's performance by 5%! Now, we're ready to save the boosted model for later use.

Storing the Model: Persisting for later use 

To make the model available for use later, we will pickle it to disk. You may hear others in industry refer to this as "persisting" a model. In this case, our model includes the full pipeline of reading the data in, converting strings to vectors, passing those vectors to the model, and predicting the class of the observation. Scikit-learn has a method that uses an external tool to persist our model. Here is the code:

Our model is stored and ready for reuse. Now it's time for a real test!

Testing the model: Does it work on new, unseen data?

While improving the performance of your model against known data is positive, nothing proves the viability of your model like a real world test. I will test our model on four articles with the following expected outcomes:

1. Article about an NFL quarterback; expected outcome → class 1

2. Article about a college quarterback; expected outcome → class 1

3. Article about football but not discussing a quarterback; expected outcome → class

4. Article about current event; expected outcome → class 0

Pay extra special attention to tests 2 and 3; if we predict the correct class for these articles, we can can rejoice because the model is generalizable enough to make relevant predictions on news articles it has not seen. We will discuss the relevance below.  

After a random search, I found several non-ESPN.com articles that fit our designed test. Again, I'll use the custom script I made to strip data from websites and return it as a structured record. Here is the code: 

We got every prediction right! Take special note of the significance of predictions 2 and 3.  

We trained our model using labels based on the names of NFL QBs only. Prediction 2 is significant because the model was able to generalize and recognize articles that discussed the "quarterback" position in general. Without human intervention, we successfully created a link between the word "quarterback" and our topic of interest (remember, we only labeled articles based on the presence of an active NFL QB's name).  Because of this association, the model is able to accurately recognize articles about college QBs as well. But our notable takeaways don't stop with the positive class but our negative class shows some pretty powerful intuition. 

Prediction 3 is significant because it proves that our model is specific enough to distinguish between articles that discuss "football" and articles that discuss "football and quarterbacks." This article is just about football, and does not mention the word "quarterback."

Based on these tests, we know that our model is sufficient for the task of identifying articles on American football quarterbacks. We're done tuning and testing! 

Making a Machine Learning Service to Deliver a Data Product

We are confident that our model performs well, but we need to deliver this product to our technical and non-technical customers. Most tutorials would end after the paragraph above and this is a major reason why data scientists struggle to find purpose when they start a new job. We all make beautiful and powerful models, but how do you get the model to the customer? The final section of this tutorial will provide an example of delivering your model.

In our scenario, we had to demonstrate the model to our non-technical customer (sports agency) but also provide a machine-readable path for our developers to access the model. Because of this dual-natured requirement, our product should be user-facing and simple enough for a non-technical audience. But, it should also have a query-able back end for our technically-inclined customers. A web application seems like a good solution. 

Here are the functional requirements:

1. Allow a user to visit a web site.

2. Allow user to paste a url of article.

3. Pass data to our model and get a prediction.

4. Show that prediction to user.

5. Allow repeats.

6. Provide same functionality for technical consumers (developers or other data scientists) but in machine readable endpoint.

Luckily, it doesn't take much to build a web application with these features. In fact, we can pretty much copy the web application example from the cherryPy documentation. It's important to note that we are not creating a robust web application (that's the developer's job). We just need a prototype to deliver the model to our customers. 

Building the Web Application

To get started, we use the CherryPy framework to make an application. One example in the tutorials section provides a good baseline of code for us to copy. The project folder for this tutorial has a folder named qb_intspector that has all the files we need to run the application; you will see explanations of each file below.  First, let's see the file structure of the application:

qb_intspector/ │ index.html │ my.db │ QBapp.py │ utilities.py │    └─── models/  │          │  │           | qbmodel_final.pkl │ └─── public/  │         │    │         └─── css/  │                      │ │                      │ style.css  │         │        │         └─── images/ │                     │ │                     │pic1.jpg │                     │pic2.jpg │                     │pic3.jpg 

Here is an explanation for each file:

• index.html - this is the code that defines how our website looks. It also contains some Javascript and jQuery language to dynamically load content based on user actions. Instructions on the Javascript and jQuery code is outside the scope of this tutorial.

• my.db - this is our SQLite database. Each time a user requests a prediction, that data is stored in the database for later use. This database can be used by developers to query predictions that were done in the past. The database is created and destroyed when the application starts and stops.

• QBapp.py - this is the workhorse code of our application. It defines all the functionality we need to run the application and pass data to our machine learning model to get the predictions.

• utilities.py - contains utility functions used by our web application. Specifically, this file contains the code to extract structured data from a web-based new article.

• models - this folder contains the scikit-learn model we built earlier.

• public - this folder contains the images and other formatting files for our HTML code.

To start the application:

1. Open a terminal or command prompt.

2. Navigate to the folder where the QBapp.py file is located.

3. Execute this command python QBapp.py.

The web application should have started. Visit http://127.0.0.1:8080/ in a browser to see our web application in action. 

Using the QB-INTSpector Web Application

Non-Technical Use: Getting Predictions Using the Web Application

Once the application is running, getting a prediction is easy.  Copy a web url and paste it into the input box and click Get Prediction. After pressing this button, the web application will reach out to the website, extract the news article data, convert the text data into an array of vectors, pass the vectors to our classifier model, get a prediction, store the data in a database, and return a message to the web application on the category of the article. Here's what happens when the article is discussing a quarterback.

All of that happens with one button click. This is far simpler than requiring a person to install Python, load libraries, and execute some code. 

Technical Use: Using the Prediction Services and Data

The last piece is giving our developers access to the model. Our web application has a backend API service that can retrieve predictions. This endpoint will return a JSON string with all the extracted data for the news article with added key value pairs for the actual prediction. Here is the code a developer would use to access your prediction service: 

Developrs can also access cached results using this code: 

Remember, this database is completely deleted when the web application is shut down. You can modify the code to load and delete the database if you want to persist this data.

Using a combination of the services below, our developers could pass in a daily stream of urls, get predictions, and load up a custom web page or production web application that satisfies our sports agent customer's needs.

Conclusion

The journey was long, but we successfully completed one full cycle through the data science pipeline. Moreover, we used a real world example that included tips on how to pick and tune the best model to achieve optimum performance. And then finally, we created a web application that can deliver the output of our model to technical and non-technical audiences. The tools and tricks you learned in this tutorial can be applied to many machine learning projects.

As a side note, the Javascript, JQuery, and HTML I used above is beyond the scope of this discussion. I wanted to focus on the model and give a quick example of a delivery mechanism. If you'd like to suggest any improvements to this process or let me know about any errors you see, feel free to reach out on Twitter! 

District Data Labs provides data science consulting and corporate training services. We work with companies and teams of all sizes, helping them make their operations more data-driven and enhancing the analytical abilities of their employees. Interested in working with us? Let us know!

Online Data Science Corporate Training

Live, online, instructor-led courses on the latest data science, analytics, and machine learning methods and tools.

Need help with machine learning?

Our consulting services help you use data to make smarter decisions, grow your business, and accomplish more with the resources you have.

Related Posts:

• How to Start Your first Data Science Project

• Getting Started in Open Source

• Data Exploration With Python, Part 1

Our Books: