Book Image

Java Data Science Cookbook

By : Rushdi Shams
Book Image

Java Data Science Cookbook

By: Rushdi Shams

Overview of this book

If you are looking to build data science models that are good for production, Java has come to the rescue. With the aid of strong libraries such as MLlib, Weka, DL4j, and more, you can efficiently perform all the data science tasks you need to. This unique book provides modern recipes to solve your common and not-so-common data science-related problems. We start with recipes to help you obtain, clean, index, and search data. Then you will learn a variety of techniques to analyze, learn from, and retrieve information from data. You will also understand how to handle big data, learn deeply from data, and visualize data. Finally, you will work through unique recipes that solve your problems while taking data science to production, writing distributed data science applications, and much more - things that will come in handy at work.
Table of Contents (16 chapters)
Java Data Science Cookbook
Credits
About the Author
About the Reviewer
www.PacktPub.com
Customer Feedback
Preface

Cleaning ASCII text files using Regular Expressions


ASCII text files can contain unnecessary units of characters that eventually are introduced during a conversion process, such as PDF-to-text conversion or HTML-to-text conversion. These characters are often seen as noise because they are one of the major roadblocks for data processing. This recipe cleans several noises from ASCII text data using Regular Expressions.

How to do it...

  1. Create a method named cleanText(String) that takes the text to be cleaned in the String format:

            public String cleanText(String text){ 
    
  2. Add the following lines in your method, return the cleaned text, and close the method. The first line strips off non-ASCII characters. The line next to it replaces continuous white spaces with a single white space. The third line erases all the ASCII control characters. The fourth line strips off the ASCII non-printable characters. The last line removes non-printable characters from Unicode:

            text = text.replaceAll("[^p{ASCII}]",""); 
            text = text.replaceAll("s+", " "); 
            text = text.replaceAll("p{Cntrl}", ""); 
            text = text.replaceAll("[^p{Print}]", ""); 
            text = text.replaceAll("p{C}", ""); 
     
            return text; 
            } 
    

    The full method with the driver method in a class will look as follows:

    public class CleaningData { 
       public static void main(String[] args) throws Exception { 
          CleaningData clean = new CleaningData(); 
          String text = "Your text here you have got from some file"; 
          String cleanedText = clean.cleanText(text); 
          //Process cleanedText 
       } 
        
       public String cleanText(String text){ 
          text = text.replaceAll("[^p{ASCII}]",""); 
            text = text.replaceAll("s+", " "); 
            text = text.replaceAll("p{Cntrl}", ""); 
            text = text.replaceAll("[^p{Print}]", ""); 
            text = text.replaceAll("p{C}", ""); 
            return text; 
       } 
    }