Text Feature Extraction is the process of transforming text into numerical feature vectors to be used by Machine Learning Algorithms. This blog will focus on two different ways in which text can be transformed into numerical values by using Python's Scikit-Learn library. Text documents or corpus cannot be directly fed into Machine Learning Algorithms because these algorithms expect data input that are numerical feature vectors or matrices. There are many ways of converting a corpus to a feature vector such as Bag of Words and TF-IDF Weighting. Before digging deeper let's first define some important terminology in the area of Text Feature Extraction.

 
IMPORTANT TERMINOLOGY:

Corpus:- This is a collection of all documents or text, usually stored as a comma separated list of strings.

Document:- This represents each element in the corpus and it is a piece of text of any length.

Token:- A document or text usually first needs to be broken down into small chunks referred to as tokens or words. This process is called tokenization and on its own can be a rather comprehensive topic as there are multiple strategies for tokenization that takes into account delimeters, n-grams and regular expressions.

Count/Frequency:- The number of times a token occurs in a text document represents that token's frequency within the document. 

Feature:- Each individual token identified across the entire corpus is treated as a feature with a unique ID in the final feature vector. Keep in mind  that there is no particular ordering to the features in a vector. 

Feature Vector:- An vector representation of a corpus where each row represents a document and the columns represent the tokens found across the entire corpus. 

Vectorization:- This is the process of converting a corpus or collection of text documents into a numerical feature vector where the columns represent the tokens found in the entire corpus and each row represents a document.

Know Your Missing Data:

Sometimes datasets contain missing data. Imagine a dataset with a column "age", where a lot of the ages are missing. This could pose a problem because it is impossible to perform statistical analysis when data is missing. Missing data can appear in any form including blanks and NaN. It is important to correctly identify the nature of missing data in your datasets. Values such as zeroes and unknown could easily be confused for missing data when for the given use case or problem statement they are not. Always  know your missing data.  

In the field of Natural Language Processing (NLP), preprocessing of text needs to occur before running any Machine Learning algorithms. This blog will provide some common examples of the types of preprocessing that need to be applied to text before any analysis can take place.

What is Text Preprocessing?

Text preprocessing involves the cleaning, normalization and standardization of text before the application of NLP techniques. Keep in mind that although there are several types of text preprocessing techniques, one must use the ones suitable for their use case in an order that makes sense. 

Given a particular text or string, it is possible to create an image containing the words of that text, where the size of each word is dependent on its frequency within the text. Such an image is referred to as a Word Cloud. Using Python's wordcloud and matplotlib libraries, a Data Scientist can easily generate a word cloud given an input string or text.

The following is a Word Cloud generated for The Mother Goose poem called "Hey, diddle, diddle".