Know Your Missing Data:

Sometimes datasets contain missing data. Imagine a dataset with a column "age", where a lot of the ages are missing. This could pose a problem because it is impossible to perform statistical analysis when data is missing. Missing data can appear in any form including blanks and NaN. It is important to correctly identify the nature of missing data in your datasets. Values such as zeroes and unknown could easily be confused for missing data when for the given use case or problem statement they are not. Always  know your missing data.  

In the field of Natural Language Processing (NLP), preprocessing of text needs to occur before running any Machine Learning algorithms. This blog will provide some common examples of the types of preprocessing that need to be applied to text before any analysis can take place.

What is Text Preprocessing?

Text preprocessing involves the cleaning, normalization and standardization of text before the application of NLP techniques. Keep in mind that although there are several types of text preprocessing techniques, one must use the ones suitable for their use case in an order that makes sense. 

Given a particular text or string, it is possible to create an image containing the words of that text, where the size of each word is dependent on its frequency within the text. Such an image is referred to as a Word Cloud. Using Python's wordcloud and matplotlib libraries, a Data Scientist can easily generate a word cloud given an input string or text.

The following is a Word Cloud generated for The Mother Goose poem called "Hey, diddle, diddle".