{"id":233,"date":"2018-09-21T17:01:54","date_gmt":"2018-09-21T17:01:54","guid":{"rendered":"https:\/\/justinmatters.co.uk\/wp\/?p=233"},"modified":"2018-10-11T12:28:33","modified_gmt":"2018-10-11T12:28:33","slug":"the-danger-of-missing-values-in-data-sets","status":"publish","type":"post","link":"https:\/\/justinmatters.co.uk\/wp\/the-danger-of-missing-values-in-data-sets\/","title":{"rendered":"The Danger of Missing Values in Data Sets"},"content":{"rendered":"

Missing values in data sets may not seem like too much of a problem at first glance. The rows can be ignored, average values can be input or the data can be marked as missing. However applying the wrong strategy can lead to serious errors of interpretation. Lets take a look at a concrete example in that old favourite of a data set, the Titanic survival training data set<\/a>. Note that for simplicity only the training data from the Kaggle challenge is being examined here.<\/p>\n

First lets import our data obtained from the link<\/a> above and\u00a0 take a quick look at it:<\/p>\n

\r\n# Import libraries\r\nimport matplotlib\r\nimport pandas as pd\r\nimport numpy as np\r\nimport seaborn as sns\r\n\r\n# Enable inline printing\r\n%matplotlib inline\r\n\r\n# Import our data\r\ntitanic_train = pd.read_csv("train.csv")\r\n# Make the Survived column more human readable\r\ntitanic_train["Survived"] = titanic_train["Survived"]\r\n    .map({1: "Yes", 0:"No"})\r\n# Then display\r\ntitanic_train.head()\r\n\r\n<\/pre>\n
We can see that some passengers have a cabin assigned to them, looking through the data we realise that the first letter of the cabin is the deck assignment. A little historical research indicates that deck may be correlated with passenger class since first class had the top decks (A-E), second class (D-F), and third class (E-G). Which deck people were on seems likely to affect survival, after all the lifeboats are on the upper decks so we decide to investigate deck assignments more closely.<\/p>\n
First we need to extract the deck information, then we can visualise the relationship between deck and survival.<\/p>\n
\r\n# create a new column using the first letter of the cabin column\r\ntitanic_train["Deck"] = titanic_train["Cabin"].dropna().\r\n    astype(str).str[0].str.upper()<\/pre>\r\n# remove anomalous deck assignment "T"\r\ntitanic_train.loc[titanic_train["Deck"] == "T"] = np.nan\r\n\r\n# Plot survival versus deck\r\nsns.countplot("Survived", data = titanic_train, hue="Deck",\r\nhue_order = ["A", "B", "C", "D", "E", "F", "G"], \r\n    order=["Yes", "No",]);\r\n<\/pre>\n
\"According
Plot of survival by deck for passengers with a known deck<\/figcaption><\/figure>\n
Looking at this naively it would appear that the survival rates on deck G is as good as the survival rate on deck A,\u00a0 the survival rates on decks D and E are better than that on deck B and the survival rate on deck F is better than that of deck C. Given that second and third class passengers occupied decks D-G it might appear that they actually fared reasonably well during the sinking. However the graph above is actually highly misleading.<\/p>\n
The first indication that all is not as it might seem is that according to the plot by deck, more than half the passengers survived. However a knowledge of history<\/a> (or plotting the whole training data set) reveals the situation is actually very different. In fact far less than half the passengers in the data set survive.<\/p>\n
\r\n# plot the whole dataset\r\nsns.countplot("Survived", data = titanic_train, \r\n    order=["Yes", "No"]);\r\n<\/pre>\n
\"A
Plot of survivors of the Titanic from the whole training set<\/figcaption><\/figure>\n
So what is the cause of the discrepancy? Well lets add a bar the passengers for whom we have no cabin data to our survival by deck plot.<\/p>\n
\r\n# Run fillna on the NaNs and plot again\r\ntitanic_train["Deck"] = titanic_train["Deck"].fillna("N")\r\n\r\n# Visualise what happened to those for whom we have \r\n# no deck indication alongside those we do\r\nsns.countplot("Survived", data=titanic_train, hue="Deck",\r\nhue_order = ["A", "B", "C", "D", "E", "F", "G", "N"], \r\n    order = ["Yes", "No"]);\r\n<\/pre>\n
\"Most
A survival by deck plot with those with no known deck added<\/figcaption><\/figure>\n
Now we can clearly see our issue, the rows with deck allocations and those without are not drawn from the same distributions. This means that inferences we draw about one population have no reason to transfer to the other population.<\/p>\n
Lets investigate a little further by adding a column noting whether or not we have a deck allocation for a given passenger so that we can plot this information directly<\/p>\n
\r\n# lets add a column indicating whether the cabin is known\r\ntitanic_train["KnownDeck"] = titanic_train["Deck"]\r\n    .apply(lambda x: "No" if x == "N" else "Yes")\r\n\r\n# Now display the distribution of survivors\r\nsns.countplot("Survived", data=titanic_train, \r\n    hue="KnownDeck");\r\n<\/pre>\n
\"Being
A plot demonstrating the correlation between known deck allocation and survival<\/figcaption><\/figure>\n
Here the difference can be seen very starkly, having a known deck is highly correlated with passenger survival.<\/p>\n
What if we break down the survivors by class, whether they survived and whether they had a known deck allocation?<\/p>\n
\r\n# Now lets plot to see if the cabins we have are a \r\n# representative sample across passenger classes\r\ngrid = sns.FacetGrid(titanic_train, row ="KnownDeck", \r\n    row_order = ["Yes", "No"], col= "Pclass", height=4, aspect=1)\r\ngrid.map(sns.countplot, "Survived", order = ["Yes", "No"]);\r\n<\/pre>\n
\"Being
A plot demonstrating the correlation between known deck allocation and survival broken down by passenger class<\/figcaption><\/figure>\n
Here we can see that regardless of class, if you survived we are more likely to have a deck allocation. Also we can see that first class passengers were most likely to survive and third class least likely to survive which is more in line with the known facts of the disaster.<\/p>\n
But why are having a known deck and survival highly correlated? Well it seems unlikely that having a known deck caused passengers to survive. More likely causation works the other way round in this case. The data set records cabin numbers where they are known and this is more likely if the passenger survived.\u00a0 The data set on survivors will have been compiled after the sinking and so the survival bias on deck numbers suggests that cabin allocations were mostly recorded for survivors. A higher proportion of first class victims have known cabin allocations than second and third class, so perhaps first class allocations were recorded by the White Star Line and the few other cabin allocations that are known were discovered by questioning survivors.<\/p>\n
In this case the effect of the missing values is fairly obvious, however imagine a less famous case and imagine that instead of retaining the rows with missing data we had initially “cleaned” our data\u00a0 using<\/p>\n
\r\ntitanic_train.dropna()\r\n<\/pre>\n
In such a case we would not necessarily get any intuition that the conclusions we are drawing are on shaky ground. After all if we already knew the answers, we would not need to look at the data. This could have led to serious errors in our conclusions. Missing values need to be treated with care, think and check before you drop them or fill them.<\/p>\n","protected":false},"excerpt":{"rendered":"
Missing values in data sets may not seem like too much of a problem at first glance. The rows can be ignored, average values can…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[11],"tags":[33,6,14,34],"class_list":["post-233","post","type-post","status-publish","format-standard","hentry","category-data-science","tag-data-science","tag-problem-solving","tag-python","tag-seaborn"],"yoast_head":"\nThe Danger of Missing Values in Data Sets - Justin's Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/justinmatters.co.uk\/wp\/the-danger-of-missing-values-in-data-sets\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"The Danger of Missing Values in Data Sets - Justin's Blog\" \/>\n<meta property=\"og:description\" content=\"Missing values in data sets may not seem like too much of a problem at first glance. The rows can be ignored, average values can…\" \/>\n<meta property=\"og:url\" content=\"https:\/\/justinmatters.co.uk\/wp\/the-danger-of-missing-values-in-data-sets\/\" \/>\n<meta property=\"og:site_name\" content=\"Justin's Blog\" \/>\n<meta property=\"article:published_time\" content=\"2018-09-21T17:01:54+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2018-10-11T12:28:33+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/justinmatters.co.uk\/wp\/wp-content\/uploads\/2018\/09\/Figure1.png\" \/>\n<meta name=\"author\" content=\"justinmatters\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"justinmatters\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/justinmatters.co.uk\\\/wp\\\/the-danger-of-missing-values-in-data-sets\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/justinmatters.co.uk\\\/wp\\\/the-danger-of-missing-values-in-data-sets\\\/\"},\"author\":{\"name\":\"justinmatters\",\"@id\":\"https:\\\/\\\/justinmatters.co.uk\\\/wp\\\/#\\\/schema\\\/person\\\/7c3e0740e1fef74f705c19f175f6f321\"},\"headline\":\"The Danger of Missing Values in Data Sets\",\"datePublished\":\"2018-09-21T17:01:54+00:00\",\"dateModified\":\"2018-10-11T12:28:33+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/justinmatters.co.uk\\\/wp\\\/the-danger-of-missing-values-in-data-sets\\\/\"},\"wordCount\":1263,\"commentCount\":0,\"image\":{\"@id\":\"https:\\\/\\\/justinmatters.co.uk\\\/wp\\\/the-danger-of-missing-values-in-data-sets\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/justinmatters.co.uk\\\/wp\\\/wp-content\\\/uploads\\\/2018\\\/09\\\/Figure1.png\",\"keywords\":[\"Data Science\",\"Problem Solving\",\"Python\",\"Seaborn\"],\"articleSection\":[\"Data Science\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/justinmatters.co.uk\\\/wp\\\/the-danger-of-missing-values-in-data-sets\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/justinmatters.co.uk\\\/wp\\\/the-danger-of-missing-values-in-data-sets\\\/\",\"url\":\"https:\\\/\\\/justinmatters.co.uk\\\/wp\\\/the-danger-of-missing-values-in-data-sets\\\/\",\"name\":\"The Danger of Missing Values in Data Sets - Justin's Blog\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/justinmatters.co.uk\\\/wp\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/justinmatters.co.uk\\\/wp\\\/the-danger-of-missing-values-in-data-sets\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/justinmatters.co.uk\\\/wp\\\/the-danger-of-missing-values-in-data-sets\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/justinmatters.co.uk\\\/wp\\\/wp-content\\\/uploads\\\/2018\\\/09\\\/Figure1.png\",\"datePublished\":\"2018-09-21T17:01:54+00:00\",\"dateModified\":\"2018-10-11T12:28:33+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/justinmatters.co.uk\\\/wp\\\/#\\\/schema\\\/person\\\/7c3e0740e1fef74f705c19f175f6f321\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/justinmatters.co.uk\\\/wp\\\/the-danger-of-missing-values-in-data-sets\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/justinmatters.co.uk\\\/wp\\\/the-danger-of-missing-values-in-data-sets\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/justinmatters.co.uk\\\/wp\\\/the-danger-of-missing-values-in-data-sets\\\/#primaryimage\",\"url\":\"https:\\\/\\\/justinmatters.co.uk\\\/wp\\\/wp-content\\\/uploads\\\/2018\\\/09\\\/Figure1.png\",\"contentUrl\":\"https:\\\/\\\/justinmatters.co.uk\\\/wp\\\/wp-content\\\/uploads\\\/2018\\\/09\\\/Figure1.png\",\"width\":386,\"height\":266},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/justinmatters.co.uk\\\/wp\\\/the-danger-of-missing-values-in-data-sets\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/justinmatters.co.uk\\\/wp\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"The Danger of Missing Values in Data Sets\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/justinmatters.co.uk\\\/wp\\\/#website\",\"url\":\"https:\\\/\\\/justinmatters.co.uk\\\/wp\\\/\",\"name\":\"Justin's Blog\",\"description\":\"Justin's Coding and Geek Blog\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/justinmatters.co.uk\\\/wp\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/justinmatters.co.uk\\\/wp\\\/#\\\/schema\\\/person\\\/7c3e0740e1fef74f705c19f175f6f321\",\"name\":\"justinmatters\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/27cf337940887c098b79716aa7025ce782bd51de3f6b07a9dcad710bbf576c59?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/27cf337940887c098b79716aa7025ce782bd51de3f6b07a9dcad710bbf576c59?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/27cf337940887c098b79716aa7025ce782bd51de3f6b07a9dcad710bbf576c59?s=96&d=mm&r=g\",\"caption\":\"justinmatters\"},\"description\":\"Data Scientist specialising in Python, PySpark, SQL and Machine Learning\",\"sameAs\":[\"https:\\\/\\\/justinmatters.co.uk\\\/wp\\\/\",\"https:\\\/\\\/uk.linkedin.com\\\/in\\\/justin-matters-edinburgh\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"The Danger of Missing Values in Data Sets - Justin's Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/justinmatters.co.uk\/wp\/the-danger-of-missing-values-in-data-sets\/","og_locale":"en_US","og_type":"article","og_title":"The Danger of Missing Values in Data Sets - Justin's Blog","og_description":"Missing values in data sets may not seem like too much of a problem at first glance. The rows can be ignored, average values can…","og_url":"https:\/\/justinmatters.co.uk\/wp\/the-danger-of-missing-values-in-data-sets\/","og_site_name":"Justin's Blog","article_published_time":"2018-09-21T17:01:54+00:00","article_modified_time":"2018-10-11T12:28:33+00:00","og_image":[{"url":"https:\/\/justinmatters.co.uk\/wp\/wp-content\/uploads\/2018\/09\/Figure1.png","type":"","width":"","height":""}],"author":"justinmatters","twitter_card":"summary_large_image","twitter_misc":{"Written by":"justinmatters","Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/justinmatters.co.uk\/wp\/the-danger-of-missing-values-in-data-sets\/#article","isPartOf":{"@id":"https:\/\/justinmatters.co.uk\/wp\/the-danger-of-missing-values-in-data-sets\/"},"author":{"name":"justinmatters","@id":"https:\/\/justinmatters.co.uk\/wp\/#\/schema\/person\/7c3e0740e1fef74f705c19f175f6f321"},"headline":"The Danger of Missing Values in Data Sets","datePublished":"2018-09-21T17:01:54+00:00","dateModified":"2018-10-11T12:28:33+00:00","mainEntityOfPage":{"@id":"https:\/\/justinmatters.co.uk\/wp\/the-danger-of-missing-values-in-data-sets\/"},"wordCount":1263,"commentCount":0,"image":{"@id":"https:\/\/justinmatters.co.uk\/wp\/the-danger-of-missing-values-in-data-sets\/#primaryimage"},"thumbnailUrl":"https:\/\/justinmatters.co.uk\/wp\/wp-content\/uploads\/2018\/09\/Figure1.png","keywords":["Data Science","Problem Solving","Python","Seaborn"],"articleSection":["Data Science"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/justinmatters.co.uk\/wp\/the-danger-of-missing-values-in-data-sets\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/justinmatters.co.uk\/wp\/the-danger-of-missing-values-in-data-sets\/","url":"https:\/\/justinmatters.co.uk\/wp\/the-danger-of-missing-values-in-data-sets\/","name":"The Danger of Missing Values in Data Sets - Justin's Blog","isPartOf":{"@id":"https:\/\/justinmatters.co.uk\/wp\/#website"},"primaryImageOfPage":{"@id":"https:\/\/justinmatters.co.uk\/wp\/the-danger-of-missing-values-in-data-sets\/#primaryimage"},"image":{"@id":"https:\/\/justinmatters.co.uk\/wp\/the-danger-of-missing-values-in-data-sets\/#primaryimage"},"thumbnailUrl":"https:\/\/justinmatters.co.uk\/wp\/wp-content\/uploads\/2018\/09\/Figure1.png","datePublished":"2018-09-21T17:01:54+00:00","dateModified":"2018-10-11T12:28:33+00:00","author":{"@id":"https:\/\/justinmatters.co.uk\/wp\/#\/schema\/person\/7c3e0740e1fef74f705c19f175f6f321"},"breadcrumb":{"@id":"https:\/\/justinmatters.co.uk\/wp\/the-danger-of-missing-values-in-data-sets\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/justinmatters.co.uk\/wp\/the-danger-of-missing-values-in-data-sets\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/justinmatters.co.uk\/wp\/the-danger-of-missing-values-in-data-sets\/#primaryimage","url":"https:\/\/justinmatters.co.uk\/wp\/wp-content\/uploads\/2018\/09\/Figure1.png","contentUrl":"https:\/\/justinmatters.co.uk\/wp\/wp-content\/uploads\/2018\/09\/Figure1.png","width":386,"height":266},{"@type":"BreadcrumbList","@id":"https:\/\/justinmatters.co.uk\/wp\/the-danger-of-missing-values-in-data-sets\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/justinmatters.co.uk\/wp\/"},{"@type":"ListItem","position":2,"name":"The Danger of Missing Values in Data Sets"}]},{"@type":"WebSite","@id":"https:\/\/justinmatters.co.uk\/wp\/#website","url":"https:\/\/justinmatters.co.uk\/wp\/","name":"Justin's Blog","description":"Justin's Coding and Geek Blog","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/justinmatters.co.uk\/wp\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/justinmatters.co.uk\/wp\/#\/schema\/person\/7c3e0740e1fef74f705c19f175f6f321","name":"justinmatters","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/27cf337940887c098b79716aa7025ce782bd51de3f6b07a9dcad710bbf576c59?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/27cf337940887c098b79716aa7025ce782bd51de3f6b07a9dcad710bbf576c59?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/27cf337940887c098b79716aa7025ce782bd51de3f6b07a9dcad710bbf576c59?s=96&d=mm&r=g","caption":"justinmatters"},"description":"Data Scientist specialising in Python, PySpark, SQL and Machine Learning","sameAs":["https:\/\/justinmatters.co.uk\/wp\/","https:\/\/uk.linkedin.com\/in\/justin-matters-edinburgh"]}]}},"_links":{"self":[{"href":"https:\/\/justinmatters.co.uk\/wp\/wp-json\/wp\/v2\/posts\/233","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/justinmatters.co.uk\/wp\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/justinmatters.co.uk\/wp\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/justinmatters.co.uk\/wp\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/justinmatters.co.uk\/wp\/wp-json\/wp\/v2\/comments?post=233"}],"version-history":[{"count":11,"href":"https:\/\/justinmatters.co.uk\/wp\/wp-json\/wp\/v2\/posts\/233\/revisions"}],"predecessor-version":[{"id":253,"href":"https:\/\/justinmatters.co.uk\/wp\/wp-json\/wp\/v2\/posts\/233\/revisions\/253"}],"wp:attachment":[{"href":"https:\/\/justinmatters.co.uk\/wp\/wp-json\/wp\/v2\/media?parent=233"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/justinmatters.co.uk\/wp\/wp-json\/wp\/v2\/categories?post=233"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/justinmatters.co.uk\/wp\/wp-json\/wp\/v2\/tags?post=233"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}