Chapter 7 Missing Values
Missing values play an important role in statistics and data analysis. Often, missing values must not be ignored, but rather they should be carefully studied to see if there’s an underlying pattern or cause for their missingness.
In R, NA is used to represent any value that is ‘not available’ or ‘missing’ (in the statistical sense). In this lesson, we’ll explore missing values further.
Any operation involving NA generally yields NA as the result. To illustrate, let’s create a vector c(44, NA, 5, NA) and assign it to a variable x.
Now, let’s multiply x by 3.
Notice that the elements of the resulting vector that correspond with the NA values in x are also NA.
To make things a little more interesting, lets create a vector containing 1000 draws from a standard normal distribution with y <- rnorm(1000).
Next, let’s create a vector containing 1000 NAs with z <- rep(NA, 1000).
Finally, let’s select 100 elements at random from these 2000 values (combining y and z) such that we don’t know how many NAs we’ll wind up with or what positions they’ll occupy in our final vector – my_data <- sample(c(y, z), 100).
Let’s first ask the question of where our NAs are located in our data. The is.na() function tells us whether each element of a vector is NA. Call is.na() on my_data and assign the result to my_na.
Now, print my_na to see what you came up with.
my_na
## [1] FALSE TRUE FALSE TRUE TRUE TRUE FALSE TRUE FALSE TRUE TRUE FALSE TRUE FALSE TRUE TRUE TRUE
## [18] FALSE FALSE FALSE TRUE TRUE TRUE TRUE FALSE TRUE FALSE TRUE TRUE TRUE FALSE TRUE TRUE FALSE
## [35] TRUE TRUE FALSE TRUE TRUE FALSE TRUE FALSE FALSE TRUE TRUE TRUE TRUE TRUE TRUE FALSE FALSE
## [52] FALSE TRUE FALSE TRUE FALSE FALSE TRUE FALSE FALSE TRUE FALSE FALSE TRUE FALSE TRUE FALSE FALSE
## [69] FALSE FALSE FALSE TRUE TRUE TRUE FALSE TRUE FALSE FALSE FALSE TRUE FALSE FALSE FALSE TRUE TRUE
## [86] FALSE FALSE TRUE TRUE FALSE FALSE FALSE TRUE FALSE TRUE FALSE TRUE TRUE TRUE FALSEEverywhere you see a TRUE, you know the corresponding element of my_data is NA. Likewise, everywhere you see a FALSE, you know the corresponding element of my_data is one of our random draws from the standard normal distribution.
In our previous discussion of logical operators, we introduced the == operator as a method of testing for equality between two objects. So, you might think the expression my_data == NA yields the same results as is.na(). Give it a try.
my_data == NA
## [1] NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA
## [36] NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA
## [71] NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NAThe reason you got a vector of all NAs is that NA is not really a value, but just a placeholder for a quantity that is not available. Therefore the logical expression is incomplete and R has no choice but to return a vector of the same length as my_data that contains all NAs.
Don’t worry if that’s a little confusing. The key takeaway is to be cautious when using logical expressions anytime NAs might creep in, since a single NA value can derail the entire thing.
So, back to the task at hand. Now that we have a vector, my_na, that has a TRUE for every NA and FALSE for every numeric value, we can compute the total number of NAs in our data.
The trick is to recognize that underneath the surface, R represents TRUE as the number 1 and FALSE as the number 0. Therefore, if we take the sum of a bunch of TRUEs and FALSEs, we get the total number of TRUEs.
Let’s give that a try here. Call the sum() function on my_na to count the total number of TRUEs in my_na, and thus the total number of NAs in my_data. Don’t assign the result to a new variable.
Pretty cool, huh? Finally, let’s take a look at the data to convince ourselves that everything ‘adds up’. Print my_data to the console.
my_data
## [1] -0.45846931 NA -1.12657803 NA NA NA -1.09048852 NA
## [9] -1.55690067 NA NA 1.67486329 NA 0.49269816 NA NA
## [17] NA -1.63264633 1.23014273 1.33047973 NA NA NA NA
## [25] 1.13544659 NA -0.21045024 NA NA NA -0.55513668 NA
## [33] NA -0.17656367 NA NA -1.74375781 NA NA 0.01378357
## [41] NA 1.00223626 -1.50885012 NA NA NA NA NA
## [49] NA -0.78409872 0.63334383 -0.15503662 NA 1.07156519 NA 1.28444098
## [57] 0.09925662 NA 0.12955128 -1.90389019 NA -0.48038530 1.41362487 NA
## [65] 0.06248780 NA 0.62023705 1.17871653 0.68353558 0.23007686 -0.19126231 NA
## [73] NA NA 0.12886637 NA -1.19906955 0.62536170 1.21810066 NA
## [81] 0.77496580 0.01989049 -1.20148903 NA NA -0.47881021 1.63311904 NA
## [89] NA 0.52547535 0.13343761 0.28199154 NA 1.01971371 NA -0.80131440
## [97] NA NA NA -0.30344346Now that we’ve got NAs down pat, let’s look at a second type of missing value – NaN, which stands for ‘not a number’. To generate NaN, try dividing (using a forward slash) 0 by 0 now.
Let’s do one more, just for fun. In R, Inf stands for infinity. What happens if you subtract Inf from Inf?
You’ve successfully completed this lesson!