Hello and welcome to a video on collapsing variables. Let's start as usual by loading in our data and any packages we
need to access. First we'll be importing pandas as PD, NumPy as NP, and the statistics package for some
stats that we're going to be calculating at the bottom. Just the next we'll mount our
drive, set our file path, read in our data, and today we're going to create the same data that you've created
from assignment one. This may look a little different from semester to semester, but you'll recognized it as similar.
Let's go ahead and run that. Okay, so now we have all of our data created
for us, all of our variables set and ready to go. We have several reasons to collapse a variable
when working with data. We might collapse data that has too many categories that make it cumbersome to visualize or interpret.
We might collapse data so that we can use certain types of statistical analyses on it. For example, we may want to perform
a chi-square test on our data, but it is continuously measured. In that case, we would want to make it categorical.
A categorical variable is one where the data can fall into one of several categories. One example may be job status,
employed or on employed. There are several ways we can collapse data. Let's use NumPy to solve this problem since
we're already familiar with it. However, you should know that there are several other methods you can use and would produce
the same results. We want to find all the values of RIV1 that are above and below the mean.
Let's pretend RIV1 represents a score in seconds on a speed test, and we'd like to know who performed above the
mean. For those above the mean, we'll give them a score of one, and those below, we'll get a score of zero. First,
we need to know the mean of RIV1. We've already calculated the mean in a previous video, so if you need to refresher on this,
go back to that. Okay, and we've got our mean set ready to go.
Now, we want to split RIV1 into two categories, using a new variable. We'll
call it D_R_IV1 for dichotomous RIV1.
So D_R_IV1 is equal to np.where. This is the function we'll be using.
And it's where from the numpy library. Where RIV1 is less
than or equal to the mean, put a zero, otherwise put a one.
So what they're saying, if it's less than or equal to this mean, it should have a zero there. If it's above that
mean, it should have a one.
Okay, let's go ahead and run that. And you can see now we've got a column of ones and zeros here.
This collapsed our variable into two categories. Those above, and those below the mean, usually just collapsing the
variable is the first step. Let's collapse another variable and then create a contingency table using the two.
So you'll will do the same thing again where we find the mean of R_DV.
And then we'll do the same thing where we create dichotomous rdv by finding
rdv less than or equal to the mean of rdv. Put a zero, otherwise put a one.
And then we'll print that. Great.
Now that we have two variables that are collapsed, let's make a two by two contingency tables so we can perform some basic calculations.
Okay. So what this code has done, I've created a contingency table just naming it contingency
table. No need to get too creative here. We're using the pd.crosstab
function from pandas. So we're taking df d_r_iv1
and d_r_dv. And we're creating a little table
from them. These are the areas where both had zero, where one had zero, one had one, one had
zero and one had one. And this is where they both had ones in the category. And to double-check our
work, we can add up the sum of all those values to make sure we're not missing anything. We know we have 170 rows of data,
remember Python starts numbering rows at zero. So those four numbers should add up to 170. They
do. Let's add some labels to our rows and columns for clarity. To do this,
we'll call contingencytable.columns. So we're using a method here this time to name
the columns and then contingencytable.index to name the rows.
Okay, and now you can see above here, we had zero and zero. So that's
below the mean and below the mean. There's 45 there. We have 14 that are above
the mean and below the mean and the other.
43 where RDV was below the mean and RIV. One was above the mean and 68 where they were both above
the mean. All right. That's it for collapsing variables. Have fun
coding.