This textbook is primarily focused on designing research, collecting data, and becoming a knowledgeable and responsible consumer of research. We won’t spend as much time on data analysis or what to do with our data once we’ve designed a study and collected it, but I will spend some time describing some important basics of data analysis that are unique to each method. Entire textbooks could be (and have been) written entirely on data analysis. In fact, if you’ve ever taken a statistics class, you already know much about how to analyze quantitative survey data. Here we’ll go over a few basics that can get you started as you begin to think about turning data from surveys and experiences into findings that you can share.

Who responds to your questionnaire?

It can be very exciting to receive those first few completed questionnaires back from respondents. Hopefully you’ll even get more than a few back, and once you have a handful of completed questionnaires, your feelings may go from initial euphoria to dread. Data are fun and can also be overwhelming. The goal with data analysis is to be able to condense large amounts of information into usable and understandable chunks.

In an experiment, as long as no one drops out, you can be assured that everyone in your sample will complete your questionnaires as part of their pretest and posttest. For surveys, it is much less likely that everyone will complete your questionnaire. The hope is that you will receive a good portion of the questionnaires you distributed back in a completed and readable format. The number of completed questionnaires you receive divided by the number of questionnaires you distributed is your response rate. Let’s say your sample included 100 people and you sent questionnaires to each of those people. It would be wonderful if all 100 returned completed questionnaires, but the chances of that happening are about zero. If you’re lucky, perhaps 75 or so will return completed questionnaires. In this case, your response rate would be 75%. The response rate is calculated by dividing the number of surveys returned by the number of surveys distributed.

Though response rates vary, and researchers don’t always agree about what makes a good response rate, having 75% of your surveys returned would be considered good—even excellent—by most survey researchers. There has been a lot of research done on how to improve a survey’s response rate. We covered some of these previously, but suggestions include personalizing questionnaires by, for example, addressing them to specific respondents rather than to some generic recipient, such as “madam” or “sir”; enhancing the questionnaire’s credibility by providing details about the study, contact information for the researcher, and perhaps partnering with agencies likely to be respected by respondents such as universities, hospitals, or other relevant organizations; sending out pre-questionnaire notices and post-questionnaire reminders; and including some token of appreciation with mailed questionnaires even if small, such as a $1 bill.

The major concern with response rates is that a low rate of response may introduce nonresponse bias into a study’s findings. What if only those who have strong opinions about your study topic return their questionnaires? If that is the case, we may well find that our findings don’t at all represent how things really are or, at the very least, we are limited in the claims we can make about patterns found in our data. While high return rates are certainly ideal, a recent body of research shows that concern over response rates may be overblown (Langer, 2003). ^[1] Several studies have shown that low response rates did not make much difference in findings or in sample representativeness (Curtin, Presser, & Singer, 2000; Keeter, Kennedy, Dimock, Best, & Craighill, 2006; Merkle & Edelman, 2002). ^[2] For now, the jury may still be out on what makes an ideal response rate and on whether, or to what extent, researchers should be concerned about response rates. Nevertheless, certainly no harm can come from aiming for as high a response rate as possible.

Building a codebook

Regardless of your response rate, a major concern of quantitative researchers once they have their big stack of completed questionnaires is condensing their data into manageable and analyzable, bits. One major advantage of quantitative methods such as surveys and experiments, as you may recall from Chapter 1, is that they enable researchers to describe large amounts of data because they can be represented by and condensed into numbers.

In order to condense your completed surveys into analyzable numbers, you’ll first need to create a codebook. A codebook is a document that outlines how a survey researcher has translated her data from words into numbers. An excerpt from a codebook can be seen in Table 12.2. As you’ll see in the table, in addition to converting response options into numerical values, a short variable name is given to each question. This shortened name comes in handy when entering data into a computer program for analysis.

The next task after creating your codebook is data entry. If you’ve utilized an online tool such as SurveyMonkey to administer your questionnaire, here’s some good news—most online survey tools come with the capability of importing survey results directly into a data analysis program. Trust me—this is excellent news. (If you don’t believe me, I highly recommend administering hard copies of your questionnaire next time around. You’ll surely then appreciate the wonders of online survey administration!)

For those who will be conducting manual data entry, there probably isn’t much I can say about this task that will make you want to perform it other than pointing out the reward of having a data set of your very own ready to analyze. At best, it is a Zen-like practice akin to raking sand. At worst, it is mind-numbingly boring. While you can pay someone else to do your data entry for you, a common practice with undergraduate and graduate research assistants, you should ask yourself whether you trust someone else to make no errors in entering your data. If errors are made in data entry, it can jeopardize the results of your project. You may want to consider whether it is worth your time and effort to do your data entry yourself.

We won’t get into too many of the details of data entry, but I will mention a few programs that survey researchers may use to analyze data once it has been entered. The first is SPSS or the Statistical Package for the Social Sciences (http://www.spss.com). SPSS is a statistical analysis computer program designed to analyze just the sort of data quantitative survey researchers collect. It can perform everything from very basic descriptive statistical analysis to more complex inferential statistical analysis. SPSS is touted by many for being highly accessible and relatively easy to navigate (with practice). Other programs that are known for their accessibility include MicroCase (http://www.microcase.com/index.html), which includes many of the same features as SPSS, and Excel, which is far less sophisticated in its statistical capabilities but is relatively easy to use and suits some researchers’ purposes just fine.

Identifying patterns

Data analysis is about identifying, describing, and explaining patterns. Univariate analysis is the most basic form of analysis that quantitative researchers conduct. In this form, researchers describe patterns across just one variable. Univariate analysis includes frequency distributions and measures of central tendency. A frequency distribution is a way of summarizing the distribution of responses on a single survey question. Let’s look at the frequency distribution for just one variable from a survey of older workers. We’ll analyze the item mentioned first in the codebook excerpt given earlier, which is on respondents’ self-reported financial security.

As you can see in the frequency distribution on self-reported financial security, more respondents reported feeling “moderately secure” than any other response category. We also learn from this single frequency distribution that fewer than 10% of respondents reported being in one of the two most secure categories.

Another form of univariate analysis that survey researchers can conduct on single variables is measures of central tendency. Measures of central tendency can be taken for variables at any level of measurement we reviewed in Chapter 9—from nominal to ratio. There are three measures of central tendency: modes, medians, and means. Mode refers to the most common response given to a question. Modes are most appropriate for nominal-level variables. A median is the middle point in a distribution of responses. In the previous example, if you wrote out all 180 responses to the question, side by side, from smallest to largest (1,1,1….5,5,5), the median would be the middle number. Finally, the measure of central tendency used for interval- and ratio-level variables is the mean. More commonly known as an average, means can be obtained by adding the value of all responses on a given variable and then dividing that number of the total number of responses.

Median is the appropriate measure of central tendency for ordinal-level variables, though it is sometimes used for interval or ratio variables whose distribution contains outliers or extreme scores that would skew the mean higher than the true center of the distribution. For example, if you asked your four friends about how much money they have in their wallets and one of them just won the lottery, the mean would be quite high, even though most of you do not have near that amount. The median value would be closer to the true center, in this case, than the mean.

In the previous example of older workers’ self-reported levels of financial security, the appropriate measure of central tendency would be the median, as this is an ordinal-level variable. If we were to list all responses to the financial security question in order and then choose the middle point in that list, we’d have our median.

In Figure 12.2, the value of each response to the financial security question is noted, and the middle point within that range of responses is highlighted. To find the middle point, we simply divide the number of valid cases by two. The number of valid cases, 180, divided by 2 is 90, so we’re looking for the 90th value on our distribution to discover the median. As you’ll see in Figure 12.2, that value is 3; thus, the median on our financial security question is 3 or “moderately secure.”

^[3]

As you can see, we can learn a lot about our respondents simply by conducting univariate analysis of measures on our survey. We can learn even more, of course, when we begin to examine relationships across multiple variables. Either we can analyze the relationships between two variables, called bivariate analysis, or we can examine relationships among more than two variables. This latter type of analysis is known as multivariate analysis.

Bivariate analysis allows us to assess covariation among two variables. We reviewed covariation in Chapter 7. This means we can find out whether changes in one variable occur together with changes in another. If two variables do not covary, they are said to have independence. This means simply that there is no relationship between the two variables in question. To learn whether a relationship exists between two variables, a researcher may cross-tabulate the two variables and present their relationship in a contingency table. A contingency table shows how variation on one variable may be contingent on variation on the other.

Let’s take a look at a contingency table. In Table 12.4, I have cross-tabulated two questions from an older worker survey: respondents’ reported gender and their self-rated financial security.

You’ll see in Table 12.4 that I collapsed a couple of the financial security response categories (recall there were five categories presented in Table 12.3). Researchers sometimes collapse response categories on items such as this in order to make it easier to read results in a table. You’ll also see that I placed the variable “gender” in the table’s columns and “financial security” in its rows. Typically, values that are contingent on other values (dependent variables) are placed in rows, while independent variables are placed in columns. This makes comparing across categories of ourindependentvariableprettysimple.

Reading across the top row of our table, we can see that around 44% of men in the sample reported that they are not financially secure while almost 52% of women reported the same. In other words, more women than men reported they are not financially secure. You’ll also see in the table that I reported the total number of respondents for each category of the independent variable in the table’s bottom row. This is also standard practice in a bivariate table, as is including a table heading describing what is presented in the table.

Researchers interested in simultaneously analyzing relationships among more than two variables conduct multivariate analysis. If I hypothesized that financial security declines for women as they age but increases for men as they age, I might consider adding age to the preceding analysis. To do so would require multivariate, rather than bivariate, analysis. This is common in studies with multiple independent or dependent variables. It is also necessary for studies that include control variables, which almost all studies do. We won’t go into detail here about how to conduct multivariate analysis of quantitative survey items here. If you are interested in learning more about the analysis of quantitative survey data, I recommend checking out your campus’s offerings in statistics classes. The quantitative data analysis skills you will gain in a statistics class could serve you quite well should you find yourself seeking employment one day.