6.5 Probability and the Normal Distribution

Probability can seem like a daunting topic for many students. In a mathematical statistics course this might be true, as the meaning and purpose of probability gets obscured and overwhelmed by equations and theory. In this chapter we will focus only on the principles and ideas necessary to lay the groundwork for future inferential statistics. We accomplish this by quickly tying the concepts of probability to what we already know about normal distributions and [latex]z[/latex]-scores.

What Is Probability?

When we speak of the probability of something happening, we are talking how likely it is that “thing” will happen based on the conditions present. For instance, what is the probability that it will rain? That is, how likely do we think it is that it will rain today under the circumstances or conditions today? To define or understand the conditions that might affect how likely it is to rain, we might look out the window and say, “It’s sunny outside, so it’s not very likely that it will rain today.” Stated using probability language: given that it is sunny outside, the probability of rain is low. “Given” is the word we use to state what the conditions are. As the conditions change, so does the probability. Thus, if it were cloudy and windy outside, we might say, “Given the current weather conditions, there is a high probability that it is going to rain.”

In these examples, we spoke about whether or not it is going to rain. Raining is an example of an event, which is the catch-all term we use to talk about any specific thing happening; it is a generic term that we specified to mean “rain” in exactly the same way that “conditions” is a generic term that we specified to mean “sunny” or “cloudy and windy.”

It should also be noted that the terms “low” and “high” are relative and vague, and they will likely be interpreted different by different people (in other words: given how vague the terminology was, the probability of different interpretations is high). Most of the time we try to use more precise language or, even better, numbers to represent the probability of our event. Regardless, the basic structure and logic of our statements are consistent with how we speak about probability using numbers and formulas.

Let’s look at a slightly deeper example. Say we have a regular, six-sided die (note that die is singular and dice is plural) and want to know how likely it is that we will roll a [latex]1[/latex]. That is, what is the probability of rolling a [latex]1[/latex], given that the die is not weighted (which would introduce what we call a bias, though that is beyond the scope of this section). We could roll the die and see if it is a [latex]1[/latex] or not, but that won’t tell us about the probability, it will only tell us a single result. We could also roll the die hundreds or thousands of times, recording each outcome and seeing what the final list looks like, but this is time consuming, and rolling a die that many times may lead down a dark path to gambling or, worse, playing Dungeons & Dragons. What we need is a simple equation that represents what we are looking for and what is possible.

To calculate the probability of an event, which here is defined as rolling a [latex]1[/latex] on an unbiased die, we need to know two things: how many outcomes satisfy the criteria of our event (stated differently, how many outcomes would count as what we are looking for) and the total number of outcomes possible. In our example, only a single outcome, rolling a [latex]1[/latex], will satisfy our criteria, and there are a total of six possible outcomes (rolling a [latex]1[/latex], rolling a [latex]2[/latex], rolling a [latex]3[/latex], rolling a [latex]4[/latex], rolling a [latex]5[/latex], and rolling a [latex]6[/latex]). Thus, the probability of rolling a [latex]1[/latex] on an unbiased die is [latex]1[/latex] in [latex]6[/latex] or [latex]\frac{1}{6}[/latex]. Put into an equation using generic terms, we get:

[latex]\text{Probability of an event}=\frac{\text{Number of outcomes that satisfy our criteria}}{\text{Total number of possible outcomes}}[/latex]

We can also use [latex]P()[/latex] as shorthand for probability and [latex]A[/latex] as shorthand for an event:

[latex]P(A)=\frac{\text{Number of outcomes that count as A}}{\text{Total number of possible outcomes}}[/latex]

Using this equation, let’s now calculate the probability of rolling an even number on this die:

[latex]P(\text{even number})=\frac{2,4,\text{or}\;6}{1,2,3,4,5,\text{or}\;6}=\frac{3}{6}=\frac{1}{2}[/latex]

So we have a [latex]50\%[/latex] chance of rolling an even number of this die. The principles laid out here operate under a certain set of conditions and can be elaborated into ideas that are complex yet powerful and elegant. However, such extensions are not necessary for a basic understanding of statistics, so we will end our discussion on the math of probability here. Now, let’s turn back to more familiar topics.

Probability in Graphs and Distributions

We will see shortly that the normal distribution is the key to how probability works for our purposes. To understand exactly how, let’s first look at a simple, intuitive example using pie charts.

Probability in Pie Charts

Recall that a pie chart represents how frequently a category was observed and that all slices of the pie chart add up to [latex]100\%[/latex], or [latex]1[/latex]. This means that if we randomly select an observation from the data used to create the pie chart, the probability of it taking on a specific value is exactly equal to the size of that category’s slice in the pie chart.

Take, for example, the pie chart in Figure 6.5.1 representing the favourite sports of [latex]100[/latex] people. If you put this pie chart on a dart board and aimed blindly (assuming you are guaranteed to hit the board), the likelihood of hitting the slice for any given sport would be equal to the size of that slice. So, the probability of hitting the baseball slice is the highest at [latex]36\%[/latex]. The probability is equal to the proportion of the chart taken up by that section.

We can also add slices together. For instance, maybe we want to know the probability to finding someone whose favourite sport is usually played on grass. The outcomes that satisfy this criterion are baseball, football, and soccer. To get the probability, we simply add their slices together to see what proportion of the area of the pie chart is in that region: [latex]36\%+25\%+20\%=81\%[/latex]. We can also add sections together even if they do not touch. If we want to know the likelihood that someone’s favourite sport is not called football somewhere in the world (i.e., baseball and hockey), we can add those slices even though they aren’t adjacent or contiguous in the chart itself: [latex]36\%+20\%=56\%[/latex]. We are able to do all of this because (1) the size of the slice corresponds to the area of the chart taken up by that slice, (2) the percentage for a specific category can be represented as a decimal (this step was skipped for ease of explanation above), and (3) the total area of the chart is equal to [latex]100\%[/latex] or [latex]1.0[/latex], which makes the size of the slices interpretable.

Probability in Normal Distributions

If the language at the end of the last section sounded familiar, that’s because its exactly the language used in Section 6.4 to describe the normal distribution. Recall that the normal distribution has an area under its curve that is equal to [latex]1[/latex] and that it can be split into sections by drawing a line through it that corresponds to a given [latex]z[/latex]-score. Because of this, we can interpret areas under the normal curve as probabilities that correspond to [latex]z[/latex]-scores.

First, let’s look at the area between [latex]z=-1.00[/latex] and [latex]z=1.00[/latex] presented in Figure 6.5.2. We were told earlier that this region contains [latex]68\%[/latex] of the area under the curve. Thus, if we randomly chose a [latex]z[/latex]-score from all possible [latex]z[/latex]-scores, there is a [latex]68\%[/latex] chance that it will be between [latex]z==1.00[/latex] and [latex]z=1.00[/latex] because those are the [latex]z[/latex]-scores that satisfy our criteria.

Just like a pie chart is broken up into slices by drawing lines through it, we can also draw a line through the normal distribution to split it into sections. Take a look at the normal distribution in Figure 6.5.3, which has a line drawn through it at [latex]z=1.25[/latex]. This line creates two sections of the distribution: the smaller section called the tail and the larger section called the body. Differentiating between the body and the tail does not depend on which side of the distribution the line is drawn. All that matters is the relative size of the pieces: bigger is always body.

As you can see, we can break up the normal distribution into [latex]3[/latex] pieces (lower tail, body, and upper tail) as in Figure 6.5.2 or into [latex]2[/latex] pieces (body and tail) as in Figure 6.5.3. We can then find the proportion of the area in the body and tail based on where the line was drawn (i.e., at what [latex]z[/latex]-score). Mathematically, this is done using calculus. Fortunately, the exact values are given to you in the Standard Normal Distribution Table, also known at the [latex]z[/latex]-table. A portion of this table is shown in table 6.5.1 (the entire z-score table). Using the [latex]z[/latex] values in the table (A), we can find the area under the normal curve in any body and can calculate the area under the normal curve in any tail.

For example, suppose we want to find the area in the body for a [latex]z[/latex]-score of [latex]1.62[/latex]. As shown in Table 6.5.1, the row for [latex]1.62[/latex] corresponds with a value of [latex]0.9474[/latex] for the proportion in the body of the distribution. This cell has been highlighted in yellow to help locate it. Thus, the odds of randomly selecting someone with a [latex]z[/latex]-score less than (to the left of) [latex]z=1.62[/latex] is [latex]94.74\%[/latex] because that is the proportion of the area taken up by values that satisfy our criteria.

Mathematically, we present this solution in the following way:

[latex]P(z<1.62)=0.9474[/latex]

Some versions of the [latex]z[/latex]-table only present the area in the body for positive [latex]z[/latex]-scores because the normal distribution is symmetrical. The area in the body of [latex]z=1.62[/latex] is equal to the area in the body for [latex]z=-1.62[/latex], though now — as illustrated in the middle distribution at the top of Table 6.5.1 — the body will be the shaded area to the right of [latex]z[/latex]. (When in doubt, drawing out your distribution and shading the area you need to find will always help.) Because the total area under the normal curve is always equal to [latex]1.00[/latex], the area in the tail is simply the area in the body subtracted from [latex]1.00[/latex] ([latex]1.00-0.9474=0.0526[/latex]). We will also provide the negative side of the [latex]z[/latex]-score table in our course, so we will be able to use it to find probabilities associated with negative [latex]z[/latex]-scores.

Let’s look at another example. This time, let’s find the area corresponding to [latex]z[/latex]-scores more extreme than [latex]z=-1.96[/latex] and [latex]z=1.96[/latex]. That is, let’s find the area in the tails of the distribution for values less than [latex]z=-1.96[/latex] (farther negative and therefore more extreme) and greater than [latex]z=1.96[/latex] (farther positive and therefore more extreme). This region is illustrated in Figure 6.5.4.

Mathematically, we are looking for

[latex]P(z<-1.96)[/latex] and [latex]P(z>1.96)[/latex]

Let’s start with the tail for [latex]z=1.96[/latex]. Our [latex]z[/latex]-table only provides the area under the curve to the left of a particular [latex]z[/latex]-score. As such, finding [latex]P(z<-1.96)[/latex] is straightforward, and we need only look to the negative side of the [latex]z[/latex]-score table to find it. Looking at the negative side of the [latex]z[/latex]-score table, we find [latex]P(z<-1.96)=0.0250[/latex]. Since the distribution is symmetric, this means we can deduct that [latex]P(z>1.96)=0.0250[/latex] as well. Finally, to get the total area in the shaded region, we simply add the areas together to get [latex]0.0500[/latex]. Thus, there is a [latex]5\%[/latex] chance of randomly getting a value more extreme than [latex]z=-1.96[/latex] or [latex]z=1.96[/latex].

Finally, we can find the area between two [latex]z[/latex]-scores by shading and subtracting. Figure 6.5.5 shows the area between [latex]z=0.50[/latex] and [latex]z=1.50[/latex]. Because this is a subsection of a body (rather than just a body or a tail), we must first find the larger of the two bodies, in this case the body for [latex]z=1.50[/latex], and subtract the smaller of the two bodies, or the body for [latex]z=0.50[/latex]. Aligning the distributions vertically, as in Figure 6.5.5, makes this clearer. From the complete [latex]z[/latex]-table, we see that the area in the body for [latex]z>=1.50[/latex] is [latex]0.9332[/latex], and the area in the body for [latex]z=0.50[/latex] is [latex]0.6915[/latex]. Subtracting these gives us [latex]0.9332-0.6915=0.2417[/latex].

Mathematically, we write:

[latex]\begin{align*}P(0.50\le z\le 1.50)&=P(z\le 1.50)-P(z\le 0.50)\\[2ex]P(0.50\le z\le 1.50)&=0.9332-0.6915\\[2ex]P(0.50\le z\le 1.50)&=0.2417\end{align*}[/latex]

Standardizing Normal Distributions

In the examples above, we have been given the [latex]z[/latex]-scores directly. However, most of the time, we are studying a random variable [latex]X[/latex] where the mean and standard deviation are different from zero and one respectively. Thus, we need to use a process called standardization, sometimes referred to in mathematical notation as [latex]X\sim N(\mu, \sigma)[/latex] to [latex]Z\sim N(0,1)[/latex], in order to be able to find the probabilities using [latex]z[/latex]-scores. To do this, use the formula,

[latex]z=\frac{x-\mu}{\sigma}[/latex]

Using this formula standardizes the random variable and finds the [latex]z[/latex]-score associated with the values of interest so that we can find the probabilities in the [latex]z[/latex]-table.

Using Percentiles and Z-scores to Find Original Value

A useful tool when studying normal distributions is being able to track backwards to find the value on the original distribution that is associated with a particular [latex]z[/latex]-score or percentile. We can do this by a process sometimes called “un-standardizing”. Essentially, we are just rearranging the standardizing formula to solve for [latex]x[/latex].

[latex]\begin{align*}z&=\frac{x-\mu}{\sigma}\\[2ex]\sigma z&=x-\mu\\[2ex]\sigma z+\mu&=x\end{align*}[/latex]

So we can find the original data value by using the formula: [latex]x=\mu +z\sigma[/latex].

Probability: The Bigger Picture

The concepts and ideas presented in this chapter are likely not intuitive at first. Probability is a tough topic for everyone, but the tools it gives us are incredibly powerful and enable us to do amazing things with data analysis. They are the heart of how inferential statistics work.

To summarize, the probability that an event happens is the number of outcomes that qualify as that event (i.e., the number of ways the event could happen) compared to the total number of outcomes (i.e., how many things are possible). This extends to graphs like a pie chart, where the biggest slices take up more of the area and are therefore more likely to be chosen at random. This idea then brings us back around to our normal distribution, which can also be broken up into regions or areas, each of which is bounded by one or two [latex]z[/latex]-scores and corresponds to all [latex]z[/latex]-scores in that region. The probability of randomly getting one of those [latex]z[/latex]-scores in the specified region can then be found on the Standard Normal Distribution Table. Thus, the larger the region, the more likely an event is, and vice versa. Because the tails of the distribution are, by definition, smaller and we go farther out into the tail, the likelihood or probability of finding a result out in the extremes becomes small.