Previously on Hypothesis Testing…

In the previous lecture, we learned about the following:

• Examples of the null and alternative hypothesis using difference in proportions.

• Basic ideas of the p-value and statistical significance.

Decision Errors

In this lecture, we will learn about:

• Decision Errors

Hypothesis Testing

A hypothesis test is a statistical technique used to evaluate competing claims using data.

• Null Hypothesis (\(H_0\)): A statement about a population parameter. We test the probability of this statement being true to decide whether to accept or reject. This hypothesis can include the \(=\), \(\ge\), or \(\le\) signs.

• Alternative Hypothesis (\(H_A\)): A statement that contradicts the null hypothesis. We determine the probability of this statement being true based on the likelihood of the null hypothesis being true. This hypothesis can include \(\ne\), \(>\), or \(<\) signs.

Outcomes of Hypothesis Testing

There are two possible outcomes of the hypothesis test:

• Reject \(H_0\): If the p-value is less than the significance level, then we reject the null hypothesis. Then, we have enough evidence to support \(H_A\).

• Fail to Reject \(H_0\): If the p-value is greater than or equal to the significance level, then we fail to reject the null hypothesis. This does not mean the the null hypothesis is true.

Making statistical decisions means that you have to deal with uncertainties.

Decision Errors

Image Source: Statistical Performance Measures by Neeraj Kumar Vaid

This meme might be over used. If you find some memes similar to this but in “non-pregnancy” context, let me know.

The Significance Level and Decisions Errors

What does this all mean? When the p-value is small, i.e., less than a previously set threshold (\(\alpha\)), we say the results are statistically significant. The value of \(\alpha\) represents how rare an event needs to be in order for the null hypothesis to be rejected. The \(\alpha\) also represents the probability of committing a type I error.

Trade-offs between Type I and Type II errors. (1/2)

Trade-offs between Type I and Type II errors. (1/2)

Images Source: Type I and Type II errors by Pritha Bhandari

Take-Home Message

• Note: Making a Type I error does not necessarily mean something was wrong with the data or that we made a computational mistake. Sometimes data can point us to the wrong conclusion. Scientific studies are often repeated to check initial findings. This is why reproducibility in Science is important!

• The significance value \(\alpha\) is the probability of making a Type I error.

• The power of the test \(1-\beta\) is the probability of rejecting the null claim when the alternative claim is true.

Examples (1/3)

• Question: In a US court, the defendant is either innocent (\(H_0\)) or guilty (\(H_A\)). What does a Type I Error represent in this context? What does a Type II Error represent?

• Answer: If the court makes a Type I Error, this means the defendant is innocent (\(H_0\) is true) but wrongly convicted. A Type II Error means the court failed to reject \(H_0\) (i.e., failed to convict the person) when they were in fact guilty (\(H_A\) true).

Examples (2/3)

• Question: Consider the opportunity cost study where we concluded students were less likely to make a DVD purchase if they were reminded that money not spent now could be spent later. What would a Type 1 Error represent in this context?

• Answer: Making a Type 1 Error in this context would mean that reminding students that money not spent now can be spent later does not affect their buying habits, despite the strong evidence (the data suggesting otherwise) found in the experiment.

Examples (2/2)

• Example: Consequences of a Type I error

Based on the incorrect conclusion that the new drug intervention is effective, over a million patients are prescribed the medication, despite risks of severe side effects and inadequate research on the outcomes. The consequences of this Type I error also mean that other treatment options are rejected in favor of this intervention.

• Example: Consequences of a Type II error

If a Type II error is made, the drug intervention is considered ineffective when it can actually improve symptoms of the disease. This means that a medication with important clinical significance doesn’t reach a large number of patients who could tangibly benefit from it.

Examples Source: Type I and Type II errors by Pritha Bhandari

Summary

In this lecture we talked about:

• Decision errors: Type I and Type II errors.

• The significance level in the context of the probability of making a Type I error.

• The power of the test.

In the next lectures, we will talk about:

• One-sided vs Two-sided hypothesis test.

• An introduction to confidence intervals and Randomization vs Bootstrapping.

• Mathematical details of the standard normal distribution. (Please read your textbook Chapter 13) Put on your math masks!