class: center, middle ## Testing Conjectures <img src="img/DAW.png" width="450px"/> <span style="color: #91204D;"> .large[Kelly McConville | Math 141 | Week 9 | Fall 2020] </span> --- ## Announcements/Reminders * Lab 7 due this week before your lab meeting. --- ## Week 9 Topics * Testing Conjectures ********************************************* ### Goals for Today * Learn the language of hypothesis testing (including p-values) * Practice framing research questions in terms of hypotheses --- ### Hypothesis Testing **Big Idea:** * Make an assumption about the population parameter. -- * Generate a sampling distribution for a *test* statistic based on that assumption. -- * See if the test statistic based on the observed sample aligns with the generated sampling distribution or not. -- * If it does, then we didn't learn much. (Didn't prove the parameter equals the assumed value but it is still plausible) -- * If it doesn't, then we have evidence that our assumption about the parameter was wrong. --- ### ESP Example **Big Idea:** * Make an assumption about the population parameter. + <span style="color:orange">ESP doesn't exist. p, probability of guessing correctly, equals 0.25.</span> -- * Generate a sampling distribution for a *test* statistic based on that assumption. <img src="wk09_mon_files/figure-html/unnamed-chunk-1-1.png" width="360" style="display: block; margin: auto;" /> --- ### ESP Example **Big Idea:** * See if the test statistic based on the observed sample aligns with the generated sampling distribution or not. + <span style="color:orange"> It is in the center-ish of the distribution. It isn't an unusual value.</span> * If it does, then we didn't learn much. (Didn't prove the parameter equals the assumed value but it is still plausible) + <span style="color:orange"> It is still possible that ESP doesn't exist.</span> <img src="wk09_mon_files/figure-html/unnamed-chunk-2-1.png" width="360" style="display: block; margin: auto;" /> --- ### ESP Example **Big Idea:** * See if the test statistic based on the observed sample aligns with the generated sampling distribution or not. + <span style="color:orange"> It is far in the tails of the distribution. It is an unusual value.</span> * If it doesn't, then we have evidence that our assumption about the parameter was wrong. + <span style="color:orange"> We have evidence ESP exists.</span> <img src="wk09_mon_files/figure-html/unnamed-chunk-3-1.png" width="360" style="display: block; margin: auto;" /> --- ## Let's Take a Step Back from Our Last Statement... * Two important words in data analysis: + Reproducibility + Replicability -- * **Reproducibility**: If I give you the raw data and my write-up, you will get to the exact same final numbers that I did. -- * **Replicability**: If you follow my study design but collect new data (i.e. repeat my study on new subjects), you will come to the same conclusions that I did. -- * Science is going through a **replication crisis** right now. + [In cancer science, many "discoveries" don't hold up](https://www.reuters.com/article/us-science-cancer-idUSBRE82R12P20120328) + [Estimating the reproducibility of psychological science](https://science.sciencemag.org/content/349/6251/aac4716) + [Psychology Is Starting To Deal With Its Replication Problem](https://fivethirtyeight.com/features/psychology-is-starting-to-deal-with-its-replication-problem/) --- ### Hypothesis Testing Framework Have two competing hypothesis: * Null Hypothesis `\((H_o)\)`: Dull hypothesis, status quo, random chance, no effect... -- * Alternative Hypothesis `\((H_a)\)`: (Usually) contains the researchers' conjecture. -- Must first take those hypotheses and translate them into statements about the population parameters so that we can test them with sample data! -- #### Example: `\(H_o\)`: ESP doesn't exist. `\(H_a\)`: ESP does exist. -- Then translate into a statistical problem: -- `\(p\)` = probably of guessing correctly out of 4 images `\(H_o\)`: `\(p = 0.25\)` (or `\(p \leq 0.25\)` ) `\(H_a\)`: `\(p > 0.25\)` --- class: center, middle, inverse ### Let's practice framing the hypotheses. --- ### Hypothesis Testing Framework Flavors of hypotheses: * `\(H_o\)`: parameter `\(=\)` null value -- * One of the following: + `\(H_a\)`: parameter `\(\neq\)` null value + `\(H_a\)`: parameter `\(>\)` null value + `\(H_a\)`: parameter `\(<\)` null value -- **Question**: But doesn't `\(H_o\)` sometimes represent `\(\leq\)` or `\(\geq\)`? --- ### Hypothesis Testing Framework Once you have set-up your hypotheses... * Collect data. -- * Assume `\(H_o\)` is correct. -- * Quantify the likelihood of the sample results using a test statistic. -- * **Test statistic**: Numerical summary of the sample data + Often is equal to the sample statistic. --- ### Hypothesis Testing Framework -- Null Distribution * **Null distribution**: Sampling distribution of the test statistic if the null hypothesis is true. -- **Question**: How do we use the null distribution to quantify the likelihood of the sample results? -- **p-value** = Probability of the observed test statistic or more extreme if `\(H_o\)` is true -- * More extreme = direction of `\(H_a\)` -- * Find the proportion of test statistics in the null distribution that are equal to or more extreme that the observed test statistic -- * If the p-value is small, we have evidence for `\(H_a\)`. + Notice I am talking about `\(H_a\)`, not `\(H_o\)` here! -- * If the p-value is not small, we don't have evidence for `\(H_a\)`. --- class: center, middle, inverse ### Let's see the ESP example (one more time) but now using `infer`. The "hypothesisTestingFramework.Rmd" file can be found in the Handouts folder.