STATS 250

More hypothesis Testing for a Population Proportion, Type 1 and 2 Errors, and Power

Two-sided p-values

If you compute a two-sided \(p-value\), you compute a z-test. This p-value must be two sided, so both large and small values are considered extreme. Hence:

$$\text{p-value} = P(z \leq \hat{p}) + P(z \geq -\hat{p}) = 2P(z \leq \hat{p})$$

For small \(n\): binomial p-test

If you have a small sample, then:

$$np_0 < 10 \text{ or } n(1-p_0) < 10$$

If this is true, then you have to go back to the binomial distribution. Instead of looking at the proportion, you look at the count \(X\) of successes. You can actually do tests with this binomial!

To test the hypothesis \(H_0: p = p_0\) you compute the count test statistic. If you have \(X\) as the number of successes in a sample of size \(n\) which has a \(Bin(n, p_0)\) distribution, then you can use the binomial distribution to compute the p-value for the test.

Let's say you have 10 subjects that you treated with a new treatment, and 9 improved. You think that the accepted value \(p\) is 0.5.

We can find the probability that \(X \geq 9\):

$$P(X \geq 9) = P(X=9)+P(X=10) = (10 \choose 9)(0.50)^9(0.50) + (10 \choose 10)(0.50)^{10} = 0.0107 = p$$

Cautions

  • When there is a small to moderate effect in the population, a small sample has little chance of providing statistically significant support for the alternative hypothesis
  • With a large sample, even a small and unimportant effect in the population may lead to a conclusion of statistical significance.
  • The p-value tells us about the statistical significance, but not the size of the effect.

Type I and Type II error

  • Type I: Rejecting \(H_0\) when \(H_0\) is true (false positive)
  • Type II: Failing to reject \(H_0\) when \(H_a\) is true

Power of the Test

The probability of a type one error is just \(\alpha\)! The probability of a type two error is denoted as \(\beta\).

The power of the test is:

$$\text{power} = 1 - P(\text{Type 2 error}) = 1 - \beta$$