What is P-Value?

The concept of p-value is often a topic of confusion and misuse among researchers, both early career and experienced ones. However, understanding and correctly interpreting the p-value is crucial in quantitative research. In this article, we will delve into what a p-value is, how to calculate it, and its statistical significance.

What is a P-Value?

The p-value, short for probability value, is the probability that your research results occurred randomly, assuming that the null hypothesis is true. P-values are commonly used in hypothesis testing to determine if there are significant differences between values or groups. The calculation of p-values is based on the test statistic for the specific test being used and relies on the known or assumed probability distribution.

For instance, let’s consider a simplified example where you are studying a new pain medicine designed to have a longer duration of effectiveness compared to the currently prescribed drug. Suppose you already know that the underlying probability distribution for both drugs follows a normal distribution. In this case, you can visualize the probability distribution function using a graph.

In your clinical trial, if your results show that the average duration of pain relief is longer for the new drug compared to the standard drug, how can you determine if this difference is statistically significant and not just a random outcome? One way to do this is through hypothesis testing.

What is a Null Hypothesis?

Formulating the null and alternative hypotheses is the initial step in hypothesis testing. The null hypothesis (H0) is the statement you are trying to disprove, typically asserting that there is no relationship between two variables or no difference between two groups. On the other hand, the alternative hypothesis (Ha) states that a relationship exists or there is a difference between the groups. The alternative hypothesis represents what you are trying to find evidence to support.

In our clinical trial example, we can create the following hypotheses:

• H0: The mean duration of the new drug’s effectiveness is equal to that of the standard drug.
• Ha: The mean duration of the new drug’s effectiveness is greater than that of the standard drug.

It is important to note that the null hypothesis assumes no difference in mean values between the two drugs. Since Ha includes the term “greater than,” this is an upper-tailed test, and we are only interested in the area under the right side of the curve.

To determine whether or not the null hypothesis can be rejected, we need to establish a criterion. This is where the critical p-value comes into play. The critical p-value corresponds to the point at which the null hypothesis can be rejected. It represents the probability of rejecting the null hypothesis when it is true and is set before conducting the experiment. The commonly used value for the critical p-value is p = 0.05, although it can vary depending on the circumstances.

To make the concept clearer, let’s compare the null hypothesis to the phrase “innocent until proven guilty.” The null hypothesis is assumed to be true unless strong evidence is found to disprove it. Statistically significant p-values can provide some of that evidence, which highlights the importance of knowing how to calculate p-values.

How to Calculate P-Values

The p-value is calculated based on the test statistic, which depends on the type of hypothesis test being conducted. Since the p-value is a probability, its calculation method and value depend on the underlying probability distribution. The p-value also varies depending on whether the test is lower-tailed, upper-tailed, or two-tailed.

To obtain the actual p-value, the probability distribution function is integrated to find the relevant areas under the curve using integral calculus. This process can be complex, but fortunately, p-values are commonly determined using tables or statistical software such as SPSS, SAS, or R.

In our simplified clinical trial example, where the underlying probability distribution is assumed to be normal, we can use a t-test to test the null hypothesis. The resulting t-test statistic will indicate the position of our result along the x-axis under the normal curve. The p-value, in this case, would be the area under the curve to the right of the test statistic.

It is important to select the appropriate hypothesis test and test statistic based on the characteristics of your data and the relationship you are investigating. The sample size and the number of independent variables can also impact the p-value.

P-Value and Statistical Significance

After completing your clinical trial and obtaining the p-value, the next step is to interpret the results and understand their significance. What does it mean when a result is statistically significant?

A statistically significant result indicates that the obtained p-value is small enough to suggest that the observed differences are unlikely to occur by chance alone. P-values range from 0 to 1, and the smaller the p-value, the less likely it is that the null hypothesis is true and the stronger the evidence for rejecting it. The critical p-value, set before the experiment, determines the threshold at which a result can be considered statistically significant. The commonly used critical p-value is 0.05.

In our clinical trial example, if the obtained p-value is p = 0.0375 and the critical p-value is set at 0.05, we can conclude that the results are statistically significant. This means that we have evidence to reject the null hypothesis. However, it is important to note that statistical significance does not provide absolute certainty that the null hypothesis is false. It simply indicates that the null hypothesis is likely false based on the results of the test.

P-Value Table

Interpreting the results of an experiment or trial based on p-values can be facilitated by using a p-value table. This table, prepared before the experiment, provides possible p-values and their corresponding interpretations.

| P-Value Range | Interpretation |
|—————|—————-|
| p > 0.05 | Results are not statistically significant; do not reject the null hypothesis |
| p < 0.05 | Results are statistically significant; generally reject the null hypothesis |
| p < 0.01 | Results are highly statistically significant; reject the null hypothesis |

Using this table, you can easily determine the significance of your results based on the obtained p-value.

How to Report P-Values in Research

In research papers, p-values, like other experimental outcomes, are typically reported in the results section and sometimes in the abstract. It is essential to provide enough information for readers to understand the context of the p-values. In addition to reporting the p-values, it is also important to include the test statistic and effect size in the results section.

To ensure clear understanding of your results, it is recommended to report the significance threshold (critical p-value) used in the methods section of your paper. For example, you can state, “In this study, the statistical significance threshold was set at p = 0.05.” Additionally, discuss the sample sizes and assumptions in the methods section, as they can significantly impact the p-value.

Comparing Two Results of a Hypothesis Test Using P-Values

What if you conduct two experiments with the same null and alternative hypotheses? Or what if you repeat the same clinical trial with different drugs? Can you compare the resulting p-values?

In general, it is not advisable to directly compare results using only p-values. A p-value only reflects the probability of the specific results occurring by chance and does not provide any information about other results or their magnitude. Therefore, obtaining a p-value of 0.04 with one drug and 0.025 with another drug does not necessarily indicate that the second drug is better.

Comparing two different results using p-values may be more feasible if the experiments are exactly the same and all other conditions are controlled, except for the variable being studied. However, considering the multitude of factors that can impact the p-value, it would be challenging to control all of them.

Why Using P-Values Alone is Insufficient for Interpreting Two Different Variables

While p-values can indicate whether or not the null hypothesis should be rejected, they are not sufficient for comparing the size differences between variables. Therefore, when discussing the results of a study, it is important to report both the statistical significance (p-value) and the effect size.

For example, suppose the sample size in our clinical trials is very large, such as 1,000, and the p-value is 0.035. The statistically significant difference between the drugs is determined based on the p-value being less than 0.05. However, when examining the actual difference in the duration of effectiveness between the drugs, we find that the new drug only lasts 2 minutes longer than the standard drug. Larger sample sizes often show even very small differences to be statistically significant. Therefore, incorporating information about effect size is necessary to make meaningful recommendations based on trial results.

Statistical significance, effect size, and p-values are all influenced by sample size. Therefore, it is crucial to report all these measures to facilitate clear understanding of the results.

Considerations When Using P-Values

While p-values are valuable tools for researchers, caution must be exercised to avoid treating them as definitive indicators of study results or misusing them. Here are a few additional considerations when using p-values:

1. Pay attention to the formatting guidelines provided by your target journal when reporting p-values in your research report. Typically, p-values are written without a leading zero and italicized. For instance, write p = 0.01 instead of p = 0.01.

2. The significance level (alpha) should be set before conducting the experiment. Setting the significance level after examining the data to ensure a positive result is considered unethical.

3. P-values do not provide any evidence for the alternative hypothesis. Rejecting the null hypothesis does not mean accepting the alternative hypothesis.

4. P-values do not prove anything; they can only provide evidence to support rejecting or not rejecting the null hypothesis. Statistics are inherently cautious.

5. Avoid using the term “insignificant” to describe results that do not meet the threshold for statistical significance. Instead, use terms like “not statistically significant” or “not significant at the specified level.”

In conclusion, understanding the concept of p-value is essential for researchers conducting quantitative research. By correctly interpreting and reporting p-values, researchers can effectively communicate the significance of their findings.