## Welcome to this App

The reproducability crisis has been a hot topic of debate in all sciences, but particular in psychological science.

The failure to reproduce assumingly established findings has led to an agitation in both the media

and the scientific community. The current situation has more than once been labelled as a replication crisis.

This educational app can be used as a visualisational aid to understand the statistical fundamentals of null

hypothesis testing and how these fundamentals relate to the current replication crisis.

In the Power Plot tab, there is a visualization and explaination of the relationship between alpha, effect size, sample size

and power. In the Hit/False Alarm Ratio tab, the relationship between prevalence, alpha and power is visualized.

Finally, in the Replication tab, the prevalence of true hypotheses given a specific replication rate is calculated and visualized.

### ENJOY!

### Explanation

In nullhypothesis testing (NHT) we search for the probability to find our data or more extreme, given that our nullhypothesis is true. We compare this probability, known as the p-value, to an arbitrary cut-off within the probability distribution (alpha). When we find that the p-value is smaller than this cut-off value, we assume that it is so unlikely for our data to come from our NH-distribution, that we choose to reject the NH.

When the Null hypothesis is true:

- Alpha is the arbitrary cut-off level to which we compare our p-value. This value is also the probability of a Type 1 error. This error refers to the situation that we reject the null hypothesis while in reality it is true.

- The true negative refers to the probability of not rejecting the null hypothesis when in reality the hypothesis is true. The probability of a true negative is equal to 1 - alpha.

When the null hypothesis is not true:

- Power refers to the probability to find an effect when in reality it really exists, i.e. the probability to reject the null hypothesis when the alternative hypothesis is true. Power is dependent on sample size and effect size and is equal to 1 - beta.

- Beta is the probability of a Type 2 error. This error occurs when in reality the null hypothesis is not true, but we fail to reject it.

When the Null hypothesis is true:

- Alpha is the arbitrary cut-off level to which we compare our p-value. This value is also the probability of a Type 1 error. This error refers to the situation that we reject the null hypothesis while in reality it is true.

- The true negative refers to the probability of not rejecting the null hypothesis when in reality the hypothesis is true. The probability of a true negative is equal to 1 - alpha.

When the null hypothesis is not true:

- Power refers to the probability to find an effect when in reality it really exists, i.e. the probability to reject the null hypothesis when the alternative hypothesis is true. Power is dependent on sample size and effect size and is equal to 1 - beta.

- Beta is the probability of a Type 2 error. This error occurs when in reality the null hypothesis is not true, but we fail to reject it.

### Prevalence of Experiments with true effect

### After running experiments: true and false significant effects

### How many of the significant results are really true?

### Explanation

The null hypothesis testing procedure is probabilistic. That is: there is always a (small) probability that an effect is not found (beta) or that the data seemed to show an effect by chance (alpha). Because of this, there will always be wrong decisions in null hypothesis testing.

Different situations can occur:

- True positive: In reality the alternative hypothesis is TRUE and through NHT the nullhypothesis is rejected.

- True negative: In reality the alternative hypothesis is FALSE and through NHT the nullhypothesis is NOT rejected.

- False positive: In reality the alternative hypothesis is FALSE but through NHT the nullhypothsis is rejected.

- False negative: In reality the alternative hypothesis is TRUE but through NHT the nullhypothesis is NOT rejected.

The amount of False/True Positives/Negatives is dependent on the chosen alpha and beta levels and the prevalence of true alternative hypothesis in the scientific field. That is: how many alternative hypothesis of all (interesting) tested hypothesis are in reality true.

### Corresponding prevalence

In this plot the relationship between replication rate and prevalence is shown. Some information that can be obtained from this plot:

- The prevalence of true alternative hypothesis in the scientific field can be obtained when a replication rate is known (e.g. in a recent study a replication rate of 0.36 is found; The Reproducibility Project: Psychology (Open Science Collaboration, 2015).

- Given a certain prevalence, the replication rate will differ if other values for alpha and beta are chosen.

This app was made by Annika Nieper, Felix Wolff, Marius Kroesche, Sofieke Kevenaar & Zenab Tamimy