Experimental probability
There are things that are even beyond what a computer can find the exact theoretical probability for. Like when we want to figure out the probability of scoring a certain number of points in a basketball game. That’s not simple because that’s going to involve what human beings are doing.
For situations like that, it makes more sense to think more in terms of experimental probability. In experimental probability, we’re just trying to get an estimate of something happening, based on data and experience that we’ve had in the past.
But this has to be taken with a grain of salt. It’s just an estimate.
“I feel a little bit of reservations even calling it a probability.”
Sal Khan
This is for at least having a sense of what may happen.
Theoretical and experimental probabilities
If we have 50 blue, 50 green marbles in a bag, the probability of picking a blue marble is 1/2. Theoretically there is a 1/2 probability.
Let’s say we actually start doing experiments and we start picking marbles. Let’s say after 10 experiments we got 7 blue and 3 green. Like this, we’re not going to get exactly 1/2.
But let’s imagine that after 10’000 experiments, we got 8’000 blue and 2’000 green marbles. If the initial 1/2 was the true probability, then we would’ve expected the half to be blue and the other half green. This (having a 8k blue) is still within the realm of possibility (if the true probability of picking a blue is 1/2), but it’s very unlikely that we’ve gotten this result with this many experiments.
So this is going to cause us some pause. We would start to think about whether it’s truly equally likely for us to pick out a green versus a blue. Something else must be going on.
Experimental versus theoretical probability simulation: Randomness
The Law of Large Numbers tells how experimental probability should get closer to theoretical probability, as we conduct more and more experiments / trails.
Probability tells us how likely something is to happen in the long run. We can calculate probability by looking at the outcomes of an experiment or by reasoning about the possible outcomes.
Random number list to run experiment
Scenario: Collect all 6 prices that are in each box of cereal, assuming that they are equally likely to appear.
When we wonder how many boxes it takes, on average, to get all 6 prizes, we can use a random number list to run our experiments. Something like this:
If we want to figure out on average, we want to do many experiments. And the more experiments we do, the more likely that our average is going to predict what it actually takes on average to get all six prizes.
Statistical significance of an experiment
Scenario: We have 500 children that are asked to watch cartoons in a private room containing a bowl of crackers. The cartoon included two commercial breaks.
First group (treatment group) watched food commercials (mostly snacks), the second group (control group) watched non-food commercials (entertainment products).
We found that the mean amount of crackers eaten by the first group is 10 grams greater than the mean amount of crackers eaten by the second group.
At this point, we can ask ourselves: What’s the probability that this could happen by chance?
Now imagine, we re-randomize the results into two new groups (by using a simulator), and measure the difference between the means of the new groups. Let’s say we repeated this 150 times and plotted the resulting differences as given below.
According to the simulation, is the result of the experiment significant?
If we just randomly put people into two groups, the situations where we get a 10 gram difference are actually very unlikely. Because according to the above graph, it’s 2/150 times. In fact, if we check only the situation where a group on average eats 10gr more, it’s 1/150 times.
This tells us that our experiment was significant. It’s very unlikely that our experiment was purely due to chance. If this was just a chance event, this would only happen roughly one in 150 times. But the fact that this happened in our experiment, makes us feel confident that our experiment is significant.
In most studies, the threshold that researchers think about is, the probability of something statistically significant, if the probability of that happening by chance is less than 5%. In our case, it is less than 1%. We would definitely say that the experiment is significant.
Disclaimer: Like most of my posts, this content is intended solely for educational purposes and was created primarily for my personal reference. At times, I may rephrase original texts, and in some cases, I include materials such as graphs, equations, and datasets directly from their original sources.
I typically reference a variety of sources and update my posts whenever new or related information becomes available. For this particular post, the primary source was Khan Academy’s Statistics and Probability series.