Before collecting data, a researcher picks a significance level, usually α=0.05. After the test, the rule is mechanical: p=0.03 is below 0.05, so the result is significant and H0 is rejected; p=0.08 is not, so H0 stands. The same p=0.03 would not be significant at α=0.01, which is why the level must be fixed in advance: choosing it after seeing p lets you get whichever verdict you wanted.
The level is also an error rate. When the null is true, a test's p-value is equally likely to fall anywhere between 0 and 1, so it lands below 0.05 in 5% of studies; that is the false-alarm rate you accept. Test many true nulls and some will come out significant by luck alone, which is why testing until something works is cheating. In the other direction, p=0.08 does not show the effect is zero; the study may simply be too small to detect it.
Statistically significant sounds like "important", but the two answer different questions. Significance asks whether chance can explain the result; importance asks whether the effect is big enough to matter. Suppose a tutoring app is tested on two groups of 50,000 students whose scores have standard deviation 15: a gain of just 0.2 points out of 100 gives p≈0.035, significant and useless. To judge importance, look at the size of the effect and its confidence interval, not at how small p is.
Think of it like this. A smoke alarm with a sensitivity dial works like a significance test. You set the dial (α) before cooking; at 0.05, the alarm sounds on about 5% of smoke-free days. A beep means "probably not just chance", not "big fire": burnt toast and a house fire both set it off. And silence does not prove there is no smoke.
Core idea. Fix a threshold α before seeing the data and call a result significant when p<α. That rule wrongly rejects a true null only α of the time, but it says nothing about how large or important the effect is.