Not sure which lesson to do first? Find your level and get a plan →

Math · Numbers & Quantities · Grade 9-12

Precision

Precision is the degree of exactness in a measurement, reflected in how many significant digits you report — $3.


Quick Answer

Precision is the degree of exactness in a measurement, reflected in how many significant digits you report — 3.140 is more precise than 3.1. Use it when you record a measurement, compare two instruments, or decide how many digits a calculated result deserves. The tool sets the limit: a ruler marked in millimetres supports 12.7 cm, not 12.7384 cm. Precision says nothing about whether the value is correct; closeness to the true value is accuracy.

Why This Matters

Every digit in a lab report or a specification is a claim about how finely something was measured. Understanding precision lets you report 12.7 cm from a millimetre ruler instead of copying 12.73846 cm off a calculator, and lets you see that a scale reading 4.0000 kg can still be wrong. Without it, extra digits pass for extra certainty and real errors hide behind decimals.

Intuitive Explanation

Hold a ruler marked in millimetres against a pencil. The end falls between the 12.7 cm and 12.8 cm marks, much nearer 12.7. You can honestly write 12.7 cm. You cannot write 12.7384 cm, because the ruler has no marks that fine. Precision is this resolution: the size of the smallest step a measurement can tell apart. A digital caliper that reads 12.74 cm is more precise, because its steps are 0.01 cm instead of 0.1 cm.

Written digits carry the same message. 12.7 cm says the true length lies within about 0.05 cm of 12.7, roughly 12.65 to 12.75. 12.70 cm says within about 0.005 cm. The trailing zero is a claim that the hundredths place was measured and came out 0, so you write it only if your tool really resolved hundredths.

Precision is easy to confuse with accuracy. Accuracy is how close a reading is to the true value; precision is how finely it is reported, and in repeated measurements how tightly the readings agree. A scale that shows 4.0000 kg every time for a bag whose true mass is 3.8 kg is highly precise and inaccurate: the digits are sharp and the value is off by 0.2 kg. To tell which a question is about, see whether it mentions a true or accepted value. If it does, the question is about accuracy; if it only compares digits or instruments, it is about precision.

Think of it like this. Precision is the zoom level of a map. A world map can show which country a city is in; a street map shows which house. Zooming in gives finer detail, but if the map was drawn wrong, the sharp view is still wrong: zoom is precision, correctness is accuracy.

Core idea. Precision is the level of detail a measurement reports, shown by its decimal places or significant figures.


When to Use

Precision is the issue when a problem asks how many digits or decimal places to report, which of two instruments or readings is finer, or what the smallest unit of a tool is. It also decides how far to round a result calculated from measurements. If the problem gives a true or accepted value and asks how close a reading is, that is accuracy, measured by percent error. If it asks you to count meaningful digits by fixed rules, use Significant Figures.

Tip. Ask: Is the question about how finely a value is measured or reported, regardless of whether it is correct?

How to Recognize It

Before you talk about precision, look at the smallest step each instrument or reported value can show.

  1. Is a true or accepted value given, such as a 5.00 g reference mass, so you can say how far a reading is from it?

    Yes points to accuracy, judged with percent error. Precision needs no true value at all; it looks only at how finely readings are recorded or how tightly repeated readings cluster.

  2. Are you comparing the last decimal place that two readings show, such as 12.7 cm against 12.68 cm?

    Yes, that is precision: 12.68 cm is recorded to the hundredth, so it is the finer reading, even if it later turns out to be the less correct one.

  3. Was the same object measured several times, with the question asking how close the readings are to each other?

    Then precision is about spread: readings of 4.81, 4.82 and 4.81 cm are precise because they agree. Whether they agree with the real length is again a question of accuracy.

  4. Does the problem ask how many digits a calculated result may keep, such as an area found from two measured sides?

    That choice rests on the precision of the inputs, but the counting rules belong to significant figures, so switch to those rules to finish.

Precision compared with Accuracy, Rounding, and Significant figures

Precision, accuracy, rounding and significant figures all talk about the digits of a measurement, so they are easy to swap. Ask whether you are judging how fine a reading is, how close it is to the truth, cutting digits off, or counting them.

Precision

Meaning
Use this when you are reporting or comparing the fineness of a measurement and how many digits are justified.
Key test
Is the question about how finely a value is measured or reported, regardless of whether it is correct?
Example
A length is reported as 12.7 cm by one student and 12.70 cm by another. Which is more precise?

Accuracy

Meaning
How CLOSE a measurement is to the true value, not how finely it is recorded.
Key test
Use when asking whether the reading is right, not how detailed it is.
Example
A clock 5 min fast is precise to the minute but inaccurate

Rounding

Meaning
The act of cutting a number to a chosen place, which sets the precision reported.
Key test
Use when performing the cut, not describing the resulting fineness.
Example
3.14159≈3.14

Significant figures

Meaning
The count of meaningful digits — the specific measure OF precision.
Key test
Use when you need to state the precision as a number of digits.
Example
0.00420 has 3 sig figs

Formula & Notation

How to read it: Precision is indicated by the number of decimal places or significant figures; e.g., 3.140 (4 sig figs) is more precise than 3.1 (2 sig figs)

Worked Examples

Example 1 — Which is more precise?

Easy

A length is reported as 12.7 cm by one student and 12.70 cm by another. Which is more precise?
Show the steps and the answer
  1. Find the last reported place of each: 12.7 stops at tenths, 12.70 goes on to hundredths.

    The last written digit shows the smallest step the measurer claims to have resolved.

  2. Turn each into a range: 12.7 means about 12.65 to 12.75 cm; 12.70 means about 12.695 to 12.705 cm.

    A narrower range of possible true lengths is exactly what 'more precise' means.

  3. Count significant figures as a check: 12.7 has 3 and 12.70 has 4.

    A trailing zero after the decimal point counts, because it records a measured hundredths digit.

Answer: 12.70 cm is more precise

Takeaway: Equal values can differ in precision: the trailing zero in 12.70 says the hundredths place was actually measured.

Example 2 — Precise but wrong

Standard

A digital scale reports 4.0000 kg every time, but the true mass is 3.8 kg. Is it precise? Accurate?
Show the steps and the answer
  1. Judge precision from the readings: 4.0000 kg is reported to 0.0001 kg, and it repeats exactly every time.

    Fine resolution and readings that agree with each other are both marks of high precision.

  2. Judge accuracy against the true value: 4.0000−3.8=0.2 kg too high.

    Accuracy can only be judged against a true or accepted value, and this problem supplies one.

  3. Find the percent error: 0.23.8×100%≈5.3%.

    The 0.2 kg error is 2000 times the scale's smallest step of 0.0001 kg, so the digits promise far more than the scale delivers.

Answer: Precise but not accurate

Takeaway: Precision and accuracy are separate checks: read the digits for one, compare with the true value for the other.

Example 3 — Find the mistake: too many digits in an average

Application

A ruler is marked in whole millimetres. Three classmates measure a leaf as 6.4 cm, 6.3 cm and 6.4 cm. Ana averages them and writes: "The leaf is 6.366666 cm long." Find the mistake and correct it.
Show the steps and the answer
  1. Check the arithmetic: (6.4+6.3+6.4)÷3=19.1÷3=6.3666…

    The calculation is fine; the problem is how many digits Ana kept.

  2. Find the precision of the measurements: millimetre marks resolve 0.1 cm, and every reading stops at tenths.

    No digit past the tenths was ever measured, so the hundredths and beyond in 6.366666 come from the calculator, not the ruler.

  3. Round the mean to the tenths place: 6.3666…≈6.4 cm.

    Three readings to the nearest millimetre cannot support a claim to a millionth of a centimetre.

Answer: About 6.4 cm, reported to tenths like the measurements.

Takeaway: A calculated result is only as precise as the measurements behind it; round it back to the place the tool actually measured.

Where You'll See It

  • Kitchen scale vs. pharmacy balance

    A kitchen scale reads flour to the nearest gram, which is fine for bread. A pharmacy balance reads to 0.001 g, because a 5 mg dose of medicine is 0.005 g and would show up as 0 g on the kitchen scale.

  • Timing a 100 m race

    Hand-held stopwatch times are usually reported to tenths of a second, because a timer's thumb is not reliable beyond that. Electronic timing reports hundredths. Two runners hand-timed at 10.8 s might be 10.76 s and 10.84 s electronically, so the finer timing separates them.

  • Engineering drawings for machine parts

    On many engineering drawings, the number of decimal places sets the allowed error: 25.0 mm may mean within 0.1 mm, while 25.00 mm means within 0.01 mm. The extra zero makes the part harder and more expensive to make, so engineers write it only where the fit needs it.

Common Mistakes

  1. The slip: Treating precision as accuracy

    Instead: precise means finely recorded, accurate means close to true; a value can be one without the other.

  2. The slip: Reporting more digits than the tool measures

    Instead: never write 3.140 from a ruler that only reads to 3.1.

  3. The slip: Adding trailing zeros to look more precise

    Instead: extra zeros claim measurement detail the instrument did not provide.


Mini Practice

Try these on your own. Tap Reveal when you want to check.

  1. A ruler is marked in millimetres. Which reading of a pencil's length fits this ruler: 14 cm, 14.3 cm or 14.32 cm?

    Hint: A millimetre is one tenth of a centimetre. How many decimal places can you read?

  2. How many significant figures does 0.00450 have?

    Hint: Leading zeros only place the decimal point. Trailing zeros after a decimal point do count.

  3. Add 12.5 m + 3.27 m and report the sum with the right precision.

    Hint: A sum can be no more precise than the least precise measurement, counted in decimal places.

  4. A rectangle measures 4.2 cm by 3.15 cm. Find its area with the right number of significant figures.

    Hint: For a product, keep as many significant figures as the measurement with the fewest.

  5. Three weighings of a sample give 5.21 g, 5.22 g and 5.21 g. The true mass is 5.50 g. Are the readings precise, accurate, both or neither?

    Hint: Precise asks whether the readings agree with each other. Accurate asks whether they are close to the true value.

  6. A kitchen scale reads to the nearest gram. Ana records a bag of flour as 250.00 g. Correct her record.

    Hint: What do the two zeros after the decimal point claim about the scale?

Want the full set?

50 practice questions for this concept — free to try, every one with a complete worked solution showing the why, not just the answer.

Frequently Asked Questions

Why does 2.50 m say more than 2.5 m if the two numbers are equal?

As numbers they are equal, but as measurements they make different claims. Writing 2.5 m says the length lies somewhere from 2.45 m to 2.55 m. Writing 2.50 m says it lies from 2.495 m to 2.505 m, a range ten times narrower. The final zero is a promise that you actually read the hundredths place and found 0 there. Drop it and you throw away information; add it without measuring and you claim information you never had.

Can averaging many readings give a more precise result than a single reading?

Yes, within limits. Random scatter, such as a stopwatch pressed a little early one time and a little late the next, partly cancels when you average. The uncertainty of the mean shrinks roughly like 1n\frac{1}{\sqrt{n}}, so the average of 100 readings is about 10 times tighter than one reading. Averaging cannot remove a bias, though: if the stopwatch itself runs fast, every reading is off the same way, and the average is precisely wrong.

Is a digital display with more digits always more precise?

Not necessarily. The digits a display shows are its resolution, the smallest step it can print. If the last digit jumps between 3 and 8 each time you weigh the same coin, the readings do not repeat to that place, and the real precision is coarser than the screen suggests. A cheap kitchen scale that shows 0.01 g may only repeat to 0.1 g. Judge precision by how well repeated readings agree, not by how many digits light up.

Where does precision matter outside a science lab?

Machined parts are a clear case. A car engine part might be specified as 86.00 mm wide with a tolerance of ±0.01\pm0.01 mm. A common workshop guideline is to measure with a tool about ten times finer than the tolerance, so the machinist uses a micrometer that reads to 0.001 mm. A tape measure that reads to 1 mm could not even tell a good part from a bad one. Pharmacies and watchmakers face the same question every day.

Learning Path

Precision

You are here

You need Decimal Representation (what each place after the point is worth) first. Precision adds the idea that each written digit is a claim about measurement, limited by the tool's smallest step and separate from accuracy. Next, Significant Figures gives exact rules for counting meaningful digits and for rounding results calculated from measured values.

See Also