Precision Examples in Math

Start with the recap, study the fully worked examples, then use the practice problems to check your understanding of Precision.

This page combines explanation, solved examples, and follow-up practice so you can move from recognition to confident problem-solving in Math.

Concept Recap

The degree of exactness in a measurement or calculation, reflected in the number of significant digits reported.

How many decimal places matter? Measuring in inches vs. millimeters.

Read the full concept explanation →

How to Use These Examples

Read the first worked example with the solution open so the structure is clear.
Try the practice problems before revealing each solution.
Use the related concepts and background knowledge badges if you feel stuck.

What to Focus On

Core idea: Precision is the level of detail a measurement reports, shown by its decimal places or significant figures.

Common stuck point: The procedure for precision is the easy part; the trap is treating precision as accuracy. Asking "Is the question about how finely a value is measured or reported, regardless of whether it is correct?" first is what keeps a correct-looking calculation from being attached to the wrong concept.

Sense of Study hint: Ask: Is the question about how finely a value is measured or reported, regardless of whether it is correct?

Worked Examples

Example 1

medium

A student measures a pencil three times and gets

15.2

cm,

15.1

cm,

15.3

cm. Another instrument gives

15.18

cm,

15.20

cm,

15.19

cm. Which set of measurements is more precise? Which is more accurate if the true length is

15.5

cm?

Answer

Set 2 is more precise; neither set is highly accurate relative to the true length of

15.5

cm.

First step

Precision concerns spread (consistency): Set 1 ranges from

15.1

–

15.3

cm (range

= 0.2

cm). Set 2 ranges from

15.18

–

15.20

cm (range

= 0.02

cm). Set 2 is more precise.

See the full worked solution + why-it-works coaching

SetupKey insightWhy it worksCommon pitfallConnection

Unlock answer keys One Family plan — every worked solution, all subjects

Example 2

hard

Add

12.3

cm and

4.56

cm, applying the rule for precision in addition.

Example 3

medium

Add

1.23

and

4.5678

using the rule for precision in addition.

Example 4

medium

A wood block is

5.6

cm long according to a ruler that measures to the nearest mm. Write the measurement with an uncertainty range.

Example 5

medium

Three measurements:

4.2

4.18

4.205

. Identify the precision (decimal places) of each and the precision of their average if reported correctly.

Example 6

hard

Compute

(2.34 \times 4.5) + 1.245

with correct sig fig and decimal-place rules at each step.

Example 7

hard

Distinguish precision and accuracy with an example: a target shooter places

5

shots in a tight cluster

10

cm to the left of the bullseye.

Example 8

hard

A scientist measures three lengths:

12.3

12.31

12.345

m. They average and report

12.32

m. Critique whether the precision is reported correctly.

Example 9

challenge

In an experiment,

A = 1.250

m and

B = 1.245

m (each to

4

sig figs). Compute

A - B

and discuss the precision loss in subtraction of close values.

Practice Problems

Try these problems on your own first, then open the solution to compare your method.

Example 1

easy

Which measurement is more precise:

7

7.0

m, or

7.00

m? How many significant figures does each have?

Example 2

medium

Multiply

3.4

\times

2.15

cm, applying the significant figures rule for multiplication.

Example 3

easy

How many significant figures does

3.40

have?

Example 4

easy

How many significant figures does

0.0050

have?

Example 5

easy

A ruler measures to the nearest mm. Can you report

3.14159

cm?

Example 6

easy

Which measurement is more precise:

5

cm or

5.0

cm?

Example 7

easy

A scale reads

4.50

kg for a known

5.00

kg mass, every time. Is it precise?

Example 8

easy

Round

7.86

2

significant figures.

Example 9

easy

How many significant figures does

1200

have (no decimal shown)?

Example 10

easy

Express

0.0034

in scientific notation.

Example 11

medium

Add

12.11 + 1.2

and report with correct precision (decimal places).

Example 12

medium

Multiply

2.1 \times 3.2

and report with correct significant figures.

Example 13

medium

A board is measured as

1.50

m and another as

1.5

m. How do their precisions differ?

Example 14

medium

A digital thermometer reads

36.6

degrees. To what precision is this, and what is the implied uncertainty?

Example 15

medium

Convert

0.00120

to scientific notation and state its sig figs.

Example 16

medium

You measure a table as

1.2

m with a tape good to

\pm 0.1

m. Should you report the area of a

1.2

m by

0.8

m top as

0.96

^2

Example 17

medium

Round

0.04985

2

significant figures.

Example 18

challenge

Two students measure the same rod: A reports

25.0

cm, B reports

25.00

cm. If only a mm ruler was used, who has reported an honest precision?

Example 19

challenge

A rectangle's sides are

3.0

cm and

4.00

cm. What is the perimeter, reported to correct precision?

Example 20

challenge

Explain why writing

2.1 \times 3.2 = 6.7200

is misleading even though the arithmetic gives

6.72

Example 21

medium

How many significant figures does

4.001

have?

Example 22

medium

Divide

7.50

2.5

and report to correct significant figures.

Example 23

easy

How many significant figures does

0.0024

have?

Example 24

easy

How many significant figures does

508.0

have?

Example 25

easy

Which is more precise:

3.4

m or

3.40

Example 26

easy

How many sig figs does

7000

have if it is written without a decimal point (ambiguous case)?

Example 27

medium

Multiply

4.56 \times 1.4

using the significant-figures rule for multiplication.

Example 28

medium

Subtract

98.7 - 12.34

and report with correct precision.

Example 29

medium

A student computes

\pi \times r^2

with

r = 2.5

cm and reports

19.6349541\ldots

^2

. What is the appropriate result respecting the precision of

r

Example 30

medium

Round

3.14159

to (a)

3

sig figs, (b)

5

sig figs.

Example 31

hard

A square has measured side

4.2

cm. Compute its area with correct precision.

Example 32

hard

A student measures the mass of an apple as

0.15

kg using a kitchen scale and as

147.32

g using a lab balance. Which measurement is more precise?

Example 33

hard

A measurement is reported as

6.20 \pm 0.05

cm. What is the percent uncertainty?

← Back to Precision Practice Mode

Related Concepts

Decimal Representation Significant Figures

Background Knowledge

These ideas may be useful before you work through the harder examples.

decimal representation