Measurement Accuracy Calculator

Almost every specification worth quoting has at least two terms, and they behave completely differently. The first is proportional to what you measured — % of reading, % rdg, ppm of reading, sometimes % of measured value. It tracks the signal: halve the reading and that error halves with it. The second is proportional to the range, or to the size of a display step, and it does not move at all when the reading changes. Manufacturers write it as % of range, % FS, ppm of range, counts, digits, or LSD.

The headline number in a marketing table is nearly always the first term alone. Quoting it as “the accuracy” is correct only near full scale, where the second term is a small share of the budget. Everywhere else it understates the error, and it understates it worst exactly where people trust it most: on a small signal read on a comfortable, generous range. Adding the terms up by hand is not difficult, which is precisely why it usually does not get done.

Because the units differ between the two halves of the expression, neither term can be compared to the other until both are converted into the unit you are actually measuring in. That conversion is what this tool does, and the breakdown it prints is the point of it: it shows in absolute volts, amps or ohms which half of the specification is in charge of your result.

Nothing about the instrument changed between those three lines. The 0.0035 % headline is honest at 4.7 V and off by a factor of fifteen at 0.1 V, and the fix is a range change, not a better meter.

There is a single reading at which the two halves of a specification are exactly equal, and it is worth knowing because it divides the range into two regimes. Below it the fixed terms dominate and your error as a percentage of reading climbs steeply as the signal falls; above it the proportional term takes over and the percentage error settles down to roughly the headline figure. The calculator reports it for every spec you enter.

It is found by setting the two sides equal. With a % of reading term of a and a fixed part F (the range, counts and offset terms added together, already converted into measurement units), the terms cross at reading = F ÷ (a/100). For the example above that is 50 µV ÷ 0.000035 = 1.43 V on a 10 V range, so anything below about 14 % of full scale is in the regime where the range term is calling the shots.

Handheld meters land in a more uncomfortable place. A spec of ±(0.05 % + 3 counts) on a 6000-count meter puts three counts at 3 mV on the 6 V range, and 3 mV ÷ 0.0005 = 6 V — the crossover sits exactly at full scale. On that range the counts term is never the smaller of the two, no matter where in the range you read. That is not a defect; it is what a three-and-a-half to four-digit display costs, and knowing it stops you from chasing a specification the display cannot deliver.

Counts terms are the ones most often left uncomputed, because the specification never tells you what a count is worth. It cannot: a count is one step of the least significant digit, and that step changes with every range. Three pieces of information will pin it down, and any one of them is enough.

The meter's rated count is the usual one — 2000, 4000, 6000, 20 000 — and one count is then the range divided by that number. A 6000-count meter on the 60 V range steps by 60 ÷ 6000 = 10 mV. The digit count works the same way with a different convention: n½ digits means n digits running 0 to 9 plus a leading digit that can only be 0 or 1, so the display resolves the nominal range into 10ⁿ steps and can over-range to 2·10ⁿ − 1 counts. A 5½-digit meter on the 10 V range steps by 100 µV and reads up to 19.9999 V. The third option is simply to read the resolution off the display: if the last digit is worth 0.001 V, enter 0.001 and skip the arithmetic.

Enter the nominal range, not the over-range ceiling. The 10 V range is 10 V even though the display will happily show 19.9999 V, and using 19.9999 as the range would inflate both the % of range term and the count size. The calculator prints the over-range ceiling separately so the two never get confused.

The result is a limit of error. It is the band the manufacturer guarantees, and it comes with conditions attached that the specification page states and the specification line does not: a calibration interval (24 hour, 90 day and 1 year figures differ, sometimes by more than a factor of two), a temperature window around the calibration temperature with a separate coefficient outside it, a warm-up time, and frequently a minimum percentage of range below which a different row of the table applies. A number computed here inherits every one of those conditions. If the instrument is out of calibration, cold, or being used below its stated floor, the arithmetic is still right and the guarantee is gone.

It is also not, on its own, a measurement uncertainty in the sense of the GUM. To feed it into a formal budget you would treat a limit with no stated distribution as rectangular and divide by √3 to get a standard uncertainty, combine that in quadrature with your other independent contributions — reference standard, thermal emf, loading, repeatability — and then multiply by a coverage factor for the confidence level you need. This tool stops one step short of that on purpose, because the assumptions in that step belong to whoever is signing the certificate, not to a web page.

The one place the two views meet is the test uncertainty ratio. Calibration procedures commonly require the accuracy of the reference to be several times better than that of the unit under test — a 4:1 ratio is the usual threshold — and that comparison has to be made in absolute units at the actual test point, not between two headline percentages. Computing both instruments' specs at the same reading, on the ranges each one will actually be on, is the only honest way to check it.

Manual entry is the primary mode: type the terms exactly as your manual prints them and leave empty any your spec does not have.

Six DC voltage specifications are included as presets. Each one was read out of the manufacturer's own PDF, and each carries the document number, the revision date, and the conditions the figures are valid under, so a number on screen can always be traced back to a page in a manual. Only the 1-year column is included; the 24-hour, 90-day and 2-year columns, the temperature coefficients and the AC, resistance and current functions are all left to manual entry, because every additional column multiplies the transcription surface by the number of models and a preset holding the wrong row is worse than no preset at all.

Reading 4.7 V on each of those instruments, on the range each one would actually select, gives ±0.2145 mV, ±0.2145 mV, ±0.1675 mV, ±1.505 mV, ±1.375 mV and ±3.35 mV respectively — a twentyfold spread for the same measurement. Notice that the Siglent lands in the middle despite a % of reading term six times looser than the Keithley's: on the 20 V range its 0.004 % of range term contributes more than its reading term does, which is the small-value-on-a-large-range effect appearing purely because 4.7 V sits low in a 20 V range.

Handheld presets quote the resolution printed in the manual's own resolution column rather than dividing the range by a display count. The Fluke 289 manual, for instance, never prints a count figure at all; 50 000 counts can be derived from its ranges and resolutions, but a derived number is not a published one and does not belong in a preset table.