Juggling Mismatches in Resolution and Least Input Increment

What is the smallest value a shop’s CNC can use?

Most machining and turning centers display coordinate and offset data to three places in the Metric (millimeter) measurement system and to four places in the Imperial (inch) measurement system. In days past, the least input increment, which is the smallest value that can be entered, was the same as the CNC’s resolution, which is the smallest value a CNC can use.

By the way, I have often suggested working in the Metric system for this very reason. Metric mode provides a better resolution than the Imperial measurement system. A measurement of 0.001 mm is less than half of 0.0001 inch. Actually, 0.001 mm is 0.000039 inch, 39 millionths of an inch. Having this better resolution will allow easier sizing and allow for longer periods between the sizing adjustments necessary for holding small tolerances.

With most current model CNCs, however, the number of decimal places the CNC displays does not necessarily determine the least input increment, nor is it necessarily equal to the CNC’s resolution. In most cases — especially for milling and turning applications — the CNC’s resolution will be much better than the least input increment the CNC can display. This means that, just because the CNC can only display data out to three or four places (again, commonly the case with mill and lathe work), it does not follow that the CNC cannot accept and use data in smaller increments.

Consider the Increment Scale

I should first point out that there are other increment systems available. With FANUC CNCs, for example, the range runs from rather coarse to extremely precise. Consider the information below that shows the various increment systems as well as the Metric and Imperial system least input increments. These are the smallest increments the CNC will display. Note that optional machine functions must be present for the more precise increment systems.

• Increment system A: 0.01 mm, 0.001 inch
• Increment system B: 0.001 mm, 0.0001 inch (used by most mills and lathes)
• Increment system C: 0.0001 mm, 0.00001 inch (used by grinding and jig-boring machines)
• Increment system D: 0.00001 mm, 0.000001 inch
• Increment system E: 0.000001 mm, 0.0000001 inch

Testing Resolution and Smallest Input Increment Mismatches

Even with a machining or turning center that displays three places in the Metric mode and four places in the Imperial mode, it’s likely that the CNC will accept and use values that are smaller than 0.001 mm or 0.0001 inch. The following test — which I approached in Imperial mode, but would apply to the Metric mode as well — can confirm this assertion:

Make the CNC display the offset page and bring the cursor to an unused offset register. Set its value to zero. For FANUC CNCs, this means typing 0 (zero) and then pressing the INPUT button. The offset register should display the value 0.0000 at this point.

Now enter an incremental adjustment of 0.00005 (which is 50 millionths, half of 0.0001 inch). For FANUC CNCs, this is done by typing 0.00005 and pressing the +INPUT button.

Nothing should change in the offset register (yet). It should still show a value of 0.0000 at the present time. But this does not mean nothing has happened. The CNC use a standard rounding process to display the value, and a value of 0.00005 will be rounded to 0.0000. But just because the value entered (0.00005) cannot be displayed properly, it does not mean the CNC is not going to use it.

Continue the test by entering another incremental adjustment of 0.00001 (10 millionths of an inch). For FANUC CNCs, this is done by typing 0.00001 and pressing the +INPUT button. Now the register should change to 0.0001. Though the CNC has rounded the value being displayed — it will use the current total of the value you have entered (0.00006 inch, 60 millionths).

We have been using offset registers to make our resolution-related point, but keep in mind that the same applies to programmed coordinates. Again, just because the CNC cannot display programmed coordinates under 0.0001 inch (or 0.001 mm) does not mean it cannot accept and use them.

As has been stated, all of this is because the CNC’s resolution is better than the least input increment it can display (in the CNC’s chosen increment system). This can have some important implications for CNC usage when small tolerances are involved.

How to Manage Small Tolerances

Consider a diameter dimension and tolerance of 1.5012/1.5005 inches. The mean value of this dimension/tolerance is 1.00085 inches. If adhering to the four-place format, it would be necessary to round the programmed coordinate to 1.5008 inches, which is not truly in the middle of the tolerance band.

If instead, the true mean value is specified in the program (1.50085), then even though the CNC cannot display it, the CNC will use that programmed value when making the motion command. Thus, sizing adjustments made using values out to five places will allow more precise sizing control and longer periods between necessary sizing adjustments.