TM 5-3895-374-10
APPENDIX C
.........................................................................................................................................................DATA ROUNDING
Numerical data displayed on the screen or printed at the printer is rounded according to the following rule:
Internally, the computer processes data to nine (9) digits of precision. If a value is printed or displayed with less
than 9 digit of precision, % ? the absolute value of the data display is increased by one least significant displayed
digit if the most significant undisplayed digit is 5 or greater.
For example, if the precision of the display of %mA/C was programmed to be xx.x then:
6.40 %mA/C
is displayed
6.4 %mA/C
6.45
is
6.5
6.54
is
6.5
6.55
is
6.6
In the above example a display of 6.5% would imply a value greater than 6.45 and less than 6.55. Stated
differently, a displayed value of 6.5% is a value of 6.5 +/-0.05.
page F - 106
SYSTEMS Equipment Corp. 8/31/89 Page (58)
