TM 5-3895-374-10APPENDIXC.........................................................................................................................................................DATA ROUNDINGNumerical data displayed on the screen or printed at the printer is rounded according to the following rule:Internally, the computer processes data to nine (9) digits of precision. If a value is printed or displayed with lessthan 9 digit of precision, % ? the absolute value of the data display is increased by one least significant displayeddigit if the most significant undisplayed digit is 5 or greater.For example, if the precision of the display of %mA/C was programmed to be xx.x then:6.40 %mA/Cis displayed6.4 %mA/C6.45is6.56.54is6.56.55is6.6In the above example a display of 6.5% would imply a value greater than 6.45 and less than 6.55. Stateddifferently, a displayed value of 6.5% is a value of 6.5 +/-0.05.page F - 106SYSTEMS Equipment Corp. 8/31/89 Page (58)
Integrated Publishing, Inc. - A (SDVOSB) Service Disabled Veteran Owned Small Business
