TM 5-3895-374-10
APPENDIX B
............................................................................................................................................... DATA RANGE and PRECISION
All computations and variables stored within the computer are done in floating point. Floating point numbers have a
precision of nine (9) digits and may be in the range of 10-36 to 10+36.Even though numbers are often displayed on the
screen or printed at the printer with less precision, the number is maintained in the computer with full 9 digit precision. For
example:
actual value
floating point value
-.0935674281
-9.35674281X 10-2
0.935674281
9.35674281X 10-1
9.35674281
9.35674281
93.5674281
9.35674281X 101
-935.674281
-9.35674281X 102
page F - 105
SYSTEMS Equipment Corp. 8/31/89 Page (57)
