Davidson Web Conversion Utility Specification

• The conversion calculation is done using high-precision SQL Server queries on the conversion factor table. 

• The SQL Server conversion engine uses full 53 bit floating point data variables to store and process the mantissa of each value ensuring a high level of mathematical accuracy.

• The conversion factors are from the NIST Special Publication 811 1995 Edition - Guide for the Use of International System of Units (SI). Physics Laboratory, National Institute of Standard & Technology.

• The behaviour of float and real follows the IEEE 754 specification on approximate numeric data types and conforms to SQL-92 standard for double precision float(53) numbers.

• The IEEE 754 specification provides four rounding modes: round to nearest, round up, round down, and round toward zero. Microsoft® SQL Server™ uses round up. All are accurate to guaranteed precision but can result in slightly different floating-point values. Because the binary representation of a floating-point number may use one of many legal rounding schemes, it is impossible to reliably quantify a floating-point value.

Disclaimer

• Whilst we have taken every effort to ensure accuracy & reliability of this conversion utility, the utility is provided by Davidson Measurement "as is" and without warranty of any kind, expressed or implied, including (but not limited to) any implied warranties of merchantability, fitness for any particular purpose, or non-infringement. While the information provided is believed to be accurate, it may include errors or inaccuracies.