Quantcast
Channel: User Schwern - Stack Overflow
Viewing all articles
Browse latest Browse all 581

Answer by Schwern for MySQL convert column data type from DECIMAL(10,5) to DECIMAL(10,2) shows different SUM outcomes

$
0
0

round(11.11499, 2) will produce 11.11000 losing 0.00999 or about 0.01. round(11.11500, 2) will produce 11.12000 gaining 0.01.

If your data is truly well-distributed and the extra decimals are truly insignificant the losses and gains due to rounding will even out. For example, we would expect temperature readings or lat/long coordinates to be well-distributed at this scale.

However, if the last three decimal places are not evenly distributed, the bias will add up as you add more numbers. With your data there is a rounding delta of 0.04032. Examples of data which might not be evenly distributed are any human-generated data, data with small sample sizes (35 is small), and data with a very small range. For example, data ranging from 12.30000 and 12.4000 is probably going to have big biases. Instrument flaws and other factors in collecting the data can introduce their own bias.

What this is telling you is two decimal places may be dropping significant data, or that there is a problem with your data.

You can avoid this build up of errors by rounding as late as possible. In your example if you store the data as decimal(10,5) and only round after summing, round(sum(net_total), 2), you get your desired 277.57.

Note that decimal(10,5) takes 6 bytes (3 bytes for the 5 integer digits and 3 bytes for the 5 decimal digits) and decimal(10,2) takes 5 bytes (4 bytes for 8 integer digits, 1 byte for 2 decimal digits), so you're not saving much on storage by changing the column type. See InnoDB Numeric Type Storage Requirements.

Alternatives include choosing a higher precision, decimal(10,3) only loses you 0.00232 but it's still 6 bytes, or you can look for the source of the bias in your data and eliminate it.


Viewing all articles
Browse latest Browse all 581

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>