• calcopiritus@lemmy.world
    link
    fedilink
    arrow-up
    1
    ·
    3 hours ago

    The offset is needed because this quantization dividing by 256 uses truncation whereas by 255 uses rounding.

    So a value of 245 (for example) could be anything between 245.00000 and 245.99999. Therefore, when dequantizing, we set the value to 245.5, which is in the middle. Between 245.00000 and 245.99999.