• pHr34kY@lemmy.world
    link
    fedilink
    arrow-up
    2
    ·
    edit-2
    6 hours ago

    So, it’s basically saying that anything less than 1/256 (0.00390625f) is 0x0.

    Anything equal to or over 0.99609375 (255/256) should be 0xff.

    Divide by the count (256), not the max index (255). I’m not sure the 0.5 offset is really needed.

    • calcopiritus@lemmy.world
      link
      fedilink
      arrow-up
      1
      ·
      3 hours ago

      The offset is needed because this quantization dividing by 256 uses truncation whereas by 255 uses rounding.

      So a value of 245 (for example) could be anything between 245.00000 and 245.99999. Therefore, when dequantizing, we set the value to 245.5, which is in the middle. Between 245.00000 and 245.99999.