• calcopiritus@lemmy.world
      link
      fedilink
      arrow-up
      1
      ·
      edit-2
      7 hours ago

      Did you read the article or just the title?

      I’m honestly surprised this comment is the one that shows at the top.

      • pHr34kY@lemmy.world
        link
        fedilink
        arrow-up
        2
        ·
        edit-2
        6 hours ago

        So, it’s basically saying that anything less than 1/256 (0.00390625f) is 0x0.

        Anything equal to or over 0.99609375 (255/256) should be 0xff.

        Divide by the count (256), not the max index (255). I’m not sure the 0.5 offset is really needed.

        • calcopiritus@lemmy.world
          link
          fedilink
          arrow-up
          1
          ·
          3 hours ago

          The offset is needed because this quantization dividing by 256 uses truncation whereas by 255 uses rounding.

          So a value of 245 (for example) could be anything between 245.00000 and 245.99999. Therefore, when dequantizing, we set the value to 245.5, which is in the middle. Between 245.00000 and 245.99999.