The firmware might be hijacking the fn key. If you use one of the keycode-reporting tools, does it report the fn key? and what is fn+enter or fn+backspace being reported as? if you’re not seeing the fn key, or if say fn+enter is reported as a singular key instead of two keys, maybe you could bind that “key” to enter?
If the Mac settings panel has an option to lock the F-keys to either F1-F12 or the keybinds, that option may be stored in the firmware, so for those keys simply reversing it in firmware may be enough.
Seems like the fn key is intercepted by the firmware then, that sucks. I’m repeating the same question differently just to ensure we’re on the same page:
If I understood correctly, your fn key is constantly “active”, making any key with an fn-modifier act as a different key, and that is why forexample the enter-button won’t be… well, enter? but if it enter acts as a different key, i.e. pressing enter gives a totally different keycode, can’t you rebind whatever keycode that turns into, into enter?
I did a super surface-level quick search, it seems like the older macbooks have had some firmware reverse engineering done. Might be some options there. Also, your issue may be a common case of a specific board trace needing new solder, which might not be too hard. Maybe that guy who does apple repairs/right to repair has something on this, or a community who may know. Louis Rossman something? can’t recall.
Also, you may’ve mentioned in your original post, but are there other keys that could work? doesn’t mac have that extra “option” key? i think i read that ctrl+fn key could be switched on a firmware level, if that means the ctrl key is then intercepted by the firmware I guess it won’t add much, but if all else fails, could be worth checking if it acts differently.
good luck, I don’t have any more ideas but I’m rooting for you, would be interesting to hear if you find a solution.