I'm thinking there's a typo in the instructions. You're showing the registry path as HKEY_CURRENT_USER\Software\Microsoft\WindowsNT\CurrentVersion\
AppCompatFlags\Layers.
I think this should be shown as
HKEY_CURRENT_USER\Software\Microsoft\Windows NT\CurrentVersion\
AppCompatFlags\Layers (note the space between Windows and NT)
It looks correct in the screen capture.
By the way, I had searched for hours for a way to set compatibility mode during installation of a program. After finding this explanation, it was a piece of cake. Thanks for taking the time to post this.