Some good news here.
I contacted Shelly support and learned that if physical button connected to the switch is turned on/off 5 times the dimmer resets itself. This can be disabled in settings in "Factory Reset" section. I think I accidently was triggering this magic sequence while testing the dimmer.
I also noticed the dimmer did not run the latest FW even though it claimed it did when I connected directly to it's AP. Since the dimmer was not connected to the Internet it obviously could not really know whether it ran the latest version or not.
With FW updated and the reset via switch option disabled in the settings I could not reproduce the problem. I did try hard.