Quite funny, I debugged this issue for about 3 days now and read a lot as well.
Right after posting my question I ran into this:
I tried the -fno-strict-aliasing option and after ~50 reboots the problem didn’t show up anymore. So I believe this helps, still not sure if it is the right thing to change.