It turns out that was not the root cause. The root cause is a bug in the MCU. From the errata:
2.2.10 Prefetch failure when branching across Flash memory banks Description
In rare cases, the code prefetch may fail upon branching and function calls across Flash memory banks, regardless of the DUAL_BANK and nSWAP_BANK option byte settings. The failing prefetch then provides an incorrect data to the CPU, which causes code execution corruption and may lead to HardFault interrupt.
Note: The following uses of the dual bank functionality remain safe:
- EEPROM emulation or other data storage in bank 2
- bank 2 used as a download or backup slot for firmware
- mirroring the code in the banks and using bank swapping upon reset
- branching and function calls across Flash memory banks, with the prefetch function deactivatedWorkaroundNone.
We will probably have to keep the prefetch turned off.
Moral of the story is to study the errata! Thanks everyone for the input!