Hi Jakob,
The SVC exception should have highest priority, zero. That setting is found in register SHPR2, at address 0xE000ED1C, in the SCB. The most-significant byte of that 32-bit register sets the priority for SVC. What value is in that register in your system?