Not sure it is related to your problem. I had somewhat similar problem on different CPU architecture and with MPU disabled and no cache. The problem was a race condition between the pending of an interrupt to interrupt controller and subsequent code flow after portYIELD() call. You may look at this aspect.
It is possible that the problem is not a cache but just an interrupt controller latency.