I think I found the culprit:
#define configTASK_DEFAULT_CORE_AFFINITY 0x1 // Core 0
If I comment that line everything is fine and works as expected.
It seems to be related to how the (passive) idle tasks are assigned to the cores.
If left uncommented, both IDLE tasks and core1 has nothing to do while idling thus piling up. When commented, both IDLE tasks have no affinity and I guess they run on the appropriate core 
Maybe something in the port code should be done to pin these idle task to their respective core instead of leaving them with no affinity? Would be safer IMHO.
I would not know where to perform these changes though.
Cheers!