First question is did you define configASSERT and enabled stack checking ?
See FreeRTOS - stacks and stack overflow checking
It seems that there is an internal data corruption almost always caused by a stack overflow.
If you do not delete the task to be notified while having the interrupt enabled, of course 
Could you also show the code of the task creation and the task code ?
Which MCU is it and which FreeRTOS version do you use ?
See also xTaskNotifyFromISR() Real Time Operating System API documentation for the correct usage of pxHigherPriorityTaskWoken resp. lu32_taskWoken.