It is generally not recommended to use SEH in Embedded applications on small RTOS. First of all, the linker will generate sections in RAM for your exception stack frames whose placement and size is hard to control; also, generally in RTOS based applications,there is no “graceful recovery,” meaning that the tasks are interacting so closely and intertwined that if one task runs into an exception, the entire system is dysfunctional.
Attempting to use SEH solely for ease of debugging is very likely subject to the Heisenberg effect, meaning that you change the environment so drastically that the runtime behavior of your release version can not reliably be reproduced in the debug version, and vice versa, your debug version will liklely introduce new problems.
I have used FreeRTOS and C++ in many applications at many customer sites in many different environments, but SEH is something every single one of them has shied away from. You can probably get it to work somehow, but the price is very high with little gain as explained above.
We use C++ exceptions with FreeRTOS. We hit a number of issues when we first started using exceptions, which were nothing to do with FreeRTOS but everything to do with compiling for embedded processors with limited flash and RAM. See Using exceptions in C++ embedded software | David Crocker's Solutions blog for my blog post about this.
The only thing I can think of which is specific to FreeRTOS (or any other RTOS) is that if any task calls functions that may throw exceptions, be sure that there is a try-catch statement that will catch those exceptions, so they they don’t try to propagate out of the top-level function executed by that task. That function (the one you pass to whichever FreeRTOS task creation function you use) should be declared extern “C” and noexcept.