How should I configure FreeRTOS to support the c++ STL standard library
There is nothing to configure for FreeRTOS. Do you have any problems using it ?
It’s more a matter of configuring your C++ runtime to be compatible with multi-threading. The biggest will be making new/delete thread safe, which tends to need malloc/free thread safe. (and then you need to look at other possible issues).
If the library is “newlib” based, there are a number of articles out there about providing that appropriate hooks to add the thread safety by defining the needed interface functions. If not, then you need to ask your code provider what needs to be done.