Integrating modern ESP32 ULP toolchain with AWS FreeRTOS

I’ve been trying to integrate the ESP32 ULP compiler with AWS FreeRTOS LTS 202012.00 and ESP-IDF v4.2. My latest attempt followed the AWS Getting Started with the DevKitC and WROVER kit guide here: Getting started with the Espressif ESP32-DevKitC and the ESP-WROVER-KIT - FreeRTOS.

To integrate the ULP toolchain I followed the instructions from the ESP-IDF documentation here: ULP Coprocessor programming - ESP32 - — ESP-IDF Programming Guide v4.2 documentation. It is written such that a component is required to access the ULP. I generated a component following the example on the AWS Getting Started guide, but I am unable to get that component to link with the rest of the build.

To keep it simple, my component contains:

  • a src directory with a c-file that loads the ULP binary
  • an include directory with an h-file containing the function prototype for that function
  • a single main.S file with the .global entry label and two simple instructions.

I strictly followed the AWS documentation for the component structure, although it is buried in a subdirectory of the project libraries directory.

When I build the project, the new include file is not visible to the rest of the project, even though COMPONENT_ADD_INCLUDEDIRS is correct. If I bypass that by using a full path to referenced h file, it still doesn’t generate an object file to link.

Because I have probably overlooked something simple, here are my changes to the CMakeFiles.txt files. This new stanza is inserted at line 18 of the main project CMakeLists.txt

message("identifying ulp component")
# add the test-ulp component
    "libraries/tester/test-ulp" ABSOLUTE

This definitely gets parsed, the message is output during build.

Here is the component CMakeFiles.txt. The message is not output during the build, a further hint that I have not included the component properly.

message("setting up test-ulp component")

set(COMPONENT_SRCS src/test-ulp.c)  #explicity list the sources
set(COMPONENT_REQURES ulp)  #must be before register component


set(ULP_S_SOURCES "ulp/main.S")
set(ULP_EXP_DEP_SRCS "../tester/src/ff_i2c2.c")
ulp_embed_binary(${ULP_APP_NAME} "${ULP_S_SOURCES}" "${ULP_EXP_DEP_SRCS}")

Lastly, in the library directory that contains our custom code, I required the new component in the CMakeLists.txt. (Perhaps this is wrong, our source code is a library, not a component)

#include the test-ulp component

Is there an example of successfully integrating the two toolchains and AWS FreeRTOS somewhere? Seeing a successfully structured project would be very useful, much more concrete than trying to correlate two independent howto guides.

Hi Ian,

I am taking a look at this and will post a reply tomorrow.



Moving to @Espressif category.

Hi @icrowe,

AWS FreeRTOS LTS 202012.00 uses ESP-IDF v3.3, are you seeing errors related to this? ESP-IDF v4.2 was introduced after this release.

Are you wanting to build this component in source, or are you adding amazon-freertos itself as a component?

I spent some time today in CMake and I almost have the ULP demo from the ESP-IDF compiling. With any luck I can post a branch that outlines the steps soon.



Hi @lundinc

Thank you for looking into this for me. I should have said I am basing this on a version of the AWS later than the LTS. The id for the revision we pulled is 441e02e3, or more fully: commit 441e02e3776833b436c9d69859d28a01bd29fa84

I’m open to either approach you suggest - whichever is most straightforward and will get me moving forward soonest. My current approach adds our high level ESP32 code as a new directory in the topmost libraries directory of the amazon-freertos repository.

Having a working example would be wonderful.

Thank you again,


Hi @icrowe,

Apologies for the delay! I have posted a working example here GitHub - lundinc2/amazon-freertos at ulp_exmaple. You can see the changes I made in the commit. I hope this is a good reference for you.

I ported the demo here esp-idf/examples/system/ulp at master · espressif/esp-idf · GitHub, and it is the first thing that runs on startup.

Before compiling, I had to modify the config defaults by menuconfig and enabling components->esp32-specific->Enable Ultra Low Power(ULP) Coprocessor as well as components->esp32-specific->RTC slow memory reserved for coprocessor = 8192. You may want to fiddle with that memory value, I ended up just using the largest.

Hope this helps! Please let me know if you need further assistance.



Hi @lundinc

Thank you. I will check out the solution this afternoon.


Hi @lundinc

I cloned your repo to and tried to build it. With your example code, I get this error when the build system gets to the point where it finds the new component:

CMake Error at vendors/espressif/esp-idf/tools/cmake/component.cmake:158 (message):
does not contain a component.
Call Stack (most recent call first):
vendors/espressif/esp-idf/tools/cmake/build.cmake:176 (__component_add)
vendors/espressif/boards/esp32/CMakeLists.txt:573 (idf_build_component)
CMakeLists.txt:70 (include)

Running -DVendor=espressif -DBOARD=esp32_wrover_kit menuconfig
causes the same error, so I don’t believe the error is because I forgot to add the required ULP configuration options.


Hey Ian, can you post the full log?

I am very careful when compiling this project, as it is sensitive to the workspace and environment variables. To compile I did the following:

git clean -xdf . #Be careful with this command! It deletes ALL files not tracked in Git. Consider '-xdn' flags to see what files get deleted. 
source vendors/espressif/esp-idf/
cmake -DVENDOR=espressif -DBOARD=esp32_wrover_kit -DCOMPILER=xtensa-esp32 -S . -B build -G"Ninja" -DCMAKE_BUILD_TYPE=debug -DCMAKE_EXPORT_COMPILE_COMMANDS=1 menuconfig build

This was on a Mac, but I will spin up a Linux VM to verify the steps.

I think the problem was CMakeLists.txt vs CMakelists.txt. That got me past this hurdle, but there is another. Possibly related to your suggestions about workspace and environment, which I will try later this evening. Here is the listing, however:
Executing action: menuconfig
Running cmake in directory /home/ian/work/firefly/example_ulp1/build
Executing “cmake -G Ninja -DPYTHON_DEPS_CHECKED=1 -DESP_PLATFORM=1 -DVendor=espressif -DBOARD=esp32_wrover_kit -DCCACHE_ENABLE=0 /home/ian/work/firefly/example_ulp1”…
– The C compiler identification is GNU 9.3.0
– The CXX compiler identification is GNU 9.3.0
– Check for working C compiler: /usr/bin/cc
– Check for working C compiler: /usr/bin/cc – works
– Detecting C compiler ABI info
– Detecting C compiler ABI info - done
– Detecting C compile features
– Detecting C compile features - done
– Check for working CXX compiler: /usr/bin/c++
– Check for working CXX compiler: /usr/bin/c++ – works
– Detecting CXX compiler ABI info
– Detecting CXX compiler ABI info - done
– Detecting CXX compile features
– Detecting CXX compile features - done
– Found Git: /usr/bin/git (found version “2.25.1”)
– Submodule update
Skipping submodule ‘…/…/…/libraries/abstractions/backoff_algorithm/’
Skipping submodule ‘…/…/…/…/libraries/abstractions/pkcs11/corePKCS11/’
Skipping submodule ‘…/…/…/libraries/abstractions/pkcs11/corePKCS11/’
Skipping submodule ‘…/…/…/libraries/abstractions/pkcs11/corePKCS11/’
Skipping submodule ‘…/…/…/libraries/abstractions/pkcs11/corePKCS11/’
Skipping submodule ‘…/…/…/libraries/coreHTTP/’
Skipping submodule ‘…/…/…/libraries/coreHTTP/’
Skipping submodule ‘…/…/…/libraries/coreHTTP/’
Skipping submodule ‘…/…/…/libraries/coreJSON/’
Skipping submodule ‘…/…/…/libraries/coreJSON/’
Skipping submodule ‘…/…/…/libraries/coreJSON/’
Skipping submodule ‘…/…/…/libraries/coreMQTT/’
Skipping submodule ‘…/…/…/libraries/coreMQTT/’
Skipping submodule ‘…/…/…/libraries/coreMQTT/’
Skipping submodule ‘…/…/…/libraries/device_defender_for_aws/’
Skipping submodule ‘…/…/…/libraries/device_defender_for_aws/’
Skipping submodule ‘…/…/…/libraries/device_defender_for_aws/’
Skipping submodule ‘…/…/…/libraries/device_shadow_for_aws/’
Skipping submodule ‘…/…/…/libraries/device_shadow_for_aws/’
Skipping submodule ‘…/…/…/libraries/device_shadow_for_aws/’
Skipping submodule ‘…/…/libraries/freertos_plus/standard/freertos_plus_tcp/’
Skipping submodule ‘…/…/libraries/freertos_plus/standard/freertos_plus_tcp/’
Skipping submodule ‘…/…/libraries/freertos_plus/standard/freertos_plus_tcp/’
Skipping submodule ‘…/…/…/libraries/jobs_for_aws/’
Skipping submodule ‘…/…/…/libraries/jobs_for_aws/’
Skipping submodule ‘…/…/…/libraries/jobs_for_aws/’
WARNING: IDF_PATH environment variable is not cleared.
If CMake is generating an error, consider clearing the IDF_PATH environment
variable, and generating a clean build. This message can be ignored if
CMake was successful.
– Component directory /home/ian/work/firefly/example_ulp1/vendors/espressif/esp-idf/components/mbedtls does not contain a CMakeLists.txt file. No component will be added
– Component directory /home/ian/work/firefly/example_ulp1/vendors/espressif/esp-idf/components/unity does not contain a CMakeLists.txt file. No component will be added
– Checking Python dependencies…
Python requirements from /home/ian/work/firefly/example_ulp1/vendors/espressif/esp-idf/requirements.txt are satisfied.
Loading defaults file /home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults…
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:21 CONFIG_NIMBLE_ENABLED was replaced with CONFIG_BT_NIMBLE_ENABLED
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:22 CONFIG_NIMBLE_MAX_CONNECTIONS was replaced with CONFIG_BT_NIMBLE_MAX_CONNECTIONS
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:23 CONFIG_NIMBLE_MAX_BONDS was replaced with CONFIG_BT_NIMBLE_MAX_BONDS
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:24 CONFIG_NIMBLE_MAX_CCCDS was replaced with CONFIG_BT_NIMBLE_MAX_CCCDS
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:25 CONFIG_NIMBLE_L2CAP_COC_MAX_NUM was replaced with CONFIG_BT_NIMBLE_L2CAP_COC_MAX_NUM
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:26 CONFIG_NIMBLE_PINNED_TO_CORE was replaced with CONFIG_BT_NIMBLE_PINNED_TO_CORE
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:27 CONFIG_NIMBLE_ROLE_CENTRAL was replaced with CONFIG_BT_NIMBLE_ROLE_CENTRAL
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:28 CONFIG_NIMBLE_ROLE_PERIPHERAL was replaced with CONFIG_BT_NIMBLE_ROLE_PERIPHERAL
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:29 CONFIG_NIMBLE_ROLE_BROADCASTER was replaced with CONFIG_BT_NIMBLE_ROLE_BROADCASTER
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:30 CONFIG_NIMBLE_ROLE_OBSERVER was replaced with CONFIG_BT_NIMBLE_ROLE_OBSERVER
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:31 CONFIG_NIMBLE_NVS_PERSIST was replaced with CONFIG_BT_NIMBLE_NVS_PERSIST
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:32 CONFIG_NIMBLE_SM_LEGACY was replaced with CONFIG_BT_NIMBLE_SM_LEGACY
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:33 CONFIG_NIMBLE_SM_SC was replaced with CONFIG_BT_NIMBLE_SM_SC
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:34 CONFIG_NIMBLE_DEBUG was replaced with CONFIG_BT_NIMBLE_DEBUG
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:35 CONFIG_NIMBLE_SVC_GAP_DEVICE_NAME was replaced with CONFIG_BT_NIMBLE_SVC_GAP_DEVICE_NAME
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:36 CONFIG_NIMBLE_GAP_DEVICE_NAME_MAX_LEN was replaced with CONFIG_BT_NIMBLE_GAP_DEVICE_NAME_MAX_LEN
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:37 CONFIG_NIMBLE_ATT_PREFERRED_MTU was replaced with CONFIG_BT_NIMBLE_ATT_PREFERRED_MTU
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:38 CONFIG_NIMBLE_SVC_GAP_APPEARANCE was replaced with CONFIG_BT_NIMBLE_SVC_GAP_APPEARANCE
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:39 CONFIG_NIMBLE_ACL_BUF_COUNT was replaced with CONFIG_BT_NIMBLE_ACL_BUF_COUNT
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:40 CONFIG_NIMBLE_ACL_BUF_SIZE was replaced with CONFIG_BT_NIMBLE_ACL_BUF_SIZE
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:41 CONFIG_NIMBLE_HCI_EVT_BUF_SIZE was replaced with CONFIG_BT_NIMBLE_HCI_EVT_BUF_SIZE
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:42 CONFIG_NIMBLE_HCI_EVT_HI_BUF_COUNT was replaced with CONFIG_BT_NIMBLE_HCI_EVT_HI_BUF_COUNT
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:43 CONFIG_NIMBLE_HCI_EVT_LO_BUF_COUNT was replaced with CONFIG_BT_NIMBLE_HCI_EVT_LO_BUF_COUNT
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:44 CONFIG_NIMBLE_MESH was replaced with CONFIG_BT_NIMBLE_MESH
/tmp/confgen_tmp867d97d2:20 line was updated to CONFIG_BT_BLUEDROID_ENABLED=n
/tmp/confgen_tmp867d97d2:34 line was updated to CONFIG_BT_NIMBLE_DEBUG=n
/tmp/confgen_tmp867d97d2:44 line was updated to CONFIG_BT_NIMBLE_MESH=n
CMake Error at vendors/espressif/esp-idf/components/esp32/project_include.cmake:21 (message):
Internal error, toolchain has not been set correctly by project (or an
invalid CMakeCache.txt file has been generated somehow)
Call Stack (most recent call first):
vendors/espressif/esp-idf/tools/cmake/build.cmake:306 (include)
vendors/espressif/esp-idf/tools/cmake/build.cmake:451 (__build_process_project_includes)
vendors/espressif/boards/esp32/CMakeLists.txt:591 (idf_build_process)
CMakeLists.txt:70 (include)

-- Configuring incomplete, errors occurred!
See also "/home/ian/work/firefly/example_ulp1/build/CMakeFiles/CMakeOutput.log".
cmake failed with exit code 1


I was able to reproduce on Amazon Linux 2, and the case sensitivity you pointed out resolved the issue. I am pushing this fix now. Thanks for catching that :slight_smile:


Success - once I pulled down the last change and followed your build instructions, everything worked as expected.

Thank you very much for your help. I’m happy to call this issue closed.


I’ve made a lot of progress with this, but at least one thing is eluding me.

There is now a custom component under esp-idf: vendors/espressif/boards/components/example_ulp

I also have a top level custom library: libraries/customjob that contains our source code. In some of that source, I would like to include the ulp_example_ulp.h file generated by the build chain so that I can access the memory locations in the shared RTC memory space.

How do I change the cmaker build files to make that possible?