Integrating modern ESP32 ULP toolchain with AWS FreeRTOS

I’ve been trying to integrate the ESP32 ULP compiler with AWS FreeRTOS LTS 202012.00 and ESP-IDF v4.2. My latest attempt followed the AWS Getting Started with the DevKitC and WROVER kit guide here: Getting started with the Espressif ESP32-DevKitC and the ESP-WROVER-KIT - FreeRTOS.

To integrate the ULP toolchain I followed the instructions from the ESP-IDF documentation here: ULP Coprocessor programming - ESP32 - — ESP-IDF Programming Guide v4.2 documentation. It is written such that a component is required to access the ULP. I generated a component following the example on the AWS Getting Started guide, but I am unable to get that component to link with the rest of the build.

To keep it simple, my component contains:

  • a src directory with a c-file that loads the ULP binary
  • an include directory with an h-file containing the function prototype for that function
  • a single main.S file with the .global entry label and two simple instructions.

I strictly followed the AWS documentation for the component structure, although it is buried in a subdirectory of the project libraries directory.

When I build the project, the new include file is not visible to the rest of the project, even though COMPONENT_ADD_INCLUDEDIRS is correct. If I bypass that by using a full path to referenced h file, it still doesn’t generate an object file to link.

Because I have probably overlooked something simple, here are my changes to the CMakeFiles.txt files. This new stanza is inserted at line 18 of the main project CMakeLists.txt

message("identifying ulp component")
# add the test-ulp component
    "libraries/tester/test-ulp" ABSOLUTE

This definitely gets parsed, the message is output during build.

Here is the component CMakeFiles.txt. The message is not output during the build, a further hint that I have not included the component properly.

message("setting up test-ulp component")

set(COMPONENT_SRCS src/test-ulp.c)  #explicity list the sources
set(COMPONENT_REQURES ulp)  #must be before register component


set(ULP_S_SOURCES "ulp/main.S")
set(ULP_EXP_DEP_SRCS "../tester/src/ff_i2c2.c")
ulp_embed_binary(${ULP_APP_NAME} "${ULP_S_SOURCES}" "${ULP_EXP_DEP_SRCS}")

Lastly, in the library directory that contains our custom code, I required the new component in the CMakeLists.txt. (Perhaps this is wrong, our source code is a library, not a component)

#include the test-ulp component

Is there an example of successfully integrating the two toolchains and AWS FreeRTOS somewhere? Seeing a successfully structured project would be very useful, much more concrete than trying to correlate two independent howto guides.

Hi Ian,

I am taking a look at this and will post a reply tomorrow.



Moving to @Espressif category.

Hi @icrowe,

AWS FreeRTOS LTS 202012.00 uses ESP-IDF v3.3, are you seeing errors related to this? ESP-IDF v4.2 was introduced after this release.

Are you wanting to build this component in source, or are you adding amazon-freertos itself as a component?

I spent some time today in CMake and I almost have the ULP demo from the ESP-IDF compiling. With any luck I can post a branch that outlines the steps soon.



Hi @lundinc

Thank you for looking into this for me. I should have said I am basing this on a version of the AWS later than the LTS. The id for the revision we pulled is 441e02e3, or more fully: commit 441e02e3776833b436c9d69859d28a01bd29fa84

I’m open to either approach you suggest - whichever is most straightforward and will get me moving forward soonest. My current approach adds our high level ESP32 code as a new directory in the topmost libraries directory of the amazon-freertos repository.

Having a working example would be wonderful.

Thank you again,


Hi @icrowe,

Apologies for the delay! I have posted a working example here GitHub - lundinc2/amazon-freertos at ulp_exmaple. You can see the changes I made in the commit. I hope this is a good reference for you.

I ported the demo here esp-idf/examples/system/ulp at master · espressif/esp-idf · GitHub, and it is the first thing that runs on startup.

Before compiling, I had to modify the config defaults by menuconfig and enabling components->esp32-specific->Enable Ultra Low Power(ULP) Coprocessor as well as components->esp32-specific->RTC slow memory reserved for coprocessor = 8192. You may want to fiddle with that memory value, I ended up just using the largest.

Hope this helps! Please let me know if you need further assistance.



Hi @lundinc

Thank you. I will check out the solution this afternoon.


Hi @lundinc

I cloned your repo to and tried to build it. With your example code, I get this error when the build system gets to the point where it finds the new component:

CMake Error at vendors/espressif/esp-idf/tools/cmake/component.cmake:158 (message):
does not contain a component.
Call Stack (most recent call first):
vendors/espressif/esp-idf/tools/cmake/build.cmake:176 (__component_add)
vendors/espressif/boards/esp32/CMakeLists.txt:573 (idf_build_component)
CMakeLists.txt:70 (include)

Running -DVendor=espressif -DBOARD=esp32_wrover_kit menuconfig
causes the same error, so I don’t believe the error is because I forgot to add the required ULP configuration options.


Hey Ian, can you post the full log?

I am very careful when compiling this project, as it is sensitive to the workspace and environment variables. To compile I did the following:

git clean -xdf . #Be careful with this command! It deletes ALL files not tracked in Git. Consider '-xdn' flags to see what files get deleted. 
source vendors/espressif/esp-idf/
cmake -DVENDOR=espressif -DBOARD=esp32_wrover_kit -DCOMPILER=xtensa-esp32 -S . -B build -G"Ninja" -DCMAKE_BUILD_TYPE=debug -DCMAKE_EXPORT_COMPILE_COMMANDS=1 menuconfig build

This was on a Mac, but I will spin up a Linux VM to verify the steps.

I think the problem was CMakeLists.txt vs CMakelists.txt. That got me past this hurdle, but there is another. Possibly related to your suggestions about workspace and environment, which I will try later this evening. Here is the listing, however:
Executing action: menuconfig
Running cmake in directory /home/ian/work/firefly/example_ulp1/build
Executing “cmake -G Ninja -DPYTHON_DEPS_CHECKED=1 -DESP_PLATFORM=1 -DVendor=espressif -DBOARD=esp32_wrover_kit -DCCACHE_ENABLE=0 /home/ian/work/firefly/example_ulp1”…
– The C compiler identification is GNU 9.3.0
– The CXX compiler identification is GNU 9.3.0
– Check for working C compiler: /usr/bin/cc
– Check for working C compiler: /usr/bin/cc – works
– Detecting C compiler ABI info
– Detecting C compiler ABI info - done
– Detecting C compile features
– Detecting C compile features - done
– Check for working CXX compiler: /usr/bin/c++
– Check for working CXX compiler: /usr/bin/c++ – works
– Detecting CXX compiler ABI info
– Detecting CXX compiler ABI info - done
– Detecting CXX compile features
– Detecting CXX compile features - done
– Found Git: /usr/bin/git (found version “2.25.1”)
– Submodule update
Skipping submodule ‘…/…/…/libraries/abstractions/backoff_algorithm/’
Skipping submodule ‘…/…/…/…/libraries/abstractions/pkcs11/corePKCS11/’
Skipping submodule ‘…/…/…/libraries/abstractions/pkcs11/corePKCS11/’
Skipping submodule ‘…/…/…/libraries/abstractions/pkcs11/corePKCS11/’
Skipping submodule ‘…/…/…/libraries/abstractions/pkcs11/corePKCS11/’
Skipping submodule ‘…/…/…/libraries/coreHTTP/’
Skipping submodule ‘…/…/…/libraries/coreHTTP/’
Skipping submodule ‘…/…/…/libraries/coreHTTP/’
Skipping submodule ‘…/…/…/libraries/coreJSON/’
Skipping submodule ‘…/…/…/libraries/coreJSON/’
Skipping submodule ‘…/…/…/libraries/coreJSON/’
Skipping submodule ‘…/…/…/libraries/coreMQTT/’
Skipping submodule ‘…/…/…/libraries/coreMQTT/’
Skipping submodule ‘…/…/…/libraries/coreMQTT/’
Skipping submodule ‘…/…/…/libraries/device_defender_for_aws/’
Skipping submodule ‘…/…/…/libraries/device_defender_for_aws/’
Skipping submodule ‘…/…/…/libraries/device_defender_for_aws/’
Skipping submodule ‘…/…/…/libraries/device_shadow_for_aws/’
Skipping submodule ‘…/…/…/libraries/device_shadow_for_aws/’
Skipping submodule ‘…/…/…/libraries/device_shadow_for_aws/’
Skipping submodule ‘…/…/libraries/freertos_plus/standard/freertos_plus_tcp/’
Skipping submodule ‘…/…/libraries/freertos_plus/standard/freertos_plus_tcp/’
Skipping submodule ‘…/…/libraries/freertos_plus/standard/freertos_plus_tcp/’
Skipping submodule ‘…/…/…/libraries/jobs_for_aws/’
Skipping submodule ‘…/…/…/libraries/jobs_for_aws/’
Skipping submodule ‘…/…/…/libraries/jobs_for_aws/’
WARNING: IDF_PATH environment variable is not cleared.
If CMake is generating an error, consider clearing the IDF_PATH environment
variable, and generating a clean build. This message can be ignored if
CMake was successful.
– Component directory /home/ian/work/firefly/example_ulp1/vendors/espressif/esp-idf/components/mbedtls does not contain a CMakeLists.txt file. No component will be added
– Component directory /home/ian/work/firefly/example_ulp1/vendors/espressif/esp-idf/components/unity does not contain a CMakeLists.txt file. No component will be added
– Checking Python dependencies…
Python requirements from /home/ian/work/firefly/example_ulp1/vendors/espressif/esp-idf/requirements.txt are satisfied.
Loading defaults file /home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults…
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:21 CONFIG_NIMBLE_ENABLED was replaced with CONFIG_BT_NIMBLE_ENABLED
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:22 CONFIG_NIMBLE_MAX_CONNECTIONS was replaced with CONFIG_BT_NIMBLE_MAX_CONNECTIONS
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:23 CONFIG_NIMBLE_MAX_BONDS was replaced with CONFIG_BT_NIMBLE_MAX_BONDS
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:24 CONFIG_NIMBLE_MAX_CCCDS was replaced with CONFIG_BT_NIMBLE_MAX_CCCDS
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:25 CONFIG_NIMBLE_L2CAP_COC_MAX_NUM was replaced with CONFIG_BT_NIMBLE_L2CAP_COC_MAX_NUM
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:26 CONFIG_NIMBLE_PINNED_TO_CORE was replaced with CONFIG_BT_NIMBLE_PINNED_TO_CORE
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:27 CONFIG_NIMBLE_ROLE_CENTRAL was replaced with CONFIG_BT_NIMBLE_ROLE_CENTRAL
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:28 CONFIG_NIMBLE_ROLE_PERIPHERAL was replaced with CONFIG_BT_NIMBLE_ROLE_PERIPHERAL
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:29 CONFIG_NIMBLE_ROLE_BROADCASTER was replaced with CONFIG_BT_NIMBLE_ROLE_BROADCASTER
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:30 CONFIG_NIMBLE_ROLE_OBSERVER was replaced with CONFIG_BT_NIMBLE_ROLE_OBSERVER
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:31 CONFIG_NIMBLE_NVS_PERSIST was replaced with CONFIG_BT_NIMBLE_NVS_PERSIST
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:32 CONFIG_NIMBLE_SM_LEGACY was replaced with CONFIG_BT_NIMBLE_SM_LEGACY
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:33 CONFIG_NIMBLE_SM_SC was replaced with CONFIG_BT_NIMBLE_SM_SC
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:34 CONFIG_NIMBLE_DEBUG was replaced with CONFIG_BT_NIMBLE_DEBUG
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:35 CONFIG_NIMBLE_SVC_GAP_DEVICE_NAME was replaced with CONFIG_BT_NIMBLE_SVC_GAP_DEVICE_NAME
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:36 CONFIG_NIMBLE_GAP_DEVICE_NAME_MAX_LEN was replaced with CONFIG_BT_NIMBLE_GAP_DEVICE_NAME_MAX_LEN
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:37 CONFIG_NIMBLE_ATT_PREFERRED_MTU was replaced with CONFIG_BT_NIMBLE_ATT_PREFERRED_MTU
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:38 CONFIG_NIMBLE_SVC_GAP_APPEARANCE was replaced with CONFIG_BT_NIMBLE_SVC_GAP_APPEARANCE
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:39 CONFIG_NIMBLE_ACL_BUF_COUNT was replaced with CONFIG_BT_NIMBLE_ACL_BUF_COUNT
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:40 CONFIG_NIMBLE_ACL_BUF_SIZE was replaced with CONFIG_BT_NIMBLE_ACL_BUF_SIZE
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:41 CONFIG_NIMBLE_HCI_EVT_BUF_SIZE was replaced with CONFIG_BT_NIMBLE_HCI_EVT_BUF_SIZE
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:42 CONFIG_NIMBLE_HCI_EVT_HI_BUF_COUNT was replaced with CONFIG_BT_NIMBLE_HCI_EVT_HI_BUF_COUNT
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:43 CONFIG_NIMBLE_HCI_EVT_LO_BUF_COUNT was replaced with CONFIG_BT_NIMBLE_HCI_EVT_LO_BUF_COUNT
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:44 CONFIG_NIMBLE_MESH was replaced with CONFIG_BT_NIMBLE_MESH
/tmp/confgen_tmp867d97d2:20 line was updated to CONFIG_BT_BLUEDROID_ENABLED=n
/tmp/confgen_tmp867d97d2:34 line was updated to CONFIG_BT_NIMBLE_DEBUG=n
/tmp/confgen_tmp867d97d2:44 line was updated to CONFIG_BT_NIMBLE_MESH=n
CMake Error at vendors/espressif/esp-idf/components/esp32/project_include.cmake:21 (message):
Internal error, toolchain has not been set correctly by project (or an
invalid CMakeCache.txt file has been generated somehow)
Call Stack (most recent call first):
vendors/espressif/esp-idf/tools/cmake/build.cmake:306 (include)
vendors/espressif/esp-idf/tools/cmake/build.cmake:451 (__build_process_project_includes)
vendors/espressif/boards/esp32/CMakeLists.txt:591 (idf_build_process)
CMakeLists.txt:70 (include)

-- Configuring incomplete, errors occurred!
See also "/home/ian/work/firefly/example_ulp1/build/CMakeFiles/CMakeOutput.log".
cmake failed with exit code 1


I was able to reproduce on Amazon Linux 2, and the case sensitivity you pointed out resolved the issue. I am pushing this fix now. Thanks for catching that :slight_smile:


Success - once I pulled down the last change and followed your build instructions, everything worked as expected.

Thank you very much for your help. I’m happy to call this issue closed.


I’ve made a lot of progress with this, but at least one thing is eluding me.

There is now a custom component under esp-idf: vendors/espressif/boards/components/example_ulp

I also have a top level custom library: libraries/customjob that contains our source code. In some of that source, I would like to include the ulp_example_ulp.h file generated by the build chain so that I can access the memory locations in the shared RTC memory space.

How do I change the cmaker build files to make that possible?


Hi Ian,

Adding this include path to your library’s CMake should resolve this issue amazon-freertos/CMakeLists.txt at 174363616c3ae6882519cf5a77315a1caa65169f · lundinc2/amazon-freertos · GitHub. I will do a deep dive to see if there is a cleaner approach.



Hi Carl,

That worked very well, thank you. But it brought up yet another question.

To access the ulp_example_ulp.h I added the path to my module CMakeLists.txt like this:


The first directory is my library include directory, and the second is the generated directory that contains ulp_example_ulp.h.

I think that problem is solved, but since I still haven’t managed a full build I don’t know for sure.

Now my followup question - my library needs to access the esp-idf h files, but that isn’t quite working correctly. If I include “driver/i2c.h” in the vendors/espressif/boards/esp32/aws_demos/application_code/main.c, then the code compiles properly.

However, if I include “driver/i2c.h” in a source from my library, I get a strange error:

/home/ian/.espressif/tools/xtensa-esp32-elf/esp-2020r3-8.4.0/xtensa-esp32-elf/bin/xtensa-esp32-elf-gcc -DAFR_ESP_LWIP -DESP_PLATFORM -DIDF_VER=“v4.2-12-g3388b409c” -D_GNU_SOURCE -I…/libraries/example/include -Iesp-idf/example_ulp/ulp_example_ulp … … …
In file included from …/vendors/espressif/esp-idf/components/driver/include/driver/i2c.h:31,
from …/libraries/example/src/ff_i2c2.c:2:
…/vendors/espressif/esp-idf/components/driver/include/driver/gpio.h:22:10: fatal error: soc/gpio_periph.h: No such file or directory
#include “soc/gpio_periph.h”

(I removed most of the gcc command line, it was very long with all the includes and options)

I dug a bit into the gcc command line in the two different parts of the source tree. From the application_code directory, there are about 60 more -I directives. There must be some magic CMake to get these include paths in my library as well. I could add them manually, but I suspect there is a more intelligent method.


Hi Ian,

You will need to add the following include path for that error "${esp_idf_dir}/components/soc/include". I will take a look and see if there is a better way then using the include path.

Hi Carl,

It is a bit more involved than that. I had to add two things to my CMakeLists.txt. First "${esp_idf_dir}/components/soc/include" to the afr_module_include_dirs section, and second AFR::common_io to the afr_module_dependencies section.

Neither alone fixed my problem. Without AFR::common_io, I get the error with soc/gpio_periph.h. Without the ${esp-idf_dir}..., I see error: unknown type name 'i2c_port_t'; did you mean 'in_port_t'?

Baby steps…


Hi Carl,

I’m still having problems integrating the ULP code with my library. Two more have been exposed. I want my code to be able to access the variables shared between the ESP32 xtensa and ULP. As you suggested, in my libraries/example/CMakeLists.txt I added the following to the afr_module_include_dirs directive:


Using a message() instruction, I know that that ${CMAKE_BINARY_DIR} correctly points to my build subdirectory at the time that the CMakeLists.txt is parsed.

When I try to build my project ( build), the file I need isn’t included. Two things are happening. The gcc command line when it tries to build an object file from my library source has multiple -I directories listed. The one added by the directive above shows up as


The ${CMAKE_BINARY_DIR} portion of the path is either missing, or blank when the actual command was generated. Obviously, this path doesn’t exist relative to the libraries directory where the compile is happening.

I’ve noticed the include path is parsed by the afr tools differently when it is absolute vs relative too. If it is relative I can artificially manufacture a path that does point to the correct directory. If it is absolute, I can’t. So to move forward, I changed the path from the ${CMAKE_BINARY_DIR}/esp-idf/... above to

With this (hacky) change, the include path is passed to gcc correctly and exposes the second problem. This library is being compiled before the ULP assembler tool chain has run. So the directory where the generated ulp_example_ulp.h file should be exists, but is empty. The build fails because

../libraries/example/src/ff_boot.c:7:10: fatal error: ulp_example_ulp.h: No such file or directory
 #include <ulp_example_ulp.h> // for the ulp_variables

How can I make my custom library be dependent on the successful compile of the ulp component?


Hi Ian,

Sorry for the difficulties you are seeing here. What I did was make the kernel component depend on the ulp_example component.

My thought is that doing the same for your library component should ensure that the ulp_example_ulp.h is generated beforehand.

Let me know if that is a step in the right direction,



Not sure that it is the right track for me. If you are talking about the change in freertos/CMakeLists.txt.

That is good for recognizing the esp32 components, but not a library from projectdir/libraries (similar to freertos_plus). Or it is, and I just don’t have the cmake chops to make it happen.

Do you think I am following the wrong path by building our code outside the esp32 part of the tree? Maybe I can refactor my code to be included in the ULP component.