Integrating modern ESP32 ULP toolchain with AWS FreeRTOS

I’ve been trying to integrate the ESP32 ULP compiler with AWS FreeRTOS LTS 202012.00 and ESP-IDF v4.2. My latest attempt followed the AWS Getting Started with the DevKitC and WROVER kit guide here: Getting started with the Espressif ESP32-DevKitC and the ESP-WROVER-KIT - FreeRTOS.

To integrate the ULP toolchain I followed the instructions from the ESP-IDF documentation here: ULP Coprocessor programming - ESP32 - — ESP-IDF Programming Guide v4.2 documentation. It is written such that a component is required to access the ULP. I generated a component following the example on the AWS Getting Started guide, but I am unable to get that component to link with the rest of the build.

To keep it simple, my component contains:

  • a src directory with a c-file that loads the ULP binary
  • an include directory with an h-file containing the function prototype for that function
  • a single main.S file with the .global entry label and two simple instructions.

I strictly followed the AWS documentation for the component structure, although it is buried in a subdirectory of the project libraries directory.

When I build the project, the new include file is not visible to the rest of the project, even though COMPONENT_ADD_INCLUDEDIRS is correct. If I bypass that by using a full path to referenced h file, it still doesn’t generate an object file to link.

Because I have probably overlooked something simple, here are my changes to the CMakeFiles.txt files. This new stanza is inserted at line 18 of the main project CMakeLists.txt

message("identifying ulp component")
# add the test-ulp component
get_filename_component(
    EXTRA_COMPONENT_DIRS
    "libraries/tester/test-ulp" ABSOLUTE
    )
list(APPEND IDF_EXTRA_COMPONENT_DIRS ${EXTRA_COMPONENT_DIRS})

This definitely gets parsed, the message is output during build.

Here is the component CMakeFiles.txt. The message is not output during the build, a further hint that I have not included the component properly.

message("setting up test-ulp component")

set(COMPONENT_ADD_INCLUDE_DIRS "include")
set(COMPONENT_SRCS src/test-ulp.c)  #explicity list the sources
set(COMPONENT_REQURES ulp)  #must be before register component

register_component()

set(ULP_APP_NAME "ulp_${COMPONENT_TARGET}")
set(ULP_S_SOURCES "ulp/main.S")
set(ULP_EXP_DEP_SRCS "../tester/src/ff_i2c2.c")
ulp_embed_binary(${ULP_APP_NAME} "${ULP_S_SOURCES}" "${ULP_EXP_DEP_SRCS}")

Lastly, in the library directory that contains our custom code, I required the new component in the CMakeLists.txt. (Perhaps this is wrong, our source code is a library, not a component)

#include the test-ulp component
set(COMPONENT_REQUIRES test-ulp)

Is there an example of successfully integrating the two toolchains and AWS FreeRTOS somewhere? Seeing a successfully structured project would be very useful, much more concrete than trying to correlate two independent howto guides.

Hi Ian,

I am taking a look at this and will post a reply tomorrow.

Thanks,

Carl

Moving to @Espressif category.

Hi @icrowe,

AWS FreeRTOS LTS 202012.00 uses ESP-IDF v3.3, are you seeing errors related to this? ESP-IDF v4.2 was introduced after this release.

Are you wanting to build this component in source, or are you adding amazon-freertos itself as a component?

I spent some time today in CMake and I almost have the ULP demo from the ESP-IDF compiling. With any luck I can post a branch that outlines the steps soon.

Thanks,

Carl

Hi @lundinc

Thank you for looking into this for me. I should have said I am basing this on a version of the AWS later than the LTS. The id for the revision we pulled is 441e02e3, or more fully: commit 441e02e3776833b436c9d69859d28a01bd29fa84

I’m open to either approach you suggest - whichever is most straightforward and will get me moving forward soonest. My current approach adds our high level ESP32 code as a new directory in the topmost libraries directory of the amazon-freertos repository.

Having a working example would be wonderful.

Thank you again,

Ian

Hi @icrowe,

Apologies for the delay! I have posted a working example here GitHub - lundinc2/amazon-freertos at ulp_exmaple. You can see the changes I made in the commit. I hope this is a good reference for you.

I ported the demo here esp-idf/examples/system/ulp at master · espressif/esp-idf · GitHub, and it is the first thing that runs on startup.

Before compiling, I had to modify the config defaults by runningidf.py menuconfig and enabling components->esp32-specific->Enable Ultra Low Power(ULP) Coprocessor as well as components->esp32-specific->RTC slow memory reserved for coprocessor = 8192. You may want to fiddle with that memory value, I ended up just using the largest.

Hope this helps! Please let me know if you need further assistance.

Thanks,

Carl

Hi @lundinc

Thank you. I will check out the solution this afternoon.

Ian

Hi @lundinc

I cloned your repo to and tried to build it. With your example code, I get this error when the build system gets to the point where it finds the new component:

CMake Error at vendors/espressif/esp-idf/tools/cmake/component.cmake:158 (message):
Directory
‘/home/ian/work/firefly/example_ulp1/vendors/espressif/boards/components/example_ulp’
does not contain a component.
Call Stack (most recent call first):
vendors/espressif/esp-idf/tools/cmake/build.cmake:176 (__component_add)
vendors/espressif/boards/esp32/CMakeLists.txt:573 (idf_build_component)
CMakeLists.txt:70 (include)

Running idf.py -DVendor=espressif -DBOARD=esp32_wrover_kit menuconfig
causes the same error, so I don’t believe the error is because I forgot to add the required ULP configuration options.

Ian

Hey Ian, can you post the full log?

I am very careful when compiling this project, as it is sensitive to the workspace and environment variables. To compile I did the following:

git clean -xdf . #Be careful with this command! It deletes ALL files not tracked in Git. Consider '-xdn' flags to see what files get deleted. 
source vendors/espressif/esp-idf/export.sh
cmake -DVENDOR=espressif -DBOARD=esp32_wrover_kit -DCOMPILER=xtensa-esp32 -S . -B build -G"Ninja" -DCMAKE_BUILD_TYPE=debug -DCMAKE_EXPORT_COMPILE_COMMANDS=1
idf.py menuconfig
idf.py build

This was on a Mac, but I will spin up a Linux VM to verify the steps.

I think the problem was CMakeLists.txt vs CMakelists.txt. That got me past this hurdle, but there is another. Possibly related to your suggestions about workspace and environment, which I will try later this evening. Here is the listing, however:
Executing action: menuconfig
Running cmake in directory /home/ian/work/firefly/example_ulp1/build
Executing “cmake -G Ninja -DPYTHON_DEPS_CHECKED=1 -DESP_PLATFORM=1 -DVendor=espressif -DBOARD=esp32_wrover_kit -DCCACHE_ENABLE=0 /home/ian/work/firefly/example_ulp1”…
– The C compiler identification is GNU 9.3.0
– The CXX compiler identification is GNU 9.3.0
– Check for working C compiler: /usr/bin/cc
– Check for working C compiler: /usr/bin/cc – works
– Detecting C compiler ABI info
– Detecting C compiler ABI info - done
– Detecting C compile features
– Detecting C compile features - done
– Check for working CXX compiler: /usr/bin/c++
– Check for working CXX compiler: /usr/bin/c++ – works
– Detecting CXX compiler ABI info
– Detecting CXX compiler ABI info - done
– Detecting CXX compile features
– Detecting CXX compile features - done
– Found Git: /usr/bin/git (found version “2.25.1”)
– Submodule update
Skipping submodule ‘…/…/…/libraries/abstractions/backoff_algorithm/’
Skipping submodule ‘…/…/…/…/libraries/abstractions/pkcs11/corePKCS11/’
Skipping submodule ‘…/…/…/libraries/abstractions/pkcs11/corePKCS11/’
Skipping submodule ‘…/…/…/libraries/abstractions/pkcs11/corePKCS11/’
Skipping submodule ‘…/…/…/libraries/abstractions/pkcs11/corePKCS11/’
Skipping submodule ‘…/…/…/libraries/coreHTTP/’
Skipping submodule ‘…/…/…/libraries/coreHTTP/’
Skipping submodule ‘…/…/…/libraries/coreHTTP/’
Skipping submodule ‘…/…/…/libraries/coreJSON/’
Skipping submodule ‘…/…/…/libraries/coreJSON/’
Skipping submodule ‘…/…/…/libraries/coreJSON/’
Skipping submodule ‘…/…/…/libraries/coreMQTT/’
Skipping submodule ‘…/…/…/libraries/coreMQTT/’
Skipping submodule ‘…/…/…/libraries/coreMQTT/’
Skipping submodule ‘…/…/…/libraries/device_defender_for_aws/’
Skipping submodule ‘…/…/…/libraries/device_defender_for_aws/’
Skipping submodule ‘…/…/…/libraries/device_defender_for_aws/’
Skipping submodule ‘…/…/…/libraries/device_shadow_for_aws/’
Skipping submodule ‘…/…/…/libraries/device_shadow_for_aws/’
Skipping submodule ‘…/…/…/libraries/device_shadow_for_aws/’
Skipping submodule ‘…/…/libraries/freertos_plus/standard/freertos_plus_tcp/’
Skipping submodule ‘…/…/libraries/freertos_plus/standard/freertos_plus_tcp/’
Skipping submodule ‘…/…/libraries/freertos_plus/standard/freertos_plus_tcp/’
Skipping submodule ‘…/…/…/libraries/jobs_for_aws/’
Skipping submodule ‘…/…/…/libraries/jobs_for_aws/’
Skipping submodule ‘…/…/…/libraries/jobs_for_aws/’
WARNING: IDF_PATH environment variable is not cleared.
If CMake is generating an error, consider clearing the IDF_PATH environment
variable, and generating a clean build. This message can be ignored if
CMake was successful.
– Component directory /home/ian/work/firefly/example_ulp1/vendors/espressif/esp-idf/components/mbedtls does not contain a CMakeLists.txt file. No component will be added
– Component directory /home/ian/work/firefly/example_ulp1/vendors/espressif/esp-idf/components/unity does not contain a CMakeLists.txt file. No component will be added
– Checking Python dependencies…
Python requirements from /home/ian/work/firefly/example_ulp1/vendors/espressif/esp-idf/requirements.txt are satisfied.
Loading defaults file /home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults…
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:21 CONFIG_NIMBLE_ENABLED was replaced with CONFIG_BT_NIMBLE_ENABLED
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:22 CONFIG_NIMBLE_MAX_CONNECTIONS was replaced with CONFIG_BT_NIMBLE_MAX_CONNECTIONS
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:23 CONFIG_NIMBLE_MAX_BONDS was replaced with CONFIG_BT_NIMBLE_MAX_BONDS
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:24 CONFIG_NIMBLE_MAX_CCCDS was replaced with CONFIG_BT_NIMBLE_MAX_CCCDS
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:25 CONFIG_NIMBLE_L2CAP_COC_MAX_NUM was replaced with CONFIG_BT_NIMBLE_L2CAP_COC_MAX_NUM
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:26 CONFIG_NIMBLE_PINNED_TO_CORE was replaced with CONFIG_BT_NIMBLE_PINNED_TO_CORE
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:27 CONFIG_NIMBLE_ROLE_CENTRAL was replaced with CONFIG_BT_NIMBLE_ROLE_CENTRAL
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:28 CONFIG_NIMBLE_ROLE_PERIPHERAL was replaced with CONFIG_BT_NIMBLE_ROLE_PERIPHERAL
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:29 CONFIG_NIMBLE_ROLE_BROADCASTER was replaced with CONFIG_BT_NIMBLE_ROLE_BROADCASTER
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:30 CONFIG_NIMBLE_ROLE_OBSERVER was replaced with CONFIG_BT_NIMBLE_ROLE_OBSERVER
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:31 CONFIG_NIMBLE_NVS_PERSIST was replaced with CONFIG_BT_NIMBLE_NVS_PERSIST
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:32 CONFIG_NIMBLE_SM_LEGACY was replaced with CONFIG_BT_NIMBLE_SM_LEGACY
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:33 CONFIG_NIMBLE_SM_SC was replaced with CONFIG_BT_NIMBLE_SM_SC
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:34 CONFIG_NIMBLE_DEBUG was replaced with CONFIG_BT_NIMBLE_DEBUG
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:35 CONFIG_NIMBLE_SVC_GAP_DEVICE_NAME was replaced with CONFIG_BT_NIMBLE_SVC_GAP_DEVICE_NAME
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:36 CONFIG_NIMBLE_GAP_DEVICE_NAME_MAX_LEN was replaced with CONFIG_BT_NIMBLE_GAP_DEVICE_NAME_MAX_LEN
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:37 CONFIG_NIMBLE_ATT_PREFERRED_MTU was replaced with CONFIG_BT_NIMBLE_ATT_PREFERRED_MTU
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:38 CONFIG_NIMBLE_SVC_GAP_APPEARANCE was replaced with CONFIG_BT_NIMBLE_SVC_GAP_APPEARANCE
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:39 CONFIG_NIMBLE_ACL_BUF_COUNT was replaced with CONFIG_BT_NIMBLE_ACL_BUF_COUNT
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:40 CONFIG_NIMBLE_ACL_BUF_SIZE was replaced with CONFIG_BT_NIMBLE_ACL_BUF_SIZE
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:41 CONFIG_NIMBLE_HCI_EVT_BUF_SIZE was replaced with CONFIG_BT_NIMBLE_HCI_EVT_BUF_SIZE
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:42 CONFIG_NIMBLE_HCI_EVT_HI_BUF_COUNT was replaced with CONFIG_BT_NIMBLE_HCI_EVT_HI_BUF_COUNT
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:43 CONFIG_NIMBLE_HCI_EVT_LO_BUF_COUNT was replaced with CONFIG_BT_NIMBLE_HCI_EVT_LO_BUF_COUNT
/home/ian/work/firefly/example_ulp1/build/sdkconfig.defaults:44 CONFIG_NIMBLE_MESH was replaced with CONFIG_BT_NIMBLE_MESH
/tmp/confgen_tmp867d97d2:20 line was updated to CONFIG_BT_BLUEDROID_ENABLED=n
/tmp/confgen_tmp867d97d2:34 line was updated to CONFIG_BT_NIMBLE_DEBUG=n
/tmp/confgen_tmp867d97d2:44 line was updated to CONFIG_BT_NIMBLE_MESH=n
CMake Error at vendors/espressif/esp-idf/components/esp32/project_include.cmake:21 (message):
Internal error, toolchain has not been set correctly by project (or an
invalid CMakeCache.txt file has been generated somehow)
Call Stack (most recent call first):
vendors/espressif/esp-idf/tools/cmake/build.cmake:306 (include)
vendors/espressif/esp-idf/tools/cmake/build.cmake:451 (__build_process_project_includes)
vendors/espressif/boards/esp32/CMakeLists.txt:591 (idf_build_process)
CMakeLists.txt:70 (include)

-- Configuring incomplete, errors occurred!
See also "/home/ian/work/firefly/example_ulp1/build/CMakeFiles/CMakeOutput.log".
cmake failed with exit code 1

Ian

I was able to reproduce on Amazon Linux 2, and the case sensitivity you pointed out resolved the issue. I am pushing this fix now. Thanks for catching that :slight_smile:

Carl

Success - once I pulled down the last change and followed your build instructions, everything worked as expected.

Thank you very much for your help. I’m happy to call this issue closed.

Ian

I’ve made a lot of progress with this, but at least one thing is eluding me.

There is now a custom component under esp-idf: vendors/espressif/boards/components/example_ulp

I also have a top level custom library: libraries/customjob that contains our source code. In some of that source, I would like to include the ulp_example_ulp.h file generated by the build chain so that I can access the memory locations in the shared RTC memory space.

How do I change the cmaker build files to make that possible?

Ian