Clean Starter project for FreeRTOS and ESP32

Partners & Sponsors AWS
1

TJirsch wrote on March 01, 2019:

Hi,

I am quite used to FreeRTOS dev with ESP32 and try to make the switch to AWS FreeRTOS.
It is great work and frees one from implementing all the connection stuff, mqtt and so on manually.

What I cannot really figure out is how to create a new clean project, that is located outside the FreeRTOS folder.

I like to separate my files from those cloned from github.

I have seen the hint to rename the demo folder, this and also restructuring the folders under the demo folder does not work. As soon as you have to change the component.mk in common, the build starts to fail, because files in the lib directory (first is app_trace, many more follow) rely on that file.

The only coice would be to change every component.mk file in the lib folders.
This work would be wasted on the next update from github.

Could you provide a starter project that works outside of the FreeRTOS Folder or at least not in the demo folder ?

Just beeing curious: Has anyone succeded to create a ESP32 Project other than within the demo folder ?

Edited by: TJirsch on Mar 1, 2019 2:33 PM

2

Tiangang-AWS wrote on March 02, 2019:

Hi,

Thanks for your valuable feedback. We are aware of this issue, and are working on solutions to make it easier for customers to integrate Amazon FreeRTOS to their projects in general. We are also evaluating CMake build system support for ESP32 right now, which might be helpful in your situation.

Meanwhile, do you mind sharing a little bit more detail in your use case, like what kind of structure you’d like to use, which folder you renamed, the list of steps you need to do and exact location of files you need to change? We’re happy to assist you and would like to know more of your experience.

Thanks

3

TJirsch wrote on March 02, 2019:

Hi, thank you for sharing the information.

as to your questions:

I have existing projects for customers that use the original ESP-IDF Framework and its proposed directory structure and build system (https://docs.espressif.com/projects/esp-idf/en/latest/api-guides/build-system.html#component-makefiles).

This basicly means: Put ESP-IDF in one place and my code in a another place with a structure mostly at my disposure and reference the ESP-IDF by setting a per project variable.
That is fine with me as I can also upgrade to new ESP-IDF versions on a per project basis.
I have integrated the embedded AWS IoT SDK into those project in a similar fashion.

The main point is separation of my project code from the library code.

I would like to switch to “native” AWS FreeRTOS and I have tried to do this with two approaches:

  1. Start from zero in a directory of my own
  2. Work from a running demo project out of the demo folder, as I cannot deliver source code to customers that is located in a demo folder and mixed with library code.

Both strategies fail sooner or later because the references set in the project Makefile and project.mk files for components are relative (…/…/) and propagated to the build process of library components. The folder depth of the project structure for the project is on the same level as components in the lib folder and the build seems to depend on that.
If I move my project folder from demos/espressif/esp32_devkitc_esp_wrover_kit/common/application_code I have to change build vars (COMPONENT_ADD_INCLUDEDIRS, EXTRA_COMPONENT_DIRS). This breaks the build, because they are relative and propagated to the build of library components in freeRTOS/lib etc.

With both strategies 1. and 2., this is where I end up editing individual component.mk files in the lib folder to make them work independently of the project vars. I have gotten really far, but all the work will be useless as soon as a new version comes out, possibly for new reasons, so this is not the way to go. I’d have gone for some small changes, but I realized I was rebuilding the build system towards the original ESP-IDF version.

Also there is library code in demos/espressif/esp32_devkitc_esp_wrover_kit/common/application_code/espressif_code that is application independent and would belong into the freeRTOS/lib folder. This code is also broken for the same reason as the folder move.
Also what I do not want, is to put my code into a folder that represents my special board (demos/espressif/esp32_devkitc_esp_wrover_kit/common/application_code/), that structure is ok for a lib, but not for a project.
Furthermore I using the original FreeRTOS also to connect to GCP (sorry), so I would like to be able to place projects in a folder structure reflecting that, but thats optional.

It might be possible to build a separate library first and then link my code with that, but I will first try to keep the structure depth and just rename the demo folder.
If none of that does not work, I will have to stick to the original version.

I think that lots of people try out the demos and then get stuck on building their own apps as I do. I would be really happy if I missed something, as i would really like to get ota updates and BLE.

4

Tiangang-AWS wrote on March 05, 2019:

Hi,

Thanks for sharing your experience, it is very valuable to us.

Looks like the root cause is that in espressif Makefile, Amazon FreeRTOS is added as extra components in demos/espressif/esp32_devkitc_esp_wrover_kit/common/application_code/espressif_code/amazon-freertos-common/component.mk, and we use relative path there which makes it difficult to copy those Makefiles and set up your own project outside of Amazon FreeRTOS source tree. Basically these files have to be updated and might need merging if new Amazon FreeRTOS version comes out,

demos/espressif/esp32_devkitc_esp_wrover_kit/make/Makefile
demos/espressif/esp32_devkitc_esp_wrover_kit/common/application_code/espressif_code/amazon-freertos-common/component.mk
demos/espressif/esp32_devkitc_esp_wrover_kit/common/application_code/espressif_code/freertos/component.mk
demos/espressif/esp32_devkitc_esp_wrover_kit/common/application_code/espressif_code/mbedtls/component.mk

There shoudn’t be any change in lib/third_party/mcu_vendor/espressif/esp-idf, let me know if I missed anything thing above. We will take these feedback into consideration when working on solutions to make it easier for customers to integrate Amazon FreeRTOS into their project.

Meanwhile, are you interested in trying out the CMake approach? It is currently available on a branch in our repo, [https://github.com/aws/amazon-freertos/tree/feature/cmake] (README is under cmake subfolder). Assuming you have this following directory structure:

  • my_project
    • amazon-freertos
    • my_app
    • CMakeLists.txt

You should be able to add amazon-freertos as a subdirectory in you CMakeLists.txt and configure your project with the following command:

cmake -DAFR_BOARD=espressif.esp32_devkitc_esp_wrover_kit -DCMAKE_TOOLCHAIN_FILE=amazon-freertos/cmake/toolchains/xtensa-esp32.cmake -S. -Bbuild

Thanks.

5

TJirsch wrote on March 07, 2019:

That sounds great. I will try the CMake approach.

Regarding the makefiles you mentioned, I think you got them all, the only change needed (If you keep the directory levels) is to change demo to app in the makefiles, place the code in the application_code folder and cleanup the rest.
I am done integrating the demo code into an app that does ota and shadow, I will look at the gg integration and then try to move the app to the CMake approach.

This would have to be done again if a new version comes out, but should be easy to achieve.
As Espressif is switching to CMake anyway I will try that next.

I really appreciate the work involved integrating so many platforms, frameworks and parties.
The OTA works like a charm and much better than the examples provided by espressif.

Again, thanks for the help.

6

Tiangang-AWS wrote on March 11, 2019:

Hi,

I just gave it a quick try, when I run make flash, it actually automatically detects the port for me. I can, however, confirm the make menuconfig does not show the serial port configuration. I’ll reach out to espressif for this issue. Meanwhile, you can also try setting the environment variable ESPPORT to your desired port. Let me know if this helps.

Thanks

7

VikT wrote on March 11, 2019:

thanks.
make flash ESPPORT=/dev/cu.SLAB_USBtoUART works!
make monitor, gives the error: make: *** No rule to make target `monitor’.

Also, how would sdkconfig play a role in this, if any?

8

Tiangang-AWS wrote on March 11, 2019:

You can use the pyserial package directly (it is used by make monitor under the hood):
python -m serial.tools.miniterm /dev/cu.SLAB_USBtoUART 115200

sdkconfig is automatically generated by CMake and put under the root directory of Amazon FreeRTOS. sdkconfig.default in demos/espressif/esp32_devkitc_esp_wrover_kit/make is still being used.

Thanks

9

VikT wrote on March 11, 2019:

Hi,
I have tried the instructions and I was able to compile the code. Unfortunately, not able to flash now. Can you suggest the steps on how to flash the code to esp32. make menuconfig is not providing the right options.

10

MatteoElimo wrote on April 14, 2019:

Hello,

the CMake solution works, thank you. It would be great if the documentation had an example of a clean project though, it can save a lot of time to new users e.g. me =)

Edited by: MatteoElimo on Apr 14, 2019 9:05 AM

11

VikT wrote on May 12, 2019:

Hi, still need more help with this task. I am able to compile and flash the demo code when I run through the amazon-freertos directory, but still not able to compile myapp at the same directory level of amazon-freertos.

My CMakeLists.txt contains:
add_subdirectory("./amazon-freertos")
project(iot-pilot)

Please assist. This has been an ordeal to separate amazon freertos from the supplied demo code.

12

TJirsch wrote on May 13, 2019:

I am quite happy with CMake and using a subdirectory under freeRTOS, but using a separate dir on the same level as the freeRTOS tree really would help keeping things clean.
This also goes for the demos itself: In my +opinion+ they should be moved out of the freeRTOS folder.
I tried to create a CMakeList.txt for the side by side solution but ended up rewriting the one that comes with freeRTOS dir.

13

VikT wrote on May 13, 2019:

Can you please share the directory structure that worked for you and if possible the modified CMakeLists.txt with your project structure. It will help a lot.

14

TJirsch wrote on May 15, 2019:

Hi,

sorry to come back so late, but I am really stuck with the build process and using freertos.
I have switched to the cmake branch and after some effort I succeeded again in "faking"my app as a demo module in the demos dir.
This needed analysing and editing the CMake files that came with freRTOS and this process will have to be done again if a new release gets out, possibly breaking the whole thing.
How am I to explain this to my boss ?

I am no CMake or build system expert and do not want to become one!

What I am asking myself is:
Why is it made so hard to build an app outside (and to great extent inside) the freeRTOS source tree ?
Am I doing something wrong ? Missing something ?
Most to nobody using freeRTOS wants to build freeRTOS and is demos for ist own sake.
Nearly everybody wants to build applications and that is made nearly impossible.

Why am I expected to investigate the internal works of the build process of something I would like to use as a library ? Am I ?
I have spent days on that now and ended up with a solution that has to be repeated and might break every time a new release comes out.

With the “old” or native FreeRTOS/IDF you have to create two directories, create an project.mk file, there was some kind of documentation for variables etc. and that’s it!

If it is so easy as you proposed in the last answer, could you please, please, please post a CMakeList.txt, a very simple main.c file, that does nothing much except linking to some freeRTOS and MQTT code and WORKS ?

I think we all can work on from there.

Edited by: TJirsch on May 15, 2019 10:46 AM

15

VikT wrote on May 16, 2019:

Don’t be so harsh on yourself!
You gave your best to solve a tough problem. Documentation and help programs on Amazon FreeRTOS are very restrictive and I am debating myself, if it makes sense to continue this path or go back to simple FreeRTOS.
In either way, appreciate your help. I sincerely hope that the build team would look into this soon if they want a mass adoption of this product. Thanks again. I will review your code on the github.

16

MatteoElimo wrote on June 25, 2019:

201906.00 brings CMake to most vendor/boards, if I understand correctly.

It still does not provide a clear way to build an application that’s not “masqueraded” as a demo, as far as I can see.

17

MatteoElimo wrote on June 25, 2019:

As far as I can see, the recent release of 201906.00 has made most of the information in this thread obsolete most of the solutions in this thread: the advice of having amazon-freertos as a subdirectory in a root CMakeLists.txt does not seem to work.
Also, the [https://github.com/aws/amazon-freertos/tree/feature/cmake] CMake feature branch does not exist anymore.

There is an issue on github about this: [https://github.com/aws/amazon-freertos/issues/832]

I suggest anybody interested contribute to that issue in order to give it relevance.

18

TJirsch wrote on June 25, 2019:

Thats cool.
I managed to get rid of the demos and was able to use FreeRTOS from a separate folder.
It required to make some changes to three CMake files within the FreeRTOS dir and obviously you had to do it again on every release.
Why the demos are not separated from the main os as singular applications is sth. i will probably never never get.
I will see if the changes still work.

19

TJirsch wrote on June 25, 2019:

Hi, thank you for providing the makefiles.
I have tried that approach and did run into a dead end, partly because of the lack of CMake expertise at that time, partly because some files seemed to be magically interwoven with the demos, etc.

I have just now completed to adapt the changes I made in a different approach to the new releases files:
FreeRTOSRoot/vendors/espressif/boards/esp32/CMakeLists.txt
and
FreeRTOSRoot/tools/cmake/afr.cmake.

I need to adjust my apps CMakeList.txt (The old demos CMaklist.txt) to some new files (networkmanager etc.) and will post the files then.

Basicly I have tweaked the Demo macros and paths to being my apps, on the way removing some of the horrible …/…/…/…/…/ ladders, i.e. AFR_ROOT/demos to AFR_ROOT/…/app.
That takes about 15 mins and maybe 30 lines; if one would change the new path using a variable, after the adjustment you could swap in every single app or demo you like by setting a CMake var.
That would be ok for me, i did not bother using a var because i have to possibly change the CMakefiles with every new FreeRTOS release, also my apps go in a static structure so I am fine with that static situation:

/project
/project/app
/project/freeRTOS

Ill try to fix my apps CMakefile tomorrow and post it.

Also don’t get me wrong: I am only using the ESP32, not one of the other boards.
I really appreciate the work involved to create a build system of that complexity.

regards, Thomas

Edited by: TJirsch on Jun 25, 2019 1:22 PM

20

TJirsch wrote on June 25, 2019:

Just a quick question: Has the CMake approach been given up or has hit been integrated into the current release ?
I have spent a considerable amount of time to make the “subdir approach” work for my projects and espressif/idf is moving to cmake anyway.