AWS FreeRTOS / with/with IP and Microchip Harmony

andrewfrazer wrote on Monday, April 16, 2018:

Hello, I’m hoping that someone can provide me with a little bit of advise on the best way to progress.

I have been using FreeRTOS on PIC32, which is bundled into Microchips Harmony Configuraiton engine. Its worked well for me once i got through the various hoops and learning. I’d now like to use some of the AWS FreeRTOS librarys ( MQQT and OTA ). When i looked at it,the project I discovered that they essnetially had been hand built ( at least the configuration ) for a specific platform. ( in my case the PIC32 Curosity board ).

I have been able to upgrade FreeRTOS from version 9 to 10, and use it with Harmony no problme. ( i used the librarys from the AWS git repo ).

I need/want to be able to take another PIC32 and differnet hardware and configure it ( ideally using harmony ), but the MQTT librarys require FreeRTOS + IP. Harmony of course does not know about how to deal with the freeRTOS stack.

So, my choice is moving away from the Harmony IP stack ( and harmony ), which means loosing the ability to very quickly add / remove new perhials… ( and also somethign we know very well ), to be able to use the MQTT library. ( but we would possibly loose some features in the microchip IP stack )… Or I need to be able to figure out if i can modify the MQTT librarys to work with the Microchip Stack.

I hope that my post made some sence.

rtel wrote on Monday, April 16, 2018:

The MQTT implementation includes an MQTT client library suitable for use
in bare metal applications, and an MQTT agent for FreeRTOS. So you
could integrate just the library, but it will be simpler to create an
application if you could also integrate the agent.

The agent (think daemon task) manages the MQTT library on behalf of the
developer, providing the developer with a simple connect, publish, and
subscribe interface, and unburdening the developer from having to do
anything special to manage the MQTT protocol itself - for example there
is no need to periodically call the MQTT library, the agent does that
transparently in the background for you.

As you point out, the agent has a couple of dependencies. Although the
provided examples use the FreeRTOS+TCP stack, it does so through an
abstraction layer (called the ‘secure sockets’ layer, as it also
abstracts away TLS), so you could provide a port of that abstraction
layer to the Harmony TCP/IP stack if you wanted to continue to use that
stack.

So you could update your project to use the FreeRTOS+TCP stack and the
provided secure sockets abstraction layer for that stack, or you could
update your project by providing your own secure sockets abstraction for
the Harmony stack. Alternatively you could use just the MQTT library
component. Let me know which way you think is best.

andrewfrazer wrote on Monday, April 16, 2018:

Thanks Richard, that really helped me get somethings sorted out in my head. In my way of thinking the best way forward ( at least for me ), would be to use Harmony and FreeRTOS 10.x ( which works nicely ). This gives me the best of all the worlds. the ‘cost’ to me is having to port the secure sockets layer. This does seem like the best way to move forward.

rtel wrote on Monday, April 16, 2018:

If you are not using TLS and the Harmony stack has a sockets like
interface then there should be a fairly straight forward mapping to the
secure sockets interface - you can just stub or not implement the TLS part.

These are the components you will need from the Amazon FreeRTOS
libraries - hope the formatting comes out ok:

Amazon FreeRTOS
   |
   +-lib
       |
       +-mqtt
       |   +-aws_mqtt_lib.c [Required to use library and  agent]
       |   +-aws_mqtt_agent.c   [Required to use the MQTT agent]
       |
       +-include
           |
           +-private              [For internal library use only!]
           |   +-aws_doubly_linked_list.h
           |   +-aws_mqtt_agent_config_defaults.h
           |   +-aws_mqtt_buffer.h
           |   +-aws_mqtt_config_defaults.h
           |   +-aws_mqtt_lib_private.c
           |   +-aws_mqtt_config_defaults.h
           |
           +-aws_mqtt_agent.h    [Include to use the MQTT agent API]
           +-aws_mqtt_lib.h      [Include to use the MQTT library API]

You will also need configuration files that you can pull from the
existing AWS examples:
aws_mqtt_lib_config.h
aws_mqtt_agent_config.h
aws_bufferpool_config.h
aws_secure_sockets_config.h

rtel wrote on Monday, April 16, 2018:

Just formatted the snippet there so it fits better and realised I missed out that you need the buffer pool library too - which supplies buffers to the MQTT library.

andrewfrazer wrote on Tuesday, April 17, 2018:

Thanks, thats really helpful!

using TLS will be required. Data has to transit the interweb!

rtel wrote on Tuesday, April 17, 2018:

Which TLS library are you using currently?

andrewfrazer wrote on Tuesday, April 17, 2018:

Not using any Library yet. Was thinking ( perhaps badly ) that i shoudl get it standing up without TLS first?

rtel wrote on Tuesday, April 17, 2018:

The Amazon examples are using mbedTLS, so you could pick that up too.

andrewfrazer wrote on Tuesday, April 17, 2018:

Thats handy. to know. Licencing is helpful as well. Hope to have the bones of a starting project soon.

andrewfrazer wrote on Wednesday, April 18, 2018:

In your list of files, Should
aws_mqtt_lib_config.h actually be aws_mqtt_config.h ?

I can’t find a aws_mqtt_lib_config.h anywhere; the aws_mqtt_config.h made the most sense to me…

Also, aws_mqtt_lib_private.c only seems to be related to some unit tests ( which are included if AMAZON_FREERTOS_ENABLE_UNIT_TESTS is defined. By adding this file to the project it causes errors ( as it has no includes for the .h file )? Is removing the file seems the best thign to do?

rtel wrote on Wednesday, April 18, 2018:

Apologies- you are correct on both counts.

andrewfrazer wrote on Wednesday, April 18, 2018:

Progress is being made; albeit a bit slowly.

  • I’ve decided to go down this path as I to be sensible, i really need encryption ( TLS ) of the messages. there is a MQTT library ( WolfMQQT ) that is supplied with Harmony, but the moment that you use the Wolf Librarys you are into ‘interesting’ licencing territory that is largely incompatible with what I need/want to do.

Amazon uses mbedTLS, and that does not have the same license issues.

Now work starts on porting the aws_secure_sockets to use the sockets that are used in Harmony.

In order to make this an easy to use reference project, I am making modifications the the demo project that is supplied with Harmony 2.05.01 ( apps/rtos/freertos/tcpip_client_server_). Work to date is at https://github.com/mrpackethead/harmony_aws_freertos_mqtt

Please NOTE this is NOT yet working. there is still work to be done!

My project ends up at C:\microchip\harmony\v2_05_01\dev\rtos\freertos\harmony_aws_freertos_mqtt

In order to make this work a few things need to be installed;

(1) The AWS Librarys. I have installed them at
C:\microchip\harmony\v2_05_01\third_party\amazon-freertos
The easiest way to get this is to clone it from Github, which has the most recent versions;
https://github.com/aws/amazon-freertos

Note that these librarys also include FreeRTOS, but it is the AWS Version which has been modified, and it is easier to use the generic FreeRTOS10.x

(2) FreeRTOS 10.x;
Harmony 2.05.01 ships with FreeRTOS9.x Fortunately upgrading it to Version 10.x is very simple.
Download FreeRTOS from Source Forge ( https://sourceforge.net/projects/freertos/ ). Copy the Folder ‘FreeRTOS’ when its expanded out ( FreeRTOSv10.0.1\FreeRTOS ) and use it to replace the FreeRTOS Folder that was installed with Harmony ( In my case it was C:\microchip\harmony\v2_05_01\third_party\rtos\FreeRTOS )

andrewfrazer wrote on Thursday, April 19, 2018:

I have removed aws_secure_sockets.c from my project, and created harmony_secure_sockets.c