FreeRTOS_select starvation

Kernel
1

hannes23 wrote on Friday, June 01, 2018:

Hello,
at my project I’ve setup a TCP server based on the one found in FreeRTOS-Plus.
The hardware is a Zynq-7000 variant, and the FreeRTOS Version is FreeRTOSv10.0.0.

The TCP server calls a HTTP worker. The server has the FreeRTOS_select() function to wait for incoming TCP packets. This function is called with a blocking time of 100 ms.
The job of the HTTP-worker is to receive a HTTP url followed by sending an amount of 10 MByte data.
This is done continuosly.
I’ve changed the HTTP-worker according to my needs in that, that the prvSendFile() routine gives to a semaphore and returns . The semaphore is then taken by a separate task (named prvHTTPClientTask), which does the sending.
Each new client gets a new prvHTTPClientTask.
So HTTP clients on th PC can work better in in parallel.
But the problem, which I describe here, also arises with a single client.

The whole job is running quiet well for a long time, (some gigabyte sent), if the task priority of the TCP-server ( 6 ) is lower than the priority of the prvHTTPClientTask ( 7 ).
If I change the priority of the TCP-server to equal or higher piority, the job will starve after a short time.
Maybe 2 - 20 of the 10-MByte sendings.
What I observe is that the FreeRTOS_select() function never returns, although a blocking time of 100 ms is given.

Can anybody give a hint here?
Thanks and greetings.

2

heinbali01 wrote on Friday, June 01, 2018:

Hi Johannes,

it is difficult to tell what goes wrong in your application without seeing it. But I’ll give it a try.

So HTTP clients on the PC can work better in in parallel.

Every task runs on the same CPU (core), so I do not think that your solution will be more efficient.

You are using FreeRTOS_select(). When you start up a new task to handle a connection, do you remove the socket from the select mask?

A problem with select() in general is that it sometimes returns immediately without blocking.

Suppose you have the TX bit enable for some socket, either you must send data, or you must disable the TX bit.
Could it be that the “starvation” is caused by select() constantly returning without ever blocking?

Please report your findings here!

3

hannes23 wrote on Monday, June 04, 2018:

Hi Hein,
thank you for your comments.

The mechanism to handle TCP connections and recv/transmit is the same as shown in the file ‘FreeRTOS_TCP_server.c’, which I took as a template.
The procedure is, that
a) FreeRTOS_select() waits for TCP-events, then if the socket is already active
b) runs through FreeRTOS_accept(), followed by
b1) prvReceiveNewClient(), when it’s a new connection or
b2) handles the rx or tx in a worker-routine.

In the original files FreeRTOS_TCP_server and FreeRTOS_HTTP_server.c work so, that upon each received data (URL) a file is sent back to the client. And all clients are handled one by one. So if a clients awaits a big amount of data, then the next clients have to wait .
To avoid this I spawn a new task for each client, which does the probably long-standing sending.

I do not remove a socket from the select-list, but only, when the connection is closed. Just like in the original file FreeRTOS_TCP_server.

And yes, like in the original FreeRTOS_HTTP_server.c file I set the tx-bit, when the sending routine returns, because no tx-space is available. The send will then be continued after the next successful return from FreeRTOS_select().

The question is simply, how can it be, that FreeRTOS_select() blocks forever, although a blocking time is given in the call.

The other question is how should I generally setup a solution for my TCP / HTTP communication needs.
I will have 1 to 4 clients requesting data exchange with possibly big amount of data, 10 MByte may be. I took as template the mentioned files, but think about to use a solution, where one task is responsible for connection to clients and spawning a worker-task for each connection. This would avoid for each data to go through FreeRTOS_accept() each time and wouldn’t use the select() routine.
Which situation has an advantage to use select()?

Any comments welcome and thank you in advance.

4

hannes23 wrote on Tuesday, June 05, 2018:

Is there an error in FreeRTOS_select() ?

Since many days I’m fighting with a problem that under certain conditions
FreeRTOS_select() not returns, although the blocking-time is max. 100 ms.

The probability that this happens is high, when the task wich calls FreeRTOS_select()
has a priority that is equal or higher than other tasks which send or receive TCP packets.
Is the priority lower, then the server has to transfer many packets until the fault happens.

To find out, where the task finally stops in the faulty situation, I entered a debug variable with different values at certain points of the called functions.

Thereby I found, that FreeRTOS_select() calls prvFindSelectedSocket() which then calls
xEventGroupWaitBits(), with portMAX_DELAY as timeout, and then in this function the debug variable was finally set before calling xTaskResumeAll().

Is this probably a bug? :frowning:
Comments and hints are greatly welcome.

5

hannes23 wrote on Tuesday, June 05, 2018:

Is there an error in FreeRTOS_select() ?

Since many days I’m fighting with a problem that under certain conditions
FreeRTOS_select() not returns, although the blocking-time is max. 100 ms.

The probability that this happens is high, when the task wich calls FreeRTOS_select()
has a priority that is equal or higher than other tasks which send or receive TCP packets.
Is the priority lower, then the server has to transfer many packets until the fault happens.

To find out, where the task finally stops in the faulty situation, I entered a debug variable with different values at certain points of the called functions.

Thereby I found, that FreeRTOS_select() calls prvFindSelectedSocket() which then calls
xEventGroupWaitBits(), with portMAX_DELAY as timeout, and then in this function the debug variable was finally set before calling xTaskResumeAll().

Is this probably a bug? :frowning:
Comments and hints are greatly welcome.

6

rtel wrote on Tuesday, June 05, 2018:

Can you please try with ipconfigTCP_HANG_PROTECTION set to 1 in FreeRTOSIPConfig.h - thanks.

7

hannes23 wrote on Wednesday, June 06, 2018:

Hello Richard,
the hang-protection was already active, but the function FreeRTOS_select() didn’t return.
What I indeed find strange is that this routine can be given a max. blocking-time, one of the inside called functions however is called with portMAX_DELAY. When this should be overruled by the hang-protection mechanism, I think it didn’t work.
What I meanwhile found is that the problem seems to vanish when the task-priority of the task, which calls FreeRTOS_select(), (TCP-Server) , is set to tskIDLE_PRIORITY.
Thanks and Greetings.

8

heinbali01 wrote on Wednesday, June 06, 2018:

Hi Johannes, FreeRTOS_select() will wait for an event group. Event groups are used all over the library, also in the API’s.

the problem seems to vanish when the task-priority of the task
which calls FreeRTOS_select(), (TCP-Server) , is set to tskIDLE_PRIORITY

That makes me wonder if your diagnosis is correct: lowering the priority of a task calling a blocking function would only lower the chances of returning ( with or without a time-out ). And that will happen if the CPU is too busy with running tasks of a higher priority.

Please make sure that the idle task gets plenty of running time, i.e. make sure that there is no task that is running constantly running or runnable ( unless you give that task the lowest possible priority ).

Also please make sure you have understood my texts here above about using [FreeRTOS_]select(). If you use the TX mask, make sure that you either send data, or you disable the TX mask.

FreeRTOS_select() and xEventGroupWaitBits()() have already been used extensively in many projects and I’m not aware of problems. It is hard to guess what goes wrong in your project without seeing the source code. Best,

9

hannes23 wrote on Thursday, June 07, 2018:

Hi Hein,
of course I’m not sure, whether I understood everything right and also for sure I cannot code error-free :slight_smile: But I try. And for a better understanding I’ve attached the hopeful necessary files.

Some aspects of this problem I have meanwhile understood a bit. In alteration of the original files
‘FreeRTOS_TCP_server.c’ and ‘FreeRTOS_HTTP_server.c’ I start an extra client task after a TCP connection is made, which should be responsible for the sending.
And here the task priority gets important.
If the TCP-server which calls select() has a higher priority, than the extra sending task isn’t active as
should be necessary. When the communication stucks, the select function in this case returns the event-bits. The tx-bit is set, but due to priority-scheme the sending task will not be running and the whole thing is stuck.
If the TCP-server has the same priority, then the select() function, when the communication is stuck, doesn’t return as mentioned above.
But if the priority of the TCP-server is lower, zero in my case, the communication is running well.

So the only question which remains is, why the select() function doesn’t return if the priorities of the TCP-server and the client-sending-task are the same.
Thanks and greetings.

10

heinbali01 wrote on Thursday, June 07, 2018:

HI Johannes, would you mind to send me an email and we’ll try to get through this by email.
My email address is hein [at] htibosch [dot] net

11

heinbali01 wrote on Thursday, June 07, 2018:

[ sorry, I pressed twice on the Post button ]