SO_REUSE with multiple tasks

I think I’ve run across a bit of a race condition when reusing the listening socket in FreeRTOS+TCP.

I have a server task that binds a port (with FREERTOS_SO_REUSE_LISTEN_SOCKET) and accepts a connection. This server task handles receiving data, and disconnect/reconnect logic.

A second ‘transmit’ task is continually transmitting in my application (using the reused socket).

I expect that when the socket is closed (from remote disconnect) both the FreeRTOS_recv() and FreeRTOS_send() calls will return with an error indicating the socket is closed. However, if the remote end disconnects, the socket moves to the closed state, but depending on where in prvTCPSendLoop() the transmit task currently is (and if the server or transmit task is executing), the socket may get marked back to the LISTEN state (since it is reused) before prvTCPSendCheck() is called. prvTCPSendCheck() then returns ‘1’, and therefore execution stays in prvTCPSendLoop(), instead of the expected behaviour of FreeRTOS_send returning an error due to socket closure.

Getting rid of the SO_REUSE option does fix the problem (behaviour of FreeRTOS_send returns error on closed socket as expected).

I might be able to fix this by waiting for the transmit task to complete (if it currently happens to be within FreeRTOS_send()), but this synchronization is outside the scope of what I expected would be necessary given that the send/receive are documented as thread safe (so far as only one task is sending or receiving). Maybe a note by the SO_REUSE option (or simply this post being present) will save someone else the same debugging effort.

v11.1.0 LTS

@SLocke-CA

Thanks for reporting this. We will try to reproduce this issue.

Is the server task that handles receiving data and disconnect/reconnect logic able to connect to the client again after the client is disconnected, if the client tries to connect again? If so, does the transmit task continue sending remaining data after the client is connected back?

Is it possible for you to share the tx and rx task code?

Thanks Tony. I may have spoken too soon, it seems the issue I’m having is a little deeper. Changing SO_REUSE just make it less apparent in that particular test case.

This could all boil down to what I expect ‘send’ and ‘recv’ to do on disconnect. A stripped down test case is below. Please excuse the messy code.

I expect that on a closed socket (or closing socket), send and recv will return without performing any functions until I have a chance to synchronize the two threads. You can see my method of doing that below.

Instead, both the server (recv) and transmit (send) tasks stay in their respective send/recv functions, without indication that the socket has closed. The socket is still marked as ESTABLISHED in both functions when this happens. Server task priority is 2, transmit task priority is 1, and IP task priority is 3 (network driver layer is 4).

I have a packet capture of what happens (and the resulting stack trace in a debugger), and it’s fairly easy for me to reproduce (I can’t upload the actual capture, so a screenshot will have to do for now).

struct ServerTaskParams {
	StreamBufferHandle_t rx_stream;
	volatile Socket_t socket;
	MessageBufferHandle_t adc_msg_buff;
	TaskHandle_t server_task;
	uint32_t tx_task_exit;
};

static struct ServerTaskParams server_params;


__BSS(BOARD_SDRAM) uint8_t data[125000];

void ServerTxTask(void* params){
	struct ServerTaskParams * serv_params = (struct ServerTaskParams *)params;
	BaseType_t res;
	while(1){
		res = FreeRTOS_send(serv_params->socket,data,125000,0);
		if(res != 125000) PRINTF("Here\n\n");
		if(res < 0 || serv_params->tx_task_exit){
			xTaskNotifyGive(serv_params->server_task);
			vTaskSuspend(NULL);
		}
	}
}

void ServerTask(void* params){
	Socket_t xListeningSocket;
	TickType_t xReceiveTimeOut = portMAX_DELAY;
	BaseType_t res=1;
	struct freertos_sockaddr xClient, xBindAddress;
	uint32_t xSize = sizeof(xClient);
	uint8_t data[8];
	struct ServerTaskParams * serv_params = (struct ServerTaskParams *)params;
	TaskHandle_t serv_tx_handle;

	serv_params->socket = NULL;

	xListeningSocket = FreeRTOS_socket(FREERTOS_AF_INET4,FREERTOS_SOCK_STREAM,FREERTOS_IPPROTO_TCP );
	FreeRTOS_setsockopt(xListeningSocket,0,FREERTOS_SO_RCVTIMEO,&xReceiveTimeOut,sizeof(xReceiveTimeOut));
	//res = 1;
	//FreeRTOS_setsockopt(xListeningSocket,0,FREERTOS_SO_REUSE_LISTEN_SOCKET,&res,sizeof(res));
	xBindAddress.sin_port = FreeRTOS_htons(5911);
	xBindAddress.sin_family = FREERTOS_AF_INET4;
	FreeRTOS_bind(xListeningSocket,&xBindAddress,sizeof(xBindAddress));
	FreeRTOS_listen(xListeningSocket,1);

	while(1){
		serv_params->socket = FreeRTOS_accept(xListeningSocket,&xClient, &xSize);
		data[0] = 'U';
		FreeRTOS_send(serv_params->socket,data,1,0);
		serv_params->tx_task_exit = 0;
		xTaskCreate(ServerTxTask,"SERV_TX",256,serv_params,1,&serv_tx_handle);
		do{
			res = FreeRTOS_recv(serv_params->socket,&data,sizeof(data),0);
			if(res >= 0) xStreamBufferSend(serv_params->rx_stream,&data,res,0);
		}while(res >= 0);
		//An error (receive result < 0) means we should disconnect and re-start listening
		FreeRTOS_shutdown(serv_params->socket,FREERTOS_SHUT_RDWR);
		serv_params->tx_task_exit = 1;
		ulTaskNotifyTake(pdTRUE,portMAX_DELAY);	//wait for tx task completion
		vTaskDelete(serv_tx_handle);
		FreeRTOS_closesocket(serv_params->socket);
		serv_params->socket = NULL;
	}
}



void Network_Init(void){

	server_params.rx_stream = xStreamBufferCreate(128,1);
	xTaskCreate(ServerTask,"SERVER",128*2,&server_params,2,&server_params.server_task);

}

Actually, it’s even easier to reproduce - just a single task that accepts the connection then sends large chunks of data in a loop as fast as possible. I would expect ‘send’ to return an error when the disconnect happens, but for some reason the socket is not leaving the established state. It does bail out of FreeRTOS_send after about 30s from disconnect.

__BSS(BOARD_SDRAM) uint8_t data[125000];
void ThroughputTask(void* params){
	Socket_t xListeningSocket;
	struct freertos_sockaddr xClient, xBindAddress;
	uint32_t xSize = sizeof(xClient);
	BaseType_t res=1;
	Socket_t xClientSocket;

	xListeningSocket = FreeRTOS_socket(FREERTOS_AF_INET4,FREERTOS_SOCK_STREAM,FREERTOS_IPPROTO_TCP );
	xBindAddress.sin_port = FreeRTOS_htons(5911);
	xBindAddress.sin_family = FREERTOS_AF_INET4;
	FreeRTOS_bind(xListeningSocket,&xBindAddress,sizeof(xBindAddress));
	FreeRTOS_listen(xListeningSocket,1);

	while(1){
		xClientSocket = FreeRTOS_accept(xListeningSocket,&xClient, &xSize);
		do{
			res = FreeRTOS_send(xClientSocket,data,125000,0);
		}while(res >= 0);
		//An error (receive result < 0) means we should disconnect and re-start listening
		FreeRTOS_shutdown(xClientSocket,FREERTOS_SHUT_RDWR);
		FreeRTOS_closesocket(xClientSocket);
		xClientSocket = NULL;
	}
}

I’m 99% sure this is the same problem as: TCP Refusing FIN

I have not had the issue after applying the patch. Sorry for the distraction!

@SLocke-CA

Thanks for reporting back.

I tried reproducing the scenario last day with the latest FreeRTOS+TCP (which had the patch) and couldn’t reproduce it with that version.