TCP/IP Issue: Regularly getting "The connection was reset" messages

Kernel
1

dibosco wrote on Monday, July 13, 2015:

Folks,

I’ve been bashing away with much success on the Atmel SAM4E and the new FreeRTOS TCP/IP stack. One issue that I am not able to get rid of is the fact that 30-40% of the time when I request a web page I’m getting a message in Firefox that says “The connection was reset”. The little animation in Firefox to show it’s waiting for a response revolves for a while before it eventually displays the Problem Loading Page page.

I don’t know how to even start tracking this down. Would anyone be able to suggest some things I could try please?

I should maybe clarify that the Atmel part running FreeRTOS is running as the web server and is serving pages to my computer just by typing the IP address into the address bar on Firefox. When I get ia page not loading, I can usually hit the cross in the address bar to stop it and reload the page and then it works fine. I assume somehow the message from the computer to the SAM4 is for some reason not getting through.

Many thanks.

Rob

2

rtel wrote on Monday, July 13, 2015:

Are you using our http server code? Or something else?

Please log the traffic using Wireshark, then attach the pcap file to a forum post (provided the file is not too big - you can filter the capture so it just shows the http traffic to your target).

Regards.

3

heinbali01 wrote on Sunday, August 09, 2015:

Hi Rob,

We handled the issue directly through email. Now for the readers of this forum, here below is a summary of the things that we have found:

Your HTML server replied with data, but you had forgotten to include a ‘Content-Length’, as in:

HTTP/1.1 200 OK
Date: Sun, 9 Aug 2015 22:27:48 GMT
Content-Length: 10<CR><LF>
<CR><LF>
contents<CR><LF>

The browser didn’t like the answers it was receiving and it kept on creating new connections. At your site, the half-closed sockets weren’t properly removed (closed) and after a while all TCP resources were exhausted.

Also I wrote that in a HTML server, sockets must only be closed after a function returns a negative value (normally FreeRTOS_recv() returning -pdFREERTOS_ERRNO_ENOTCONN). In principle, the browser is responsible for shutting down unused connections.

Later you asked me why it takes so much time for your bootloader to download its firmware. I answered with the following text:

If you have very little RAM, try at least to make ‘Win’ double as large as ‘MSS’. See:

http://www.freertos.org/FreeRTOS-Plus/FreeRTOS_Plus_TCP/API/setsockopt.html

with the option FREERTOS_SO_WIN_PROPERTIES.

(
With ‘Win’ I mean the Window size of the TCP-slicing-windows. It is the maximum amount of data that can be outstanding before receiving an ACK. The RX buffer size must always be at least as big as ‘Win’, preferably double the size of ‘Win’.
‘MSS’ stands for Maximum Segment Size: it is the amount of nett TCP-data (or payload) that will fit in a single Ethernet packet. In the source-code it may be defined as:

#define ipconfigTCP_MSS (ipconfigNETWORK_MTU - ipSIZE_OF_IP_HEADER - ipSIZE_OF_TCP_HEADER)

)

TCP performance is greatly enhanced if there can be at least 1 outstanding packet.

In the following example you will get ‘Win = 2 * MSS’ for both directions:

/* Declare an xWinProperties structure. */
xWinProperties_t  xWinProps;

/* Clear the struct, just in case it will
get extended in the future. */
memset( &xWinProps, '\0', sizeof( xWinProps ) );

/* Fill in the required buffer and window sizes. */
/* Unit: bytes */
xWinProps.lTxBufSize = 2 * ipconfigTCP_MSS;
/* Unit: MSS */
xWinProps.lTxWinSize = 2;
/* Unit: bytes */
xWinProps.lRxBufSize = 2 * ipconfigTCP_MSS; // two or more
/* Unit: MSS */
xWinProps.lRxWinSize = 2;

If your MCU has very little RAM you might want to try a smaller MSS:

#define  ipconfigTCP_MSS        536

and get an RX buffer size of at least 2 * 536 = 1072.

If you have enough RAM, you may want to use the maximum of:

#define  ipconfigTCP_MSS       1460

Note that you do not set ipconfigTCP_MSS directly, it will be derived from what you define for MTU:

#define ipconfigNETWORK_MTU    1500

So in order to get an MSS of 536, you’ll define MTU as 536 + 40 = 576:

#define ipconfigNETWORK_MTU     576

After changing the TCP buffer sizes, you wrote: “The speed of the bootloader is staggering now. It programs it faster than a JTAG interface”.

Thanks for that,
Regards.

4

dibosco wrote on Wednesday, August 12, 2015:

OK, I’m back from my holiday and since Friday I’ve been bashing my head against this server brick wall! I thought I’d sorted it this morning, but hadn’t, so now Sourceforge is up and running nicely, I’m hoping someone can put me out of my misery!

Here’s a pcap of the webserver stripped down, only retunring a hit counter. I’ve had this working for over 100 hits sometimes and then, like this example, just 12 times.

Whether or not I put Content-length in makes no difference and from what I can gather it’s optional. Plus, it seems like it’s an issue with the web server not sending out the reply (and indeed not even receiving the request).

I only seem to be able to attach one file, so I’ll attach the web wever threrad on a separate post.

I’ve copied the TCP echo server and modified it slightly. I am sure I’m missing a very important point that I’ve become completely blind to. I would stress it does essentially work - I have a sepatate function I call that parses all sorts of different requests - but just every now and agasin it refuses to send the page back.

My desktop is 192.168.0.35 and the FreeRTOS device is 192.168.0.56 in the pcap.

5

dibosco wrote on Wednesday, August 12, 2015:

And here are the two webserver threads.

I would stress that I have tried it without the FreeRTOS_shutdown as Hein advises, but then it doesn’t work at all. When I do that everything hangs and I don’t even get one response from my web server.

Many thanks!

Rob

PS In case this might help, I went back to my dev board with the USB interface to give me some extra debug information over the CLI. When I
got a failure just now I had this message:
"Recv d bytes xip:351 in status -1062731741

This looks like it must come from this bit of code:

	if( lByteCount > 0 && ( pxSocket->u.xTcp.ucTcpState < eSYN_RECEIVED || pxSocket->u.xTcp.ucTcpState >= eCLOSE_WAIT ) )
	{
		FreeRTOS_printf( ( "Recv %ld bytes %lxip:%u in status %d\n",
			lByteCount,

			/* IP address of remote machine. */
			pxSocket->u.xTcp.ulRemoteIP,

			/* Port on remote machine */
			pxSocket->u.xTcp.usRemotePort,
			pxSocket->u.xTcp.ucTcpState ) );
	}

in FreeRTOS_Sockets.c

Maybe that gives someone an idea of what might be happening?

6

heinbali01 wrote on Thursday, August 13, 2015:

Hi Rob,

Indeed I see 11 successful connections in your PCAP. The 12th time, the browser gets connected but there is no response. Then it starts sending TCP Keep-Alive messages (every 10 sec) which are all responded to.
It looks like the FreeRTOS_accept() call doesn’t finish.

Could you try the following:

static const TickType_t xReceiveTimeOut = pdMS_TO_TICKS( 1000 );

    for( ;; )
    {
        /* Wait for a client to connect. */
        xConnectedSocket = FreeRTOS_accept( xListeningSocket, &xClient, &xSize );
        if( xConnectedSocket != NULL )
        {
            /* Spawn a task to handle the connection. */
            xTaskCreate(  );
        }
    }

In my projects I can not use portMAX_DELAY, because every task has to report to a WDT.

Please have a look at this:

#define LENGTH_FIELD_SIZE    7
    iSendLength = snprintf( RetString, sizeof RetString,
        "HTTP/1.1 200 OK\r\n"
        "Content-type: text/html;charset=ISO-8859-1\r\n"
        "Content-Length: %*d\r\n\r\n",
        LENGTH_FIELD_SIZE, 0);
    iLengthPosition = iSendLength - 4 - LENGTH_FIELD_SIZE;

    iContentLength = snprintf( RetString + iSendLength, sizeof RetString - iSendLength,
        "Hello this is the page. Count = %d\r\n"
        "<a href=\"manual_control.html\">Manually control the lights</a> <br> <br>\r\n\r\n",
        SentTimes );

    sprintf( RetString + iLengthPosition, "%*d", LENGTH_FIELD_SIZE, iContentLength );
    /* Restore the first character after the number, which was a '\r' */
    RetString[ iLengthPosition + LENGTH_FIELD_SIZE ] = '\r';

    iSendLength += iContentLength;
    lRc = FreeRTOS_send( xConnectedSocket, RetString, iSendLength, 0 );

For snprintf(), you might consider using:

FreeRTOS-Plus\Demo\Common\Utilities\printf-stdarg.c

That version of snprintf() will print-out this more properly:

"Recv d bytes xip:351 in status -1062731741

Now I will have a look at your later post.

Regards,
Hein

7

heinbali01 wrote on Thursday, August 13, 2015:

Here attached you’ll find the printf-stdarg.c that I wrote about.

It has got a special format “%xip” or “%lxip” to print IP addresses

NB. The library will need 2 external functions:

BaseType_t xApplicationMemoryPermissions( uint32_t aAddress )
{
BaseType_t xResult = 0;

    /* Check here if the memory address has Read-
    and or Write access. */
	if( xHasReadAccess( aAddress ) )
	{
		xResult = 1;
	}
	if( xHasWriteAccess( aAddress ) )
	{
		xResult |= 2;
	}
	return xResult;
}

void vOutputChar( const char cChar, const TickType_t xTicksToWait  )
{
	/* Included for backward compatibility only. */
	( void ) cChar;
	( void ) xTicksToWait
}

Regards.

8

dibosco wrote on Thursday, August 13, 2015:

Hi Hein,

Thanks for that!

FWIW I did already have printf-stdarg.c in my project with snprintf commented out.

I get errors when I try to compile the new printf-stdarg.c:

undefined reference to `tiny_printf’

Plus the lines in the those two functions give me errors:

undefined reference to xHasReadAccess' undefined reference to xHasWriteAccess’

The vOutputChar function that does nothing is just to keep a calling function happy I guess?

So, ignoring that - using your method - and just using the original stdio version [of snprintf] and using the xReceiveTimeOut set to 1000ms, I got it to work 72 times and was getting as little excited when it fell over.

I would say that I have used my own method of inserting Content-length after our emails a few weeks ago which seems to work identically to yours. Neither my version or your version seem to be the issue.

I’m going to put some more CLI debug messages in at various places in an attempt to track it down further.

Cheers,

Rob

9

dibosco wrote on Thursday, August 13, 2015:

Some further info. I changed the code to this:

        xConnectedSocket = FreeRTOS_accept( xListeningSocket, &xClient, &xSize );
        if( xConnectedSocket != NULL )
            {
            usUsedStackSize = 1000;
            FreeRTOS_printf( ("We have received a request\n" ) );
            /* Spawn a task to handle the connection. */
            xTaskCreate( webServerConnectionInstance, "WebServerInstance", usUsedStackSize, ( void * ) xConnectedSocket, tskIDLE_PRIORITY, NULL );
            }
       else
          {
          FreeRTOS_printf( ("No request\n" ) );
          }

You can see in the .txt file I attached that the No request is printed [every one second].

At the end of the file, it’s fallen over in that FreeRTOS is no longer responding to reuqests from the broswer; you can, however, see that the debug messages are still coming saying that there has been no request every one second.

So, if I am understanding this correctly, it’s the TCP/IP stack that is just no longer processing the incoming request from the browser?

Are there any messages I could put in further down the stack that might track this down?

I should add that at one point I did get a failure, but requesting the page by stopping the broswer and hitting refresh worked. The second time it failed, it won’t work at all.

Thanks!

Rob

Edit: I added another CLI debug message text file that shows one instance where I got two or three failures, just clicked the link again while it [the broswer] was waiting, then I got two

"We have received a request Sending length 197 "

messages. Which, I think, suggests both requests are received, but the first somehow gets stuck and needs the “nudge” of a second request.

10

heinbali01 wrote on Friday, August 14, 2015:

Hi Rob,

About [v]s[n]printf() :

You can use printf-stdarg.c if you define:

    BaseType_t xApplicationMemoryPermissions( uint32_t aAddress )
    {
        return 3;
    }

    void vOutputChar( const char cChar, const TickType_t xTicksToWait )
    {
    }

In the earlier (antique) version of printf-stdarg.c, sprintf() could be called with a NULL pointer, and then all characters would be forwarded to vOutputChar(), which is very slow.

If you do provide a pointer, vOutputChar() will never be called.

About xApplicationMemoryPermissions() : sometimes you are debugging code and by mistake you write:

    /* By mistake you put a %s in stead of %d */
    sprintf(buf, "Result = %s", iValue);

The integer value of ‘iValue’ (-1) probably points to non-existent memory and the debugging code itself will cause your application to crash. which is frustrating.

The mentioned function can prevent this by telling if the memory referred to has RD and/or WR permissions. It will print out: "Result = INV_MEM"

Why not simply use the standard provided family of [v]s[n]printf() ?

  • Because it uses a huge amount of stack space > 1000 bytes
  • Because it doesn’t seem to thread-safe, at least in Xilinx
  • Because it may omit to put a nul character if the result doesn’t fit
    char buffer[ 5 ];
    snprintf( buffer, sizeof( buffer ), "%s", "Hello world" );

    /* Now buffer will contain a non-terminated string 'hello',
    whereas the 'printf-stdarg.c' version will print 'hell' */

Last week I lost several hours finding out why strings were corrupted like this:

"Test string 504\n"
"Test string 505\n"
"Test string 50e\n"  <= "506 expected: corrupted string
"Test string 507\n"

Five tasks (with equal priority) were using GCC’s standard snprintf() routine simultaneously, without ever blocking.

Regards.

11

dibosco wrote on Friday, August 14, 2015:

Morning Hein,

I’m still getting issues, sorry.

If I add those two functions as you put them above I get the error:

undefined reference to `tiny_printf’

It turns out tiny_printf was in the original versionn of printf-stdarg.c that was in my project. If I just try to add that function to this new file you sent me:

int tiny_printf(const char *format, …)
{
va_list args;

    va_start( args, format );
    return tiny_print( 0, format, args, 0 );

}

On the return line I get the errors:

too many arguments to function ‘tiny_print’
void value not ignored as it ought to be

I must say, I’ve never used snprintf! You may have been meaning one makes the mistake of putting %s rather than me specifically though! :smiley: