Precompile FAT partition into flash

Kernel
1

miclam wrote on Friday, March 17, 2017:

Hello,
I’m trying to use the FreeRTOS +TCP HTTP server which requires +FAT for which I want to create a partition within the upper part of the LPC1758’s 512kb flash. Also I need to deliver default HTML files in the factory build. In the demos is an example which writes an HTML file from a static variable in flash to a ramdisk. In my situation, that would mean I’d copy a string from flash to flash. I’d like to avoid that redundance.

So my question is, if and how it is possible to precompile a fat partition including files so it can be flashed together with the firmware.

Best regards,
Michael

2

heinbali01 wrote on Saturday, March 18, 2017:

Hi Michael, are you familiar with Linux tools?

In Linux you can create a so-called loop device. You can format that as a small FAT16 partition (with the smallest possible cluster size), and fill it with your HTTP-files. When that is done, you can flush and close the loop device and transform the contents of a binary file into one big declaration in C:

    __attribute__((aligned( 4 )))
		const uint8_t ucHTTPDrive[ HTTP_SIZE ] = {
		/* here data. */
	};

The alignment helps memcpy(), which works much faster with aligned pointers.

It would need a simple read-only +FAT driver that reads sectors from that image in flash. Please look at ff_ramdisk.c for an example of such a driver.

3

miclam wrote on Wednesday, March 22, 2017:

Hello Hein,
thanks for your advice, that would be an option. Even though I would need a solution for windows as well.

But reading more into FAT I found another problem. The minimum FAT16 size is 4.1MB. Does this account for the +FAT implementation as well? That would be definitly too big for my 512kb flash (where I intent to use 128-256 kb). Also the limited 32kb RAM is a problem, regarding all the buffers used for caching.

Might FAT12 be a solution? And is it possible to reduce the cluster size to 256byte, as this is the minimum write size the LPC1758 allows?

My initial motivation to implement +FAT was, that the +TCP http server requires it per default. How much work would it be to change it for e.g. the uip http-fs?

Best regards,
Michael

4

heinbali01 wrote on Wednesday, March 22, 2017:

Hi Michael,

that would be an option
Even though I would need a solution for windows as well.

As well? You can create the C source file under Linux and compile under Windows. The nice thing is that the Linux process can easily be automated with a script.

And is it possible to reduce the cluster size to 256byte,
as this is the minimum write size the LPC1758 allows?

I’m afraid that the 512-byte size is almost hard-coded into the library. The reason is that most media use 512-byte sectors.

My initial motivation to implement +FAT was, that the +TCP http server requires it per default.
How much work would it be to change it for e.g. the uip http-fs?

If your embedded web page is very simple, then you might just store a few HTTP files in flash and simulate the calls to ff_fopen(), ff_fread(), and ff_fclose().

I have heard of uIP, but not “uip http-fs”.

The minimum FAT16 size is 4.1MB.
Does this account for the +FAT implementation as well?

+FAT uses ffconfigMIN_CLUSTERS_FAT16 and ffconfigMIN_CLUSTERS_FAT32 to decrease these boundaries. By default, their values are 4086 and 65525 (clusters).

Might FAT12 be a solution?

+FAT also has FAT12 included, but to be honest, I don’t know user’s experiences with it. I rarely use it my self.

Also the limited 32kb RAM is a problem, regarding all the buffers used for caching.

A HTTP server, a +FAT driver, and a small RAM disk?
To put it positively, that’s quite a challenge!


    /* Some parameters to reduce the RAM foot-print of +TCP.
    Settings are not yet tested. */
    #define ipconfigNETWORK_MTU                    552  /* To get an MSS of 512 bytes. */
    #define ipconfigUSE_TCP_WIN                      0  /* No TCP sliding windows. */
    #define ipconfigNUM_NETWORK_BUFFER_DESCRIPTORS   3
    #define ipconfigTCP_RX_BUFFER_LENGTH			( 2 * ipconfigTCP_MSS )
    #define ipconfigTCP_TX_BUFFER_LENGTH			( 2 * ipconfigTCP_MSS )

Regards.

5

miclam wrote on Wednesday, March 22, 2017:

No RAM disk, but Fash disk. But I guess my best option will be to change the underlaying file system in the HTTP server. The http-fs I was talking about comes from Adam Dunkels httpd implementation: https://github.com/adamdunkels/uip/tree/master/apps/webserver

There the HTML files are stored as static variables directly in .data section in flash. Thus read-only, but nearly no overhead. As far as I have seen there are just 3 calls to ff_fopen, ff_fclose and ff_fread, I will change those and provide the sources. Do you see the possibility to officially abstract the filesystem in the HTTP server, so +TCP does not depend on +FAT anymore?

Thanks for the +TCP parameters, I’ll try them out!

Regards, Michael

6

rtel wrote on Wednesday, March 22, 2017:

I’ve not read the whole thread, so maybe my comments are not
appropriate, but:

No RAM disk, but Fash disk. But I guess my best option will be to change
the underlaying file system in the HTTP server. The http-fs I was
talking about comes from Adam Dunkels httpd implementation:
https://github.com/adamdunkels/uip/tree/master/apps/webserver

If this is the uIP webserver I’m familiar with then the file system is
not really a file system, but an array of character arrays. That would
be very difficult to make writable unless all your files are the same
length, or you don’t mind having lots of padding at the end of files so
each file can consume the same space.

Do you see the possibility to
officially abstract the filesystem in the HTTP server, so +TCP does not
depend on +FAT anymore?

I would be grateful if you could elaborate here. As you imply, +TCP
does not depend on +FAT, but the HTTP server is always going to depend
on a file system. It is provided with +FAT, and +FAT already has an
abstraction layer to give it a ‘standard’ (stdio) API option (and that
is the API that is documented) - albeit with ff_ appended to the
expected stdio function names.

7

miclam wrote on Thursday, March 23, 2017:

If this is the uIP webserver I’m familiar with then the file system is
not really a file system, but an array of character arrays. That would
be very difficult to make writable unless all your files are the same
length, or you don’t mind having lots of padding at the end of files so
each file can consume the same space.

Yes, you are right. But as HTTP just requires a read-only filesystem this is enough.

By abstraction I mean, if it is possible to further abstract the ff_fopen, ff_fclose and ff_fread methods, so that it is possible to replace them. Please see my attached modifications in FreeRTOS_HTTP_server.c and FreeRTOS_server_private.h . Basically I just changed the mentioned methods and the file handle data type. My modifications are marked with an // ML: comment.

Also I changed #include "ff_stdio.h" and as in FreeRTOS_server_private.h HTTP and FTP are mixed I commented out the +FAT specific definitions in struct xFTP_CLIENT.

8

miclam wrote on Thursday, March 23, 2017:

Now that I could actually run my code, I have an updated version of the FreeRTOS_HTTP_server.c and FreeRTOS_server_private.h .

9

rtel wrote on Thursday, March 23, 2017:

Forgive me for saying, but this seems backward.

FreeRTOS_HTTP_Server.c is a file that has to work with multiple file
systems. The normal way of doing that is to use some kind of thin port
layer, or abstraction, that is completed for each file system. Most
file systems have a similar stdio style interface, so this is a simple
task. FreeRTOS_HTTP_Server.c stays the same and you somehow externally
map the function calls made by FreeRTOS_HTTP_Server.c to the API used by
your file system.

If you instead updated FreeRTOS_HTTP_Server.c so it can be used with a
specific file system, and then later want to change file systems, or use
FreeRTOS_HTTP_Server.c in a different project that has a different file
system, then you will end up with two inconsistent versions of
FreeRTOS_HTTP_Server.c.

Are you asking us to update +FAT so it works with your updated
FreeRTOS_HTTP_Server.c?

10

miclam wrote on Thursday, March 23, 2017:

Are you asking us to update +FAT so it works with your updated
FreeRTOS_HTTP_Server.c?

No, the other way around. I’d like to have HTTP without dependency on +FAT.

Currently FreeRTOS_HTTP_Server.c depends on +FAT by including ff_stdio.h and using the ff_* methods.

FreeRTOS_HTTP_Server.c is a file that has to work with multiple file
systems. The normal way of doing that is to use some kind of thin port
layer, or abstraction, that is completed for each file system. Most
file systems have a similar stdio style interface, so this is a simple
task. FreeRTOS_HTTP_Server.c stays the same and you somehow externally
map the function calls made by FreeRTOS_HTTP_Server.c to the API used by
your file system.

That is what I wish for, some kind of port abstraction maybe in portable/FileSystem/ with subdirectories like Plus-FAT or http-fs or something else, that one can include/exclude from build. If there is a common header FileSystem.h (in analogy to NetworkInterface.h) which defines TCP_fopen, TCP_fread, … (or a more suitable name) and an abstracted file handle type, that header can be used in FreeRTOS_HTTP_Server.c .

Finally in the port files one can write wrapper methods like my “prv_fread” or simply define TCP_fopen to ff_fopen.

Forgive me for saying, but this seems backward.

My implementation was actually quick and dirty to prove that the http-fs will work (it does) and to show that HTTP does not need to depend on +FAT. I would be glad if a file system abstraction is wished by you and I’m looking for help on doing so.

11

rtel wrote on Friday, March 24, 2017:

Currently FreeRTOS_HTTP_Server.c depends on +FAT by including
|ff_stdio.h| and using the |ff_*| methods.

I’m not following this. You can provide your own implementation of
ff_stdio.h and the ff_ functions - that is how abstraction works - you
provide a shim between the function being called and the function you
want to map it too. If you change the name of the functions then you
will need to provide implementations of the new function names. What
has been gained? Currently the API is ‘standard’ in that it is what
stdio expects, but with ff_ prefixed to the function names so they don’t
clash with the standard C functions. Changing it will make it harder to
port to different file systems as in most cases the port is just a
#define from one to the other.

12

miclam wrote on Tuesday, March 28, 2017:

You can provide your own implementation of
ff_stdio.h and the ff_ functions

Somehow I wasn’t aware that I could simply overwrite +FAT’s names if I’m not using it. That’s what I did now, thanks to your advice, and it works great. No need to mess with FreeRTOS_HTTP_server.c .

I append my implementation of ff_stdio.h and ff_stdio.c relying on httpd-fs ‘file system’ for which routines exist to convert text files into c-arrays, which can be loaded into data section of flash. Works fast and great for me.

Thanks a lot for your help,
Michael

13

heinbali01 wrote on Tuesday, March 28, 2017:

Hi Michael, thanks for sharing these sources. It looks good.

It is indeed common practice to use a set of defines. There is a library WolfSSL that has a port for +TCP and +FAT.

#elif defined(FREERTOS_TCP)

    #define RECV_FUNCTION(a,b,c,d)  FreeRTOS_recv((Socket_t)(a),(void*)(b), (size_t)(c), (BaseType_t)(d))
    #define SEND_FUNCTION(a,b,c,d)  FreeRTOS_send((Socket_t)(a),(void*)(b), (size_t)(c), (BaseType_t)(d))

	#define XFILE                   FF_FILE *
	#define XFOPEN                  ff_fopen
	#define XFSEEK                  ff_fseek
	#define XFTELL                  ff_ftell
	#define XREWIND(F)              ff_fseek(F, 0, FF_SEEK_SET)
	#define XFREAD                  ff_fread
	#define XFWRITE                 ff_fwrite
	#define XFCLOSE                 ff_fclose
	#define XSEEK_END               FF_SEEK_END
	#define XBADFILE                NULL
#elif defined(FREESCALE_MQX)

Another thing I wanted to mention: if you want to create an image of a small FAT16 volume: you can create a RAM-disk in the FreeRTOS/Windows simulator. Fill it with files, and have the application make a dump of the image.
But you probably won’t need that anymore. Regards