Queue feature request

anonymous wrote on Sunday, February 12, 2012:

Is it possible to add variable length item size when creating new Queue with uxItemSize == 0?

rtel wrote on Sunday, February 12, 2012:

That won’t be possible without adding a different queue type, because the queue storage is allocated when the queue is created.

The queue design is intended to be as flexible as possible.  It works by copy rather than by reference because that is by far the most efficient and effective way to work (no matter what any critic says).  If you want to queue fixed size items, then you don’t have to worry about who owns/releases the memory because the queue takes its own copy. 

If you want the queue to work by reference (as other systems do), then you can do that too simply by queuing a pointer which is exactly what these other systems do anyway so there is no efficiency loss.

If you want to queue variable sized items then have the queue item size equal to sizeof( void * ), and queue a pointer to the item.  You must then take care as to who owns the memory being pointed to.

Regards.

anonymous wrote on Sunday, February 12, 2012:

I mean storing items by value with they actual size.
This will give more flexibilty, but requires addition byte per item cost.
I can write possible changes if needed.

rtel wrote on Sunday, February 12, 2012:

I don’t think the extra byte is an additional cost - the size would have to be stored somewhere.  If you weren’t doing it explicitly then it would be stored inside the API controlled data, or inside the block that was allocated.

Regards.

anonymous wrote on Sunday, February 12, 2012:

Yes. Inside xQUEUE::pcHead.
When xQueueSendExt(Queue, pvItemToQueue, uxItemSize, xTicksToWait ) called, API saves uxItemSize and pvItemToQueue[0…pvItemToQueue) bytes continuously.
When xQueueReceiveExt( xQueue, pvBuffer, pxBufferSize, uxBufferCapacity,  xTicksToWait ) called, API reads saved size and copies necessary amount into the pvBuffer.

rtel wrote on Sunday, February 12, 2012:

Interesting idea, but how does the task calling the receive function know how big pvBuffer needs to be?

Regards.

anonymous wrote on Sunday, February 12, 2012:

In most situation task has local buffer or something else.
In addition, function xQueueReceiveExt get uxBufferCapacity and if stored item bigger then uxBufferCapacity, it will save stored size into pxBufferSize variable and returns errNOT_ENOUGH_MEMORY

richard_damon wrote on Sunday, February 12, 2012:

I will point out also that currently a queue with item size of 0 is used by the Semaphore code to make a queue which doesn’t store any data.

One other difficulty that you are going to run into when trying to queue up variable sized data. Lets say you want to queue up a 10 byte buffer, but the queue only has 5 byte of room left. You have two choices now, you can queue up the first 5 bytes, suspend the task inserting for room to become available (and if the timeout happens remove those 5 bytes), and somehow mark the queue as “busy” so if a higher priority task come along and want to push data into it, it can’t, otherwise you break your buffers up. This basically adds the need of a semaphore to the queue for access control.

The other option is the task doesn’t start to put the buffer, but now needs to wait somehow for more space, even though the queue isn’t currently “full” by the basic definition.  If adding such a wait isn’t an issue, perhaps it could be coded as an object size of -1.

Note also that you are going to need a different sort of put/get routine to access the queue, as you now need to include the byte count to put, and should also include the max byte count to get (and maybe an access function for the size of the next message). You then also need to decide how to handle a message to big for the get, is the remainder left as a new message (by putting back a new remaining byte count), or is the rest of the message just thrown away.

In the mean time, you can code up a “queue” that does this sort of action, the creation of which creates a byte oriented queue for storing the data and two semaphores, “puts” grab the write semaphore and then place the count followed by data, then release the semaphore, gets grab the read semaphore and grab the data, dealing with too much data as decided, the release the read semaphore.

anonymous wrote on Sunday, February 12, 2012:

>I will point out also that currently a queue with item size of 0 is used by the Semaphore code to make a queue which doesn’t store any data

It’s not a problem. We can pass -1, for example.

>One other difficulty that you are going to run into when trying to queue up variable sized data. Lets say you want to queue up a 10 byte buffer, but the queue only has 5 byte of room left.

Same as with ordinary Queues, sleep until someone Reseives item or timeout.

>Note also that you are going to need a different sort of put/get routine to access the queue, as you now need to include the byte count to put

Yes.

rahimoval wrote on Monday, October 24, 2016:

Hi All,
Sorry for updating the old thread, but I really need this feature? Was that implemented? Any links are appreciated.
Thanks

rtel wrote on Monday, October 24, 2016:

No - you can create a queue that holds structures though, then have a
structure member that points to the data, and another structure member
that holds the size of the data.

In the TCP/IP stack we use a stream buffer (effectively a thread safe
circular buffer) in which we post data, and then the size of the date.
The receiver then receives the size first, so knows how many more bytes
to read out of the stream buffer.