I am adapting the Cellular Interface Library to an AT modem that has support for higher layer protocols in this case MQTT. The interface is similar to that for sockets so I am trying to adapt the sockets implementation. For the most part where it conforms to the apparent assumptions of the library it is straightforward but I am struggling with where it doesn’t. The challenge is binary data and I’m not sure the challenges would be any different if just trying to use the Raw AT command API.
The particular issue has to do with the receiving of data. In this case the details are indicated in the associated URC and the command to read the data has no text prefix just the raw binary data followed by OK where the length is known from the URC. Something like
AT+SQNSMQTTRCVMESSAGE=0,“mytopic”
mybinarydata
OK
The problem is I think CELLULAR_AT_MULTI_DATA_WO_PREFIX is the only option to accept binary data and it seems to expect at least a line of text response before the data. I don’t see a way to handle this with the library. Can anyone advise how to handle this?
In addition in case of error the response is
AT+SQNSMQTTRCVMESSAGE=0,“mytopic”
ERROR
which I think adds complication because in this case there is a text response. I’m not too concerned with this though because I’m not aware of a legitimate condition where I should get an error.
Assuming you keep the length of “mybinarydata” ( N ) from URC in your context.
Call _Cellular_TimeoutAtcmdDataRecvRequestWithCallback to send the AT command with a data prefix callback function.
Check the first 5 bytes in the pLine. Returns CELLULAR_PKT_STATUS_OK if “ERROR” is received. “ERROR” token should be added in your pCellularSrcTokenErrorTable. _Cellular_TimeoutAtcmdDataRecvRequestWithCallback will return error.
Returns CELLULAR_PKT_STATUS_SIZE_MISMATCH until N + 6 (“\r\nOK\r\n”) bytes are received.
Check the “OK” string is received at the end of pLine. Returns CELLULAR_PKT_STATUS_OK with pDataStart pointed to the first byte of mybinarydata in pLIne and N for pDataLength. The binary data will be stored in the response.
Thanyou for the reply and the guidance. That is similar to what I have tried but the problem is it seems to really want a text response. This code in _handleAllReceived() will call my prefix callback.
while( keepProcess == true )
{
/* Pktio is reading command. Skip over the change line and leading NULL character.
* And the reason we don't consider the variable bytesInBuffer is because
* that the input variable bytesInBuffer is bounded by the caller already.
*/
while( ( bytesRead > 0U ) && ( ( *pTempLine == '\r' ) || ( *pTempLine == '\n' ) || ( *pTempLine == '\0' ) ) )
{
pTempLine++;
bytesRead = bytesRead - 1U;
}
/* Preprocess line. */
keepProcess = _preprocessLine( pContext, pTempLine, &bytesRead, &pStartOfData );
if( keepProcess == true )
{
keepProcess = _findLineInStream( pContext, pTempLine, bytesRead, ¤tLineLength );
}
if( keepProcess == true )
That initial while() loop will skip over all \r,\n,\0 so if my binary data starts with \r,\n,\0 those bytes will be skipped. Then my callback is called from _preprocessLine() but then _findLineInStream() is called afterwards which will replace the first \r or \n with \0 so if my binary data contains a \r or \n it will be overwritten. The net effect is that leading \r,\n,\0 are missed and a later \r or \n is overwritten with \0. So I can kinda get this working but \r, \n, \0 in the data is problematic. If I avoid \r, \n, \0 then I think it works but then I think so would the much simpler CELLULAR_AT_WO_PREFIX.
Thank you for pointing this out.
We could consider to call the prefix callback earlier before skipping over “\r\n\0”. However, this could cause backward compatible problem.
We need more consideration to support this kind of AT command. If you have any suggestion, you are very welcome to share here or create a PR.
Thanks for following up. It’s ironic because there is a text prefix before the > prompt for sending data where the library doesn’t expect one and there is none for receiving data where the library wants one so both required some changes to the library.
For sending the library assumes a bare > response which is somewhat an arbitrary decision so I think this one was straightforward to address.
But receiving is what we have been discussing and honestly I think the issue is really an unfortunate choice in their AT command set, but it is what it is. To address this I added a new command type
CELLULAR_AT_MULTI_DATA_WO_PREFIX, /**< multiple line data response with or without a prefix. */
#if CELLULAR_SEQUANS_HACKS // Added for MQTT data receive.
CELLULAR_AT_MULTI_DATA_NO_PREFIX, /**< multiple line data response with no prefix. */
#endif // CELLULAR_SEQUANS_HACKS
which behaves exactly the same as CELLULAR_AT_MULTI_DATA_WO_PREFIX everywhere except for one change
#if CELLULAR_SEQUANS_HACKS // Added for MQTT data receive.
if( pContext->PktioAtCmdType != CELLULAR_AT_MULTI_DATA_NO_PREFIX )
{
#endif // CELLULAR_SEQUANS_HACKS
/* Pktio is reading command. Skip over the change line and leading NULL character.
* And the reason we don't consider the variable bytesInBuffer is because
* that the input variable bytesInBuffer is bounded by the caller already.
*/
while( ( bytesRead > 0U ) && ( ( *pTempLine == '\r' ) || ( *pTempLine == '\n' ) || ( *pTempLine == '\0' ) ) )
{
pTempLine++;
bytesRead = bytesRead - 1U;
}
#if CELLULAR_SEQUANS_HACKS
}
#endif // CELLULAR_SEQUANS_HACKS
/* Preprocess line. */
This causes it to not skip over the leading \r\n characters which then must be handled in the callback. So in essence the \r\n become the prefix which also has a fortunate side effect that the \r that gets overwritten with \0 is the \r in the ‘prefix’ which is outside of the data so that doesn’t hurt anything now.
You have some choices in the callback regarding what you look for and what you consume. In this case the data is followed by a \r\n and the OK response is lead by a \r\n so I consume these with the data so that the \r\n is seen as the data prefix and anything else the final response (should be OK). The one caveat though is the ERROR response which I don’t see a bulletproof way to handle because the data could contain ‘ERROR’ but as I said I think this is a shortcoming of the AT command. I include the \r\n in that check just to make it less likely to be fooled.
In any case here is the meat of the callback
/* Can't do anything without at least 2 bytes. */
if( lineLength < 2 )
{
pktStatus = CELLULAR_PKT_STATUS_SIZE_MISMATCH;
}
else
{
/* If the first two bytes are \r\n then this is first call. */
if( strncmp(pLine, "\r\n", 2) == 0 )
{
/* Need at least 5+2 more bytes to look for ERROR\r\n response. */
if( lineLength < 2 + 5 + 2 )
{
pktStatus = CELLULAR_PKT_STATUS_SIZE_MISMATCH;
}
else
{
/* Check for ERROR\r\n response */
if( strncmp(pLine + 2, "ERROR\r\n", 5 + 2) == 0 )
{
/* Store the leading \r\n as data to get past it. This will be discarded later
* because the ERROR response will be interpretted as the final command status.
*/
pDataStart = pLine;
*pDataLength = 2;
}
else
{
/* The data follows the leading \r\n. */
pDataStart = pLine + 2;
*pDataLength = 0 + (uint32_t)pCallbackContext + 2 + 2;
LogDebug( ( "DataLength %s at pktIo = %d", pDataStart, *pDataLength ) );
}
*ppDataStart = pDataStart;
}
}
else
{
}
}
}
The pCallbackContext is the length of the data from the URC. Since the context is a single integer I pass the value as the pointer instead of an actual pointer to it.
I think this is ‘safe’ in that it won’t break anything else because it doesn’t alter the original behavior at all it only adds this new behavior. This seems to work well - I’ve been using it in my work for the past week and not seen any issues.
As I said there was another change needed for sending because the > is prefixed but that one is I think straightforward and again I don’t think is at risk of breaking anything. I don’t know if these would be worthwhile changes to the library but they did result in the MQTT integration into the library that I wanted. I said ‘into’ but except for the few changes I needed to make the library to handle the AT command behavior the implementation is really ‘on top of’ the library but it is styled and structured to look as though part of the library.
Thank you for your suggestion. We will look into it. Once we have some thoughts about the problem and your suggestion, we will reply and discuss with you here.
Currently, cellular interface can’t support raw binary recv data without a prefix to indicate the start of binary data in the input stream well. The reason is that it is still possible that a URC string is received after sending the command but before “mybinarydata” is received. In that case, it is difficult for cellular interface to distinguish the URC and binary data.
AT+SQNSMQTTRCVMESSAGE=0,“mytopic”\r\n
=> A URC string is received here.
mybinarydata\r\n
OK\r\n
If the modem returns a prefix string to indicate the start of the binary data, we probably don’t need this patch. Currently, we intend not to merge this patch. Thank you for discussing with us.