CellularLib: Error in processing RAT. Token 11

Partners & Sponsors AWS
1

Hi,

Im running OTA on top of the CellularLib with the uBlox SARA-R4. Occasionally my application does not recover from the following error:

[ERROR] [CellularLib] [cellular_3gpp_urc_handler.c:255] Error in processing RAT. Token 11
[ERROR] [CellularLib] [cellular_common.c:517] _Cellular_TranslateAtCoreStatus: Status 5

Prior to this error the following events occured:

#----------------------------------------------------------#
Unsolicited Network Registration Event Detected... 
networkRegistrationMode==REGISTRATION_MODE_UNKNOWN
csRegistrationStatus==REGISTRATION_STATUS_REGISTRATION_DENIED
psRegistrationStatus==REGISTRATION_STATUS_NOT_REGISTERED_SEARCHING
#----------------------------------------------------------#
#----------------------------------------------------------#
Unsolicited Network Registration Event Detected... 
networkRegistrationMode==REGISTRATION_MODE_UNKNOWN
csRegistrationStatus==REGISTRATION_STATUS_REGISTRATION_DENIED
psRegistrationStatus==REGISTRATION_STATUS_ROAMING_REGISTERED
#----------------------------------------------------------#

My strategy for handling registration events other than:
REGISTRATION_STATUS_REGISTERED_HOME and REGISTRATION_STATUS_ROAMING_REGISTERED is to wait up to 30 seconds after an event and if the modem does not restore the connection within that time I turn the RF off, wait a second and turn the RF back on to force a re-scan. This works for restoring from most network problems, but this does not restore the connection after a “Error in processing RAT. Token 11” log entry.

Anyone here who knows how to handle this correctly? A complete modem power-down/power-up cycle perhaps?

2

@BaxEDM : That is quite an unexpected error. I’ve forwarded this to the cellular library developers and they should be able to provide more assistance.

Until then:
Can you capture a log of the data sent / received on the serial port when this event occurs?

Can you share your serial port open and read function implementations?

Can you clarify the specific module you are using and it’s firmware revision?

+CREG with AcTStatus==11 is a valid value according to the latest 3GPP/ETSI spec, however that would indicate your module is connected to a 5GNR eNodeB. This does not seem likely.

If AcTStatus is actually being reported as 11, I would contact U-Blox support for an explanation.

3

Thank you for your help. First I need to set something up to log the modem Tx as well as the Rx lines into a single log file. Logging one or the other with a single Rx log port is straightforward. But logging both simultaneously into a single log file is not, since the Rx and Tx traffic can occur simultaneously. So I can’t AND the traffic and connect that result to a single Rx.

I will figure it out. I will also have to run for about a week or so to capture the event as is happens not frequently.

The modem I am using is the uBlox SARA-R410M with 02B-01 as firmware indicator I believe.

The serial port open and read implementations are a copy of the STM32L4S5 discovery example implementation I found here:

4

Hi Mike,

I would like to know more about the problem.

From your description,

This works for restoring from most network problems, but this does not restore the connection after a “Error in processing RAT. Token 11” log entry.

Does that mean the RAT is always wrong even after RF off/on the modem when “Error in processing RAT. Token 11” is observed?

It looks like you are using Cellular_RegisterUrcNetworkRegistrationEventCallback API for network registration status.
Print the pInputLine in Cellular_CommonUrcProcessCreg can also provide more information for us.

If you want to try power cycle the cellular modem, you can use Cellular_Cleanup and setupCellular(). Comm interface will be closed and opened by cellular library. The implementation of comm interface should control the power of cellular modem in open and close function.

5

Hi,

After a RF-off / RF-on cycle the RAT error is the first thing that appears in the log. here is the full log:

RAT error 2.zip (128.4 KB)

I will look into expanding the log to get more info and will try to log the modem UART.

The error is hard to reproduce as there are other CellularLib errors occuring during a long run too. For each of these errors, which are probably unrelated, I will create a new topic.

Getting the cellular library to run was not that difficult. Getting it to run reliably over a long time and coping with the various problems cellular connections inherently exhibit is the hard part to me it seems. The current lack of robustness for these exceptions in a serious implementation for a reliable product prevents a workable solution.

I am hoping these problems can be resolved as I am now so heavily invested in this hardware and software solution.

6

I managed to reproduce the error. This time I was also able to log the complete UART traffic between the modem and the STM32. Here’s the log.

RAT error log.zip (5.0 KB)

My application is now stuck in an endless loop, where it tries to restore the cellular connection by turning RF off, waiting 1 sec then turning RF back on, then waiting up to 30 seconds for the connection to come back up. The connection does not come back up however, instead the “Error in processing RAT, Token 11” is given every time.
If I reboot, things work as normal again.

7

Last night my application hung again due to a RAT error. The application log as well as the log of the modem tell the same story.

RAT_Error_logs.zip (1.7 MB)

The time stamp in the log for the start of the problem is <2022.06.26 01:48:46.188>

It starts out with two unsolicited network registration events:

#----------------------------------------------------------#
Unsolicited Network Registration Event Detected... 
networkRegistrationMode==REGISTRATION_MODE_UNKNOWN
csRegistrationStatus==REGISTRATION_STATUS_NOT_REGISTERED_SEARCHING
psRegistrationStatus==REGISTRATION_STATUS_ROAMING_REGISTERED
#----------------------------------------------------------#
#----------------------------------------------------------#
Unsolicited Network Registration Event Detected... 
networkRegistrationMode==REGISTRATION_MODE_UNKNOWN
csRegistrationStatus==REGISTRATION_STATUS_NOT_REGISTERED_SEARCHING
psRegistrationStatus==REGISTRATION_STATUS_NOT_REGISTERED_SEARCHING
#----------------------------------------------------------#

The modem then reports a “+UUSOCL: 0” to indicate that the socket has been closed.

Then two new unsolicited network registration events are given:

#----------------------------------------------------------#
Unsolicited Network Registration Event Detected... 
networkRegistrationMode==REGISTRATION_MODE_UNKNOWN
csRegistrationStatus==REGISTRATION_STATUS_REGISTRATION_DENIED
psRegistrationStatus==REGISTRATION_STATUS_NOT_REGISTERED_SEARCHING
#----------------------------------------------------------#
#----------------------------------------------------------#
Unsolicited Network Registration Event Detected... 
networkRegistrationMode==REGISTRATION_MODE_UNKNOWN
csRegistrationStatus==REGISTRATION_STATUS_REGISTRATION_DENIED
psRegistrationStatus==REGISTRATION_STATUS_ROAMING_REGISTERED
#----------------------------------------------------------#

In an attempt to restore the cellular connection, my application then turns the RF off, waits a second and then turns the RF back on to force a network rescan. The modem uart log shows the two consecutive commands for this: “AT+CFUN=4” @ timestamp <2022.06.26 01:49:17.479> then “AT+CFUN=1” @ timestamp <2022.06.26 01:49:19.652>.

After that the modem reports:

<2022.06.26 01:49:22.994>

+CGREG: 5,"2B61","1A6D867",11


<2022.06.26 01:49:24.638>

+CGREG: 5,"FFFE","1A6D867",9

This leads to the error in the application log:

[ERROR] [CellularLib] [cellular_3gpp_urc_handler.c:255] Error in processing RAT. Token 11
[ERROR] [CellularLib] [cellular_common.c:517] _Cellular_TranslateAtCoreStatus: Status 5

Then both logs show that no matter how many times the RF-off/wait/RF-on cycle is performed, the connection is not restored. After calling RF-on, the modem than always gives +CGREG: 5,“2B61”,“1A6D867”,11.

What is the correct approach to solve this issue?

Since I do not know if the issue is caused by the CellularLib, by the uBlox modem or the combination of both, I have also raised a uBlox support ticket for this issue. Case CA-167423.

8

Hi Mike,

SARA-R4 supports set MNO profile command.
It is better to set the MNO profile according to your network.
It looks like you are using Vodafone. In that case, you can set the MNO to 19.

You can also consider to use RAT priority API to set the RAT searching priority to CELLULAR_RAT_CATM1 if CAT-M1 is your preferred RAT.

CellularRat_t ratPriority[ 1 ] = { CELLULAR_RAT_CATM1 };
cellularStatus = Cellular_SetRatPriority( CellularHandle, ratPriority, 1 );

We are also consulting UBLOX about the unsupported RAT problem. If there is any update, we will update in this thread.

+CGREG: 5,"2B61","1A6D867",11
9

Hi Ching-Hsin,

In-line with your suggestions, I’ve changed the config to set the MNO to 19. I’ve also added the lines:

CellularRat_t ratPriority[ 1 ] = { CELLULAR_RAT_CATM1 };
cellularStatus = Cellular_SetRatPriority( CellularHandle, ratPriority, 1 );

Which are now executed inside of Cellular_setup()

The modem now connects faster, which I really like, however I got another problem for this in return.

I can simulate a bad connection with a switchable RF attenuator, connected in series with the antenna. Prior to the changes listed above, my application would restore a bad connection by forcing a rescan. Now my application does not recover anymore, because it is waiting for a callback that does not arrive. The modem reports that the connection has been re-established:

+CREG: 0

+CGREG: 5,"FFFE","1A6D803",7

+CEREG: 5,"5271","1A6D803",7

but the callback never arrives. So my application now keeps cycling to restore the connection indefinitely. Please see the attached log:

callback_not _executed.zip (14.9 KB)

10

It seems I do get the callback, but the registration status is reported incorrectly. From a log:

+CREG: 0

+CGREG: 5,"FFFE","1A6D803",7

+CEREG: 5,"5271","1A6D803",7

results in two callbacks with registration status:

#----------------------------------------------------------#
Unsolicited Network Registration Event Detected... 
networkRegistrationMode==REGISTRATION_MODE_UNKNOWN
csRegistrationStatus==REGISTRATION_STATUS_NO_REGISTERED_SEARCHING
psRegistrationStatus==REGISTRATION_STATUS_NOT_REGISTERED_SEARCHING
#----------------------------------------------------------#
#----------------------------------------------------------#
Unsolicited Network Registration Event Detected... 
networkRegistrationMode==REGISTRATION_MODE_UNKNOWN
csRegistrationStatus==REGISTRATION_STATUS_NO_REGISTERED_SEARCHING
psRegistrationStatus==REGISTRATION_STATUS_ROAMING_REGISTERED
#----------------------------------------------------------#
11

I’m a bit lost, I reverted to my code prior to these changes a the problem persisted. It did work just fine before. Here’s the logic I use for determining when the connection is restored:

			if((pRegStatus==REGISTRATION_STATUS_REGISTERED_HOME)&&(cRegStatus==REGISTRATION_STATUS_REGISTERED_HOME)){
				cellular_state = CONNECTED;
			}
			if((pRegStatus==REGISTRATION_STATUS_ROAMING_REGISTERED)&&(cRegStatus==REGISTRATION_STATUS_ROAMING_REGISTERED)){
				cellular_state = CONNECTED;
			}
12

You can use psRegStatus ( packet switch registration status ) in CEREG URC for cellular_state if your module is registered on the LTE network Cat M1.

csRegStatus ( circuit switch registration status ) in CREG URC is for circuit switch network. It is not required if you are using Cat M1 network.

13

@Fresh
Thank you for the explanation, csRegstatus can then also be used for Nb-IoT, next to Cat-M1 right?

I have implemented the changes, setting MNO to 19 as well as setting the priority to CELLULAR_RAT_M1.

Now for the first time, I was able to run for 24 hours without running into problems. I will keep running for about a week to make sure there are no other possible bugs. I will report back here later.

14

Hi Mike,

NB-IoT utilizes LTE carrier. Therefore, psRegstatus in CEREG is also used for NB-IoT.

15

@BaxEDM, this ticket will be automatically closed in 7 days. You can reopen it if needed.

16

This topic was automatically closed after 6 days. New replies are no longer allowed.