Mege OTA and greengrass demo together

TJirsch wrote on September 21, 2019:

I thought that you wanted to do OTA Updates via a MQTT Session connected to a Greengrass host.
This did not work for me, because the MQTT host of Greengrass does not support/offer QOS1, which is needed by the MQTT Session used by OTA, but only QOS1.

embeddedx wrote on September 21, 2019:

Hello,

This is the thing policy


{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "iot:*",
        "greengrass:*"
      ],
      "Resource": [
        "*"
      ]
    }
  ]

For the greengrass group I have these subscriptions:


IoT Cloud                   HelloWorld_Publisher              TEST
HelloWorld_Publisher   IoT Cloud                              TEST

I’m not sure if this setup has sth to do witht OTA, but I don’t think so.
What I noticed I just got only on message from the greengrassdemo and one callback!

I stopped the subscription and the publishing of the greengrass demo, I used it for just fetching the greengrass endpoint and credentials, and this what I got from in the cloudwatch log


2019-09-21 08:56:01.966 TRACEID:886fc0b9-8612-c390-35c5-7813975d00ec PRINCIPALID:30388d8fc1290ef9cbc4caa641b343c67f447c1720f935fbbd50eeaab8218c9a [INFO] EVENT:MQTT Client Disconnect MESSAGE: IpAddress: 105.80.18.254 SourcePort: 60933

2019-09-21 08:47:35.917 TRACEID:fc6ab13e-e2b3-312c-d6dc-9837b26e691a PRINCIPALID:MyFirstGroup_Core-gci [INFO] EVENT:GetThingShadow THINGNAME:MyFirstGroup_Core-gci

2019-09-21 08:50:25.844 TRACEID:c710fe91-2037-77a5-1f1f-1f1838ebb40d PRINCIPALID:HelloWorld_Publisher [INFO] EVENT:GetThingShadow THINGNAME:HelloWorld_Publisher

2019-09-21 08:50:28.991 TRACEID:3c02cf43-1c04-450a-c6e1-6970788965c0 PRINCIPALID:713567411902 [INFO] EVENT:MQTT Client Connect MESSAGE:Connect Status: SUCCESS

2019-09-21 08:50:28.991 TRACEID:3c02cf43-1c04-450a-c6e1-6970788965c0 PRINCIPALID:713567411902 [INFO] EVENT:MQTT Client Connect MESSAGE: IpAddress: 105.80.18.254 SourcePort: 62001

2019-09-21 08:50:29.259 TRACEID:960477a0-2e8d-3018-f1da-7bcb608dd41f PRINCIPALID:713567411902 [INFO] EVENT:MQTTClient Subscribe TOPICNAME:$aws/things/HelloWorld_Publisher/shadow/update/rejected MESSAGE:Subscribe Status: SUCCESS
2019-09-21 08:50:29.259 TRACEID:5fe2e9dd-a328-554d-e7fe-5cf49164f289 PRINCIPALID:713567411902 [INFO] EVENT:MQTTClient Subscribe MESSAGE: IpAddress: 105.80.18.254 SourcePort: 62001
2019-09-21 08:50:29.259 TRACEID:960477a0-2e8d-3018-f1da-7bcb608dd41f PRINCIPALID:713567411902 [INFO] EVENT:MQTTClient Subscribe MESSAGE: IpAddress: 105.80.18.254 SourcePort: 62001
2019-09-21 08:50:29.259 TRACEID:154921b6-233a-5545-ebd9-56e5d4d10a4c PRINCIPALID:713567411902 [INFO] EVENT:MQTTClient Subscribe TOPICNAME:$aws/things/HelloWorld_Publisher/shadow/delete/accepted MESSAGE:Subscribe Status: SUCCESS
2019-09-21 08:50:29.259 TRACEID:154921b6-233a-5545-ebd9-56e5d4d10a4c PRINCIPALID:713567411902 [INFO] EVENT:MQTTClient Subscribe MESSAGE: IpAddress: 105.80.18.254 SourcePort: 62001
2019-09-21 08:50:29.259 TRACEID:5fe2e9dd-a328-554d-e7fe-5cf49164f289 PRINCIPALID:713567411902 [INFO] EVENT:MQTTClient Subscribe TOPICNAME:$aws/things/HelloWorld_Publisher/shadow/update/accepted MESSAGE:Subscribe Status: SUCCESS
2019-09-21 08:50:29.264 TRACEID:4eb01bc7-956b-900f-88cb-a8c1f99e5665 PRINCIPALID:713567411902 [INFO] EVENT:MQTTClient Subscribe TOPICNAME:$aws/things/HelloWorld_Publisher/shadow/delete/rejected MESSAGE:Subscribe Status: SUCCESS
2019-09-21 08:50:29.264 TRACEID:1304f6b5-3ca9-76db-49fc-fcfdaf1916ef PRINCIPALID:713567411902 [INFO] EVENT:MQTTClient Subscribe MESSAGE: IpAddress: 105.80.18.254 SourcePort: 62001
2019-09-21 08:50:29.264 TRACEID:bfffbd1b-fca1-8a07-c8b0-7f4b35f7b4a8 PRINCIPALID:713567411902 [INFO] EVENT:MQTTClient Subscribe MESSAGE: IpAddress: 105.80.18.254 SourcePort: 62001
2019-09-21 08:50:29.264 TRACEID:bfffbd1b-fca1-8a07-c8b0-7f4b35f7b4a8 PRINCIPALID:713567411902 [INFO] EVENT:MQTTClient Subscribe TOPICNAME:$aws/events/presence/disconnected/HelloWorld_Publisher MESSAGE:Subscribe Status: SUCCESS
2019-09-21 08:50:29.264 TRACEID:1304f6b5-3ca9-76db-49fc-fcfdaf1916ef PRINCIPALID:713567411902 [INFO] EVENT:MQTTClient Subscribe TOPICNAME:$aws/things/HelloWorld_Publisher/shadow/update/documents MESSAGE:Subscribe Status: SUCCESS
2019-09-21 08:50:29.267 TRACEID:0000a23a-5ba9-64bc-72d8-96b32cd2fa88 PRINCIPALID:713567411902 [INFO] EVENT:MQTTClient Subscribe TOPICNAME:$aws/events/subscriptions/subscribed/HelloWorld_Publisher MESSAGE:Subscribe Status: SUCCESS
2019-09-21 08:50:29.267 TRACEID:0000a23a-5ba9-64bc-72d8-96b32cd2fa88 PRINCIPALID:713567411902 [INFO] EVENT:MQTTClient Subscribe MESSAGE: IpAddress: 105.80.18.254 SourcePort: 62001
2019-09-21 08:50:29.269 TRACEID:891df95c-4f45-5b5d-baa9-70894445ecdf PRINCIPALID:713567411902 [INFO] EVENT:MQTTClient Subscribe TOPICNAME:$aws/events/subscriptions/unsubscribed/HelloWorld_Publisher MESSAGE:Subscribe Status: SUCCESS
2019-09-21 08:50:29.307 TRACEID:7a7c8353-aa2a-d693-4a20-e045bb7f7b46 PRINCIPALID:713567411902 [INFO] EVENT:MQTTClient Subscribe TOPICNAME:$aws/things/HelloWorld_Publisher/shadow/delete/accepted MESSAGE:Subscribe Status: SUCCESS

2019-09-21 08:50:29.264 TRACEID:4eb01bc7-956b-900f-88cb-a8c1f99e5665 PRINCIPALID:713567411902 [INFO] EVENT:MQTTClient Subscribe MESSAGE: IpAddress: 105.80.18.254 SourcePort: 62001
2019-09-21 08:50:29.269 TRACEID:891df95c-4f45-5b5d-baa9-70894445ecdf PRINCIPALID:713567411902 [INFO] EVENT:MQTTClient Subscribe MESSAGE: IpAddress: 105.80.18.254 SourcePort: 62001
2019-09-21 08:50:29.307 TRACEID:7a7c8353-aa2a-d693-4a20-e045bb7f7b46 PRINCIPALID:713567411902 [INFO] EVENT:MQTTClient Subscribe MESSAGE: IpAddress: 105.80.18.254 SourcePort: 62001
2019-09-21 08:50:29.348 TRACEID:b81ead84-95fb-1869-600a-9261b64cdcdc PRINCIPALID:713567411902 [INFO] EVENT:MQTTClient Subscribe TOPICNAME:$aws/events/presence/connected/HelloWorld_Publisher MESSAGE:Subscribe Status: SUCCESS

2019-09-21 08:56:01.966 TRACEID:886fc0b9-8612-c390-35c5-7813975d00ec PRINCIPALID:30388d8fc1290ef9cbc4caa641b343c67f447c1720f935fbbd50eeaab8218c9a [INFO] EVENT:MQTT Client Disconnect MESSAGE:Disconnect Status: SUCCESS

2019-09-21 08:56:25.803 TRACEID:870336e5-7631-8454-c498-66b69845688b PRINCIPALID:30388d8fc1290ef9cbc4caa641b343c67f447c1720f935fbbd50eeaab8218c9a [INFO] EVENT:MQTT Client Connect MESSAGE:Connect Status: SUCCESS

2019-09-21 08:51:07.360 TRACEID:5a83a76d-6cb0-9aa9-6572-48eef53c12b5 PRINCIPALID:HelloWorld_Publisher [INFO] EVENT:GetThingShadow THINGNAME:HelloWorld_Publisher
2019-09-21 08:56:25.803 TRACEID:870336e5-7631-8454-c498-66b69845688b PRINCIPALID:30388d8fc1290ef9cbc4caa641b343c67f447c1720f935fbbd50eeaab8218c9a [INFO] EVENT:MQTT Client Connect MESSAGE: IpAddress: 105.80.18.254 SourcePort: 62198

2019-09-21 08:44:58.255 TRACEID:bf158382-f521-806d-85ea-aec8b414bf98 PRINCIPALID:MyFirstGroup_Core-gci [INFO] EVENT:GetThingShadow THINGNAME:MyFirstGroup_Core-gci
2019-09-21 09:01:16.100 TRACEID:a5340491-15b0-8bf8-c99c-8f0c774aab09 PRINCIPALID:30388d8fc1290ef9cbc4caa641b343c67f447c1720f935fbbd50eeaab8218c9a [INFO] EVENT:MQTT Client Disconnect MESSAGE:Disconnect Status: SUCCESS

2019-09-21 09:01:16.100 TRACEID:a5340491-15b0-8bf8-c99c-8f0c774aab09 PRINCIPALID:30388d8fc1290ef9cbc4caa641b343c67f447c1720f935fbbd50eeaab8218c9a [INFO] EVENT:MQTT Client Disconnect MESSAGE: IpAddress: 105.80.18.254 SourcePort: 62198

2019-09-21 09:01:43.165 TRACEID:a5ad13b0-e543-97c4-45ae-104c70917986 PRINCIPALID:30388d8fc1290ef9cbc4caa641b343c67f447c1720f935fbbd50eeaab8218c9a [INFO] EVENT:MQTT Client Connect MESSAGE:Connect Status: SUCCESS
2019-09-21 09:01:43.165 TRACEID:a5ad13b0-e543-97c4-45ae-104c70917986 PRINCIPALID:30388d8fc1290ef9cbc4caa641b343c67f447c1720f935fbbd50eeaab8218c9a [INFO] EVENT:MQTT Client Connect MESSAGE: IpAddress: 105.80.18.254 SourcePort: 62392
2019-09-21 09:02:00.737 TRACEID:29275edc-22af-d4d4-f419-3b458cd0f365 PRINCIPALID:713567411902 [INFO] EVENT:HTTP Client Disconnect MESSAGE: IpAddress: 105.80.18.254 SourcePort: 62001

2019-09-21 09:02:00.737 TRACEID:29275edc-22af-d4d4-f419-3b458cd0f365 PRINCIPALID:713567411902 [ERROR] EVENT:HTTP Client Disconnect MESSAGE:Disconnect Status: CLIENT_ERROR Failure reason:DUPLICATE_CLIENT_ID

The last message is kinda interesting.

This is the pcap log of the ESP32
[https://drive.google.com/file/d/1IYcZpBveEK7KO29uazPkFOGm7UAgE3aX/view?usp=sharing]

Edited by: embeddedx on Sep 21, 2019 2:35 AM

TJirsch wrote on September 21, 2019:

As far as I remember this did not work when I tried, because GG MQTT offers only QOS0 while the OTA Agent needs QOS1.
I tried to fiddle with the OTA code to make it use QOS0 but gave up on some point.
The other thing you will need to do is to figure out the GG subscriptions for the ota agent.
You can get them from the greengrass hosts logs as they are reported.
Wildcard subscriptions on the root do not work.

OTA over greengrass would be a really cool feature, because the need for the IoT Thing to connect to the Internet directly would be reduced.

embeddedx wrote on September 21, 2019:

Thanks for this information.

I don’t get this point, do you mean that the broker supports only QOS0 and will not respond to QOS1 requests?

You can get them from the greengrass hosts logs as they are reported.
Wildcard subscriptions on the root do not work.
Could you explain this more. I also have an issue with the greengrass logging (the log folder can’t be accessed with permission denied error) on EC2 instance!

Thank

embeddedx wrote on September 22, 2019:

Thanks for your reply.
I’m trying to do similar to what you did.
The problem is I can’t have two client one for the greengreass and one for IOT cloud, the TLS layer populate with error code -12 and -16 and I don’t find those two errors.
I went to the MQTT demo trying to have two client connection and I had the same error code -16


status = _establishMqttConnection( awsIotMqttMode,
                                           NULL,
                                           pNetworkServerInfo,
                                           pNetworkCredentialInfo,
                                           pNetworkInterface,
                                           &mqttConnection );
         if( status == EXIT_SUCCESS )
         {
             IotLogInfo("Sucess 1 \r
");
                 status = _establishMqttConnection( awsIotMqttMode,
                                           NULL,
                                           pNetworkServerInfo,
                                           pNetworkCredentialInfo,
                                           pNetworkInterface,
                                           &mqttConnection2 );

            if( status == EXIT_SUCCESS )
            {
                IotLogInfo("Sucess 2 \r
");                               
            } 
            else
            {
                    IotLogInfo("Fail 2 \r
");
            }
         }
         else
         {
             IotLogInfo("Fail 1 \r
");
         }

embeddedx wrote on September 23, 2019:

Hi,
I still have the TLS error.
I’m trying to have two separate client connection as I mentioned and creation of the second cliemt connection fails.

embeddedx wrote on September 22, 2019:

Yes that what I’m trying to do, but there is already documentation about OTA over greengrass, so how come the MQTT doesn’t support it?!

Thanks

Gaurav-Aggarwal-AWS wrote on September 23, 2019:

Were you able to resolve this TLS error?

Regarding the approach of sharing the same MQTT connection from Greengrass demo to perform OTA over greengrass, I was able to solve the NETWORK_ERROR during MQTT publish by changing MQTT Publishs to QoS0 instead of QoS1 in the OTA code. But I am still not able to get the OTA job notification and OTA data on the device.

Note that this is not something which is officially supported and I am trying this for the first time myself. I’ll try to get more information about this.

Thanks.

embeddedx wrote on September 22, 2019:

Sorry the document is about the core update not the devices connected to it.
However having two MQTT clients one for greengrass and the other for OTA over IOT core is failing!
Have you managed to have both demos working together?

TJirsch wrote on September 22, 2019:

I did that with an app completely separate from the demos, because in my opinion the demos are harder to adapt than to start from scratch.

I started building the app to connect to greengrass shadows properly and then integrated the ota functionality from the demos, example code and guessing.

It works with two mqtt connections. I also needed to connect to an existing onsite non AWS mqtt host and had to start tweaking mbedtls parameters to squeeze the tree connections in.
I ended up with a scheduled task scenario, that terminated the two mqtt connections, started ota, checked for update jobs and restarted the other “normal” mqtt connections.

The check does not take longer than one or two minutes and it was ok for this usecase to loose connections to the iot backend for that time.

aggarg wrote on September 24, 2019:

I am now able to get OTA via Greengrass working. There are some code changes needed:

  1. Use MQTT connection to Greengrass for OTA (basically similar to what you did already).
  2. Change MQTT Quality of Service to QoS0 from QoS1 in the OTA Agent - replace all IOT_MQTT_QOS_1 with IOT_MQTT_QOS_0 in libraries/freertos_plus/aws/ota/src/aws_iot_ota_agent.c.

After the above code changes, you need to establish the following Subscriptions for your Greengrass group (Replace greengrass_thing with your thing name):


Device to Cloud
$aws/things/greengrass_thing/jobs/$next/get
$aws/things/greengrass_thing/jobs/+/update
$aws/things/greengrass_thing/streams/+/get/cbor

Cloud to Device
$aws/things/greengrass_thing/jobs/$next/get/accepted
$aws/things/greengrass_thing/jobs/notify-next
$aws/things/greengrass_thing/streams/+/data/cbor

After the above changes, it works for me.

Thanks.

TJirsch wrote on September 24, 2019:

I have tried to share a MQTT Connection via Greengrass with the OTA Agent.
From what I see, you get the same errors as I do:


"438 3328 [OTA Task] [OTA_CheckForUpdate] Failed to publish MQTT message."

This message corresponds to a message on greengrass(from my logs):

"[2019-09-25T00:26:08.19+02:00][ERROR]-MQTT QoS Error: Greengrass does not support QoS 1."

The message in the greengrass logs always occurs near the message on the esp32 from the OTA Task.

The OTA Task wants to use QOS1, Greengrass only supports QOS0, so , OTA over GG, no can do.
Look in the greengrass logs for yourself with:
tail -f /greengrass/ggc/var/log/system/*.log /greengrass/ggc/var/log/system/localwatch/localwatch.log

If you supply the OTA Agent with a MQTT Connection directly to the IoT Backend, all is fine, you can use the same client id/certs etc. in this case.
The clientid must be unique +per Broker+, so you can connect to GG and the cloud with the same id because the two are different brokers.

What seems to work now, is sharing an MQTT connection for Thing Shadow and normal MQTT to the same broker, both for Greengrass and IoT Backend. This is an aw(e|s)some piece of work and removes the need for a third Mqtt connection, if you want to do ota, shadow and mqtt to aws at the same time.

I am using a timertask to switch between “normal operations” and “go look for an ota update” mode to keep memory usage down.

Gaurav-Aggarwal-AWS wrote on September 24, 2019:

What I mentioned in the above post is that after changing the OTA agent to use QoS0 instead of QoS1 and establishing the above mentioned subscriptions in the Greengrass group, sharing MQTT connection to Greengrass core for OTA works for me.

Thanks.

TJirsch wrote on September 25, 2019:

I have changed the QOS Settings in the ota_agent.c file and now the ota updates work in parallel to the shadow updates on one mqtt connection over greengrass and also if the mqtt connection is directly connected to the cloud.
That is really cool ! Thank you.

Gaurav-Aggarwal-AWS wrote on September 25, 2019:

I did not create a separate OTA task but looking at the code you shared in the beginning of this post, Greengrass publishes should work. Here are the few things to check:

  1. Do you have the required subscription in your Greengrass group i.e. from device to cloud for topic “freertos/demos/ggd”?
  2. Is MQTT_AGENT_Publish in the function prvSendMessageToGGC is being called? You can check that by putting a log message before calling MQTT_AGENT_Publish.
  3. Subscribe to the topic “freertos/demos/ggd” on the AWS IoT console to see if those messages are being published. Note that the demo only publishes 3 messages which is configured using the ggdDEMO_MAX_MQTT_MESSAGES macro.

Thanks.

embeddedx wrote on September 25, 2019:

Thanks fof your efforts
When do you create the OTA task?
It seems the greengrass stopped publishing after after OTA task creation!

Thanks

TJirsch wrote on September 26, 2019:

I would like to create a PR for the QoS change, what would be the correct way to do so ?
I can fork the repo an create a PR from that.

Gaurav-Aggarwal-AWS wrote on September 27, 2019:

Yes, you can create a PR that way. However, we would probably want to make the change in a more configureable way rather than changing them to QoS0. I have asked my colleagues to take a look and see if they can address it.

Thanks.

embeddedx wrote on October 02, 2019:

Hi,
By following the mentioned updates and creating the new subscriptions, requesting OTA job information is now possible.
However I got an error in receiving one block and this cause the connection to be closed!

https://gist.github.com/ahmedwahdan/606bc99e9b2a26b3f07f8ff1ee2bef6e


670 15713 [iot_thread] State: Active  Received: 200   Queued: 200   Processed: 200   Dropped: 0
671 15807 [OTA Task] [prvIngestDataBlock] Received file block 202, size 1024
672 15808 [OTA Task] [prvIngestDataBlock] Remaining: 670
673 15808 [OTA Task] [prvIngestDataBlock] Received file block 203, size 1024
674 15808 [OTA Task] [prvIngestDataBlock] Remaining: 669
675 15833 [iot_thread] WAHDAN : vRunOTAUpdateDemo : Inside the while loop
676 15833 [iot_thread] State: Active  Received: 202   Queued: 202   Processed: 202   Dropped: 0
677 15953 [iot_thread] WAHDAN : vRunOTAUpdateDemo : Inside the while loop
678 15953 [iot_thread] State: Active  Received: 202   Queued: 202   Processed: 202   Dropped: 0
679 16058 [OTA Task] [INFO ][MQTT][160580] (MQTT connection 0x3fff2190) MQTT PUBLISH operation queued.
680 16058 [OTA Task] [prvPublishGetStreamMessage] OK: $aws/things/HelloWorld_Publisher/streams/AFR_OTA-2374f622-5de3-4930-
8cbe-4e9d5d508b0d/get/cbor
681 16060 [OTA Task] [prvIngestDataBlock] Received file block 204, size 1024
682 16060 [OTA Task] [prvIngestDataBlock] Remaining: 668
683 16073 [iot_thread] WAHDAN : vRunOTAUpdateDemo : Inside the while loop
684 16073 [iot_thread] State: Active  Received: 203   Queued: 203   Processed: 203   Dropped: 0
685 16193 [iot_thread] WAHDAN : vRunOTAUpdateDemo : Inside the while loop
686 16193 [iot_thread] State: Active  Received: 203   Queued: 203   Processed: 203   Dropped: 0
687 16260 [NetRecv] [ERROR][NET][162600] Error 0 while receiving data.
688 16260 [NetRecv] [WARN ][NET][162600] Receive requested 1140 bytes, but 97 bytes received instead.
689 16260 [NetRecv] [ERROR][MQTT][162600] (MQTT connection 0x3fff2190) Error processing incoming data. Closing connection.

690 16261 [NetRecv] [INFO ][MQTT][162610] (MQTT connection 0x3fff2190) Network connection closed.
691 16309 [OTA Task] [WARN ][MQTT][163090] (MQTT connection 0x3fff2190) Attempt to use closed connection.
692 16309 [OTA Task] [ERROR][MQTT][163090] (MQTT connection 0x3fff2190) New operation record cannot be created for a close
d connection
693 16309 [OTA Task] [prvPublishGetStreamMessage] Failed: $aws/things/HelloWorld_Publisher/streams/AFR_OTA-2374f622-5de3-4
930-8cbe-4e9d5d508b0d/get/cbor
694 16313 [iot_thread] WAHDAN : vRunOTAUpdateDemo : Inside the while loop
695 16313 [iot_thread] State: Active  Received: 203   Queued: 203   Processed: 203   Dropped: 0
696 16433 [iot_thread] WAHDAN : vRunOTAUpdateDemo : Inside the while loop
697 16433 [iot_thread] State: Active  Received: 203   Queued: 203   Processed: 203   Dropped: 0
698 16553 [iot_thread] WAHDAN : vRunOTAUpdateDemo : Inside the while loop
699 16553 [iot_thread] State: Active  Received: 203   Queued: 203   Processed: 203   Dropped: 0
700 16559 [OTA Task] [WARN ][MQTT][165590] (MQTT connection 0x3fff2190) Attempt to use closed connection.
701 16559 [OTA Task] [ERROR][MQTT][165590] (MQTT connection 0x3fff2190) New operation record cannot be created for a close
d connection
702 16559 [OTA Task] [prvPublishGetStreamMessage] Failed: $aws/things/HelloWorld_Publisher/streams/AFR_OTA-2374f622-5de3-4
930-8cbe-4e9d5d508b0d/get/cbor


Edit:
This also occurs with the normal OTA demo, this might be related to the QOS0 that we changes, I don’t know!
Edit2:
I have successful OTA update through greengrass.
I think this may be kind of race condition!
Shouldn’t this case be handled, like dropping this packet and re-request?

Edited by: embeddedx on Oct 2, 2019 1:15 AM

embeddedx wrote on October 08, 2019:

Hi,
Another issue is that the first connection to IOT core to retrieve the Greengrass core information works, but the second time fails with network_error and I need to restart the router(in my case I use the mobile as access point), and it doesn’t matter if I closed the connection or not, any ideas?