sys/rest_client: a multi-transport rest-client for RIOT #17785
Conversation
The RSSI values reported by LoRa transceiver can be less than -127. Therefore, `int8_t` is not enough. This commit defines the RSSI of `netdev_lora_rx_info` as `int16_t` and adapt the drivers accordingly (sx126x, sx127x). (cherry picked from commit 9955a35)
(cherry picked from commit d6899f4)
…a/fix_rssi_val [treewide] lora: use int16_t for RSSI value [backport 2022.01]
(cherry picked from commit 1025341)
…est_pkg_relic_stacksize tests/pkg_relic: increase stacksize [backport 2022.01]
(cherry picked from commit 8b45ed8)
…tt_rtc_utils drivers/rtt_rtc: select rtc_utils [backport 2022.01]
(cherry picked from commit ec7926f)
…dhoc_test_netif_multi tests/pkg_edhoc: handle multiple interfaces [backport 2022.01]
(cherry picked from commit de50d55)
…timer64_flag sys/ztimer64/util.c: fix ztimer64_set_timeout_flag [backport 2022.01]
(cherry picked from commit ccf31eb)
…f767zi_fix_adc_pin_config boards/nucleo-f767zi: Fix adc pin config in periph_conf.h [backport 2022.01]
(cherry picked from commit 56d343d)
…/pr-release-notes-2022-01 release-notes.txt: add 2022.01 release notes [backport 2022.01]
VERSION: add 2022.01 version file
Without initializing async_cb to NULL it might be a value != NULL, which leads to sock->async_cb.gen() being called mistakenly in _netconn_cb.
I believe all of these are just backport commits that don't exist in master. My branch is currently not based on master but on the 2022.01 release. I'll change that later though. |
|
@chrysn I'm answering you here, because I think it is the better place to discuss.
If you are asking for an already available abstraction model then no. Using MQTT-based transport I had to abstract a few things that don't make sense for every implemented protocol. MQTT for instance requires that we connect to the broker first, so I introduced a |
|
Getting all these mentioned transports under the hood of one REST client sounds intriguing. Rebasing to current master will definitely help with the reviews .. currently, the code is polluted with all these unrelated backport changes, which complicates the reviewing process (at least for me). Out of curiosity: do you have additional resources (prosa, figures, etc.) that further conceptually describe how you plug all these transports together? Another question (because the title says rest-client, maybe just a terminology clash?): does this also work for CoAP servers (so, on IoT devices)? From my first glance, I'd say yes? |
We could get rid of the backport commits, but mind that even a rebase to master does remove most of the other commits, because this PR has a few unmerged dependencies which it inherits all its commits from. Namely: The important part here is 9f11b19
Not for the client itself, but for the overall architecture (with two different HTTP APIs as examples):
As you can see above my goal is to communicate with common HTTP REST APIs that are not IoT-aware. Calling it HTTP client is even worse in my opinion, because it does not speak HTTP (but it could!). Because of this and the fact that HTTP and REST are somewhat synonymous for this kind of API in the web (see https://github.com/confluentinc/kafka-rest) I decided to go for the name
Do you mean by using CoAP Observe? This is not planned right now. I believe that would require yet another component that would subscribe on the node and manage all data changes by doing HTTP calls. But this could be done on a higher hardware level like a Raspberry Pi. The focus here is on keeping the logic for doing requests on the microcontroller itself, working in client mode, and evaluate the feasibility thereof. The target is Class 1 microcontrollers and preliminary measurements are promising. Well... it really depends on the API we want to request, but RAM and ROM requirements are fine. |
|
This is really interesting. Never really thought about it, but it enables some interesting use-cases. Especially when it doesn't have such a big impact of performance or binary size. Would be interesting to compare an app 1 to 1. |
Using tinydtls on RIOT OS is not limited to GNRC as network stack. It is also working with e.g. lwIP, see: RIOT-OS#17552 Therefore the name WITH_RIOT_GNRC is misleading.
HendrikVE
force-pushed
the
draft/rest_client
branch
from
9f11b19 to
2b14bbf
Compare
Contribution description
I wanted to share my progress on developing a REST-client for RIOT in the form of a draft PR to get some early feedback while still working on it. This does not mean that it is not working yet. On the contrary, I have a test setup which already does a lot of the stuff we could want. More on that below.
What does the REST-client has to offer?
Supported right now are CoAP, CoAP with DTLS (CoAPs), MQTT and MQTT-SN
CoAP and CoAPs: confirmable and non-confirmable
MQTT and MQTT-SN: QoS 0, 1, 2
How is this supposed to work? MQTT can't make calls to an HTTP API on the internet!
While this statement is true for native MQTT, it does not apply to CoAP: RFC 7252 specifies how CoAP can be cross-proxied to HTTP. For MQTT-based transport I developed a custom protocol which is build on top of MQTT and an application which will, similarly to a CoAP-HTTP-Cross-Proxy, translate the request to an HTTP call and send back a response.
I also did some modifications on CoAP components. In
examples/rest_clientyou can find a demo application that will take a struct and serializes it into CBOR payload and sends it. So now you might think: "How is the HTTP API supposed to understand CBOR, they mostly talk JSON?" Well, I modified/extended californium and the CoAP code a bit. I introduced two new options.Target-Content-FormatandTarget-Accept. They work exactly like their counterpartsContent-FormatandAcceptin terms of standardized media types.RIOT => Californium: The modified californium cross-proxy will set
Target-AcceptasAcceptheader in the HTTP request and translates the payload fromContent-FormattoTarget-Content-Format.Californium <= HTTP API: When receiving the response the cross-proxy translates the payload from the format given in the HTTP
Content-Typeheader to the format given by theAcceptheader of the original CoAP request, thus making it possible to send CBOR payload from the IoT device and receiving standard JSON on a common HTTP API. This might be a nice addition to the standard. CoAP interested people might have an opinion about that?What software is needed for the backend?
The software stack I am using for the backend consists of the following components:
For MQTT: mosquitto + custom mqtt2http app
For MQTT-SN: paho.mqtt-sn.embedded-c + mosquitto + custom mqtt2http app
For CoAP(s): californium modified cross proxy example
And on top of all that use a HTTP API we would like to request.
What about security?
CoAPs is supported. The package
wolfMQTT, which is used for MQTT-based transport, has TLS support built in, but my port to RIOT (#15969) does not yet support it and the code for MQTT-SN does not have any encryption built in yet at all. I asked the developers if anything is planned right now in this regard. They said it is on the list for future feature additions, but they did not mention a possible time for an implementation.In one of my use cases for example the RIOT application talks to californium via DTLS and Californium is set up to talk to the Kafka REST proxy via mTLS.
The custom mqtt2http application does similar things.
Are there limitations?
We still need to keep in mind that we are operating on constrained devices. It is illusory to think that we could simply call every HTTP API out there. Some APIs just require too big request payloads or they produce too big responses. I tried it with the Kubernetes API. Responses mostly are way too big to even bother trying to work with it in most cases. Doing a request can be fine though.
What would be a use case?
In my case, I want to publish data and process it in an application which is following event-driven architecture. A popular software for this purpose is Apache Kafka. Kafka by itself is only accessible by different client libraries available for different programming languages. But Confluent also offers kafka-rest which will expose a Kafka instance via HTTP API.
An alternative approach to that specific use case could be to use e.g. Eclipse Hono, which can also be used with Kafka. But the power of the REST-client is that we, respecting the limitations, can call any HTTP API and therefore we are able to cover way more potential use cases.
What does it cost me to use it?
If you are interested in numbers you can ask me for it, I can provide (temporary valid) data and even nice graphs for build sizes and bytes sent over the network. But as an example: For my
nucleo-f207zgthe examplegcoapneeds about 59 kB ROM, while my test application needs 47 kB ROM for CoAP based transport. Mind that this comparison is not fair, because the test application does not need to parse user input. But it shows that it is possible to run the application on class 1 devices with ~100kB ROM.Try it yourself
I created a project using
docker-composewhich will set up everything you need (You don't want to do it manually, really). You can find it here if you want to try it out. Simply rundocker-compose up --build. Then runexamples/rest_cliente.g. withBOARD=nucleo-f207zg LWIP_IPV4=0 LWIP_IPV6=1 TRANSPORT_COAP_SECURE=1 QOS=0 make flash term.P.S. Please ignore all commits except the topmost one. All other commits are only dependencies. I haven't cleaned up yet as many things are yet to change so this would be a waste of time ;)