Adding NGSI-LD support

[jason-fox]

It is necessary to add support for NGSI-LD to all IoT Agents. This is a large amount of work and needs to be split according. This issue is to discuss how the code can be incorporated and link the relevant PRs.

The main issue here is that there is a lot of work to review, so I've tried to split it into smaller PRs

1. Add NGSIv2 metadata support to attributeAlias plugin #839 - NGSI-LD relies very heavily on metadata support (a.k.a. Properties of Properties) - whilst adding the code it was noticed that aliasing didn't work nicely with metadata. Effectively this is a bug fix.
2. Prepend NGSI-v1 to the v1 tests, NGSI-v2 to the v2 tests #840 - This is a simple Grep to be able to distinguish between NGSI-v2 and NGSI-LD tests. Very minor change across lots of files - I split it out to reduce the size of the PR
3. Basic NGSI-LD active measures support #841 - This is the main body of work for measures support
4. Add GeoJSON and DateTime, unitCode and observedAt NGSI-LD support #843 - Temporal and GeoProperty support
5. Add NGSI-LD multi-measures support #847 - Add NGSI-LD Multimeasures support
6. Add NGSI-LD basic Command support #848 - Add NGSI-LD Command support
7. Update NGSI support to use ES6 syntax. #850 - Update the NGSI interactions to use ES-6

Dependencies

• Basic NGSI-LD active measures support #841 is dependent upon Add NGSIv2 metadata support to attributeAlias plugin #839 ... and this is just the first island of stability - there is more to code.
• Add GeoJSON and DateTime, unitCode and observedAt NGSI-LD support #843 is dependent upon Basic NGSI-LD active measures support #841
• Add NGSI-LD multi-measures support #847 is dependent upon Add GeoJSON and DateTime, unitCode and observedAt NGSI-LD support #843
• Add NGSI-LD basic Command support #848 is dependent upon Add NGSI-LD multi-measures support #847 - this should now be basically usable (assuming the Context Broker interfaces are stable)
• Update NGSI support to use ES6 syntax. #850 is just a simple automatic update of the files within /services/devices/, /services/ngsi/ and /services/northBound to properly use let and const - May as well use this opportunity.

[jason-fox]

Things which haven't been done yet (which I'd prefer to raise as separate smaller PRs)

• GeoProperty support Add GeoJSON and DateTime, unitCode and observedAt NGSI-LD support #843
• Proper unitCode metadata Add GeoJSON and DateTime, unitCode and observedAt NGSI-LD support #843
• Map TimeInstant onto observedAt metadata Add GeoJSON and DateTime, unitCode and observedAt NGSI-LD support #843
• Multimeasure Support Add NGSI-LD multi-measures support #847
• Actuator Support (commands and results) Add NGSI-LD basic Command support #848
  • relies on Context Forwarding PATCH which is still in progress.
• Polling Commands
  • tests need updating - functionality should work.
• Lazy Attributes - relies on Context Forwarding GET which is still in progress.
• Documentation

The thing is that this is still experimental, but would be massive if it is all collected into one PR.

Basically need to discuss how this can be reviewed, tested and added into the codebase in a timely manner.

[jason-fox]

The following repo provides a test harness for the NGSI-LD support.

https://github.com/jason-fox/tutorials.IoT-Agent/tree/test

To run the test harness just type:

./services start

Load the Postman collection into Postman and run interactively.

Alternatively use the following cUrl commands:

Provision Service Group

curl -L -X POST 'http://localhost:4041/iot/services' \
-H 'fiware-service: openiot' \
-H 'fiware-servicepath: /' \
-H 'Content-Type: application/json' \
--data-raw '{
 "services": [
   {
     "apikey":      "4jggokgpepnvsb2uv4s40d59ov",
     "cbroker":     "http://orion:1026",
     "entity_type": "Motion",
     "resource":    "/iot/d",
     "trust": "08ca25ba8ea0f322f50244c413fce2e32feed43d"
   }
 ]
}'

Provision Device

curl -L -X POST 'http://localhost:4041/iot/devices' \
-H 'fiware-service: openiot' \
-H 'fiware-servicepath: /' \
-H 'Content-Type: application/json' \
--data-raw '{
 "devices": [
   {
     "device_id":   "motion001",
     "entity_name": "urn:ngsi-ld:Motion:001",
     "entity_type": "Motion",
     "timezone":    "Europe/Berlin",
     "attributes": [
       { "object_id": "c", "name":"count", "type":"Integer"}
      ],
      "static_attributes": [
         {"name":"refStore", "type": "Relationship","value": "urn:ngsi-ld:Store:001"}
      ]
   }
 ]
}
'

Dummy Device Measure

curl -L -X POST 'http://localhost:7896/iot/d?k=4jggokgpepnvsb2uv4s40d59ov&i=motion001' \
-H 'Content-Type: text/plain' \
--data-raw 'c|1'

Read Measure as NSGI-LD Data

curl -L -X GET 'http://localhost:1026/ngsi-ld/v1/entities/urn:ngsi-ld:Motion:001' \
-H 'fiware-service: openiot' \
-H 'fiware-servicepath: /' \
-H 'Link: <https://fiware.github.io/tutorials.Step-by-Step/tutorials-context.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"'

[fgalan]

As a general comment, I think the idea of working the functionality in several branches (each one branching from the previous), creating and merging PRs one by one is a good idea.

PRs #839 and #840 seems to be small and straighforward. I have provided some feedback on them and I think once addresses they can be merged (I'll check with @AlvaroVega for a second opinion, anyway).

With regards to #841 and #843 (and all the other that may come in that line) I think it will be a good idea to create a branch from master ("prelanding-ngsi-ld" for instance) then merge the PRs on that branch and run CI on that branch (it's simpler for us run CI testing when the branch lives in our repository). We have used this approach other times for big works (e.g. NGIv2 implementation).

If you agree, I'll take care of creating that "prelanding-ngsi-ld" (I'd do so after de #839 and #840 merges).

[fgalan]

If you agree, I'll take care of creating that "prelanding-ngsi-ld" (I'd do so after de #839 and #840 merges).

PR #839 and PR #840 have been just merged. Should I create "prelanding-ngsi-ld" "feature/842_ngsi_ld" branch from master and change to it the base branch for PR #841 and RB #843?

[jason-fox]

That is fine by me .

[fgalan]

Branch has been created (at the end the name is: feature/842_ngsi_ld and PR #841and PR #843 has been changed to use it as base branch.

[fgalan]

Basic implementation should be completed in PR #849, which now steps to CI e2e phase before merging.

However, we would like to propose a modification for the future (with the category of "technical debt" in order not blocking the merge of PR #849).

Instead of having a global setting for the NGSI version (v1, v2 or LD) the parameter can be specified at three different levels:

1. Global setting (in config.js or equivalent env var)
2. Per configuration group
3. Per device

being the last one preferred over the formers. I mean, device takes precedence over configuration group and configuration group takes preference over global setting.

2 and 3 will need an slight modification of the provisioning API to include the new field in the provision API (suggestion: "ngsiVersion": "v1|v2|ld").

I'd like to remark this pattern has been already used for other configurable aspects in IOTA. For instance, timestamp setting that can be defined globally, per configuration group and per device). A similar approach is used in the JEXL contribution and explicitAttrs contribution.

Making NGSI version selectable at global/configuration/device level is important given that, as you know, NGSIv2-NGSILD interoperability is a main requirement and this is a way in which we could have the same IOTA instance running both versions of the API at the same time, in a very configurable and flexible manner.

[jason-fox]

Whereas this does make sense for attributes or changing the payload, I don't think that a per-device switch pushing two data formats into a single context broker makes much sense for NGSI-LD/NGSI v2. There are multiple NGSI-LD brokers (five or six at last count) - only one also has an NGSI-v2 interface, and that is missing a lot of recent changes. I won't implement something like this with the current round of development - it is too much of a moving target - best to add it as a backlog issue to technical debt if you feel it is important.

It should be remembered that combined NGSI v1/v2 into one broker was not added when the NGSI v1/v2 messaging was added to the IoT Agents - and NGSI v1/v2/LD follows the same paradigm - this makes messaging a bit different to the other attribute switch use cases you have cited.

Personally for combined v2/LD, I would use two separate IoT Agent instances and connect them to separate NGSI v2 and NGSI-LD brokers. This keeps my backing databases clean, and would allow me to switch in and out other NGSI-LD brokers as necessary. In the case that I need to duplicate context data my preferred solution/ workaround for now would be to shadow device data using subscriptions.

[jason-fox]

Merge completed.

[fgalan]

Two items in #842 (comment) are still unchecked.

Maybe some specif issues should be created for them?