Skip to content

Re-introduce GeoJSON conversion under a flag #1020

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 11 commits into
base: master
Choose a base branch
from
Open

Re-introduce GeoJSON conversion under a flag #1020

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
6a8fa69
Revert hotfix
jason-fox Apr 12, 2021
a8456b2
Re-add functionality under a flag
jason-fox Apr 12, 2021
3892f4f
Add autocastGeoJSON
jason-fox Apr 12, 2021
3d6ec0b
Revert debug flag
jason-fox Apr 12, 2021
f4795e8
Switch to parameter based approach
jason-fox Apr 14, 2021
d8506fd
Add an attribute flag
jason-fox Apr 14, 2021
5d82f8e
Update templates
jason-fox Apr 14, 2021
41333d5
Update descriptions
jason-fox Apr 14, 2021
9417381
autocastGeoJSON to autocast
jason-fox Apr 27, 2021
9d01da9
Merge branch 'master' into feature/geo-json2
jason-fox Jun 24, 2021
2370063
Merge branch 'master' into feature/geo-json2
jason-fox Sep 17, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
127 changes: 76 additions & 51 deletions doc/advanced-topics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,37 +1,36 @@
## Advanced Topics

* [Secured access to the Context Broker](#secured-access-to-the-context-broker)
* [GeoJSON support NGSI-LD only](#geojson-support-ngsi-ld-only)
* [Metadata support](#metadata-support)
* [NGSI LD data and metadata considerations](#ngsi-ld-data-and-metadata-considerations)
* [NGSI-LD Linked Data support](#ngsi-ld-linked-data-support)
* [Autoprovision configuration (autoprovision)](#autoprovision-configuration-autoprovision)
* [Explicitly defined attributes (explicitAttrs)](#explicitly-defined-attributes-explicitattrs)
* [Configuring operation to persist the data in Context Broker (appendMode)](#configuring-operation-to-persist-the-data-in-context-broker-appendmode)
* [Data mapping plugins](#data-mapping-plugins)
* [Development](#development)
* [Provided plugins](#provided-plugins)
* [Timestamp Compression plugin (compressTimestamp)](#timestamp-compression-plugin-compresstimestamp)
* [Attribute Alias plugin (attributeAlias)](#attribute-alias-plugin-attributealias)
* [Event plugin (addEvents)](#event-plugin-addevents)
* [Timestamp Processing Plugin (timestampProcess)](#timestamp-processing-plugin-timestampprocess)
* [Expression Translation plugin (expressionTransformation)](#expression-translation-plugin-expressiontransformation)
* [Multientity plugin (multiEntity)](#multientity-plugin-multientity)
* [Bidirectionality plugin (bidirectional)](#bidirectionality-plugin-bidirectional)
* [Autoprovision configuration (autoprovision)](#autoprovision-configuration-autoprovision)
* [Explicitly defined attributes (explicitAttrs)](#explicitly-defined-attributes-explicitattrs)
* [Configuring operation to persist the data in Context Broker (appendMode)](#configuring-operation-to-persist-the-data-in-context-broker-appendmode)
* [Old IoTAgent data migration](#old-iotagent-data-migration)

- [Secured access to the Context Broker](#secured-access-to-the-context-broker)
- [GeoJSON support NGSI-LD only](#geojson-support-ngsi-ld-only)
- [Metadata support](#metadata-support)
- [NGSI LD data and metadata considerations](#ngsi-ld-data-and-metadata-considerations)
- [NGSI-LD Linked Data support](#ngsi-ld-linked-data-support)
- [Autoprovision configuration (autoprovision)](#autoprovision-configuration-autoprovision)
- [Explicitly defined attributes (explicitAttrs)](#explicitly-defined-attributes-explicitattrs)
- [Configuring operation to persist the data in Context Broker (appendMode)](#configuring-operation-to-persist-the-data-in-context-broker-appendmode)
- [Data mapping plugins](#data-mapping-plugins)
- [Development](#development)
- [Provided plugins](#provided-plugins)
- [Timestamp Compression plugin (compressTimestamp)](#timestamp-compression-plugin-compresstimestamp)
- [Attribute Alias plugin (attributeAlias)](#attribute-alias-plugin-attributealias)
- [Event plugin (addEvents)](#event-plugin-addevents)
- [Timestamp Processing Plugin (timestampProcess)](#timestamp-processing-plugin-timestampprocess)
- [Expression Translation plugin (expressionTransformation)](#expression-translation-plugin-expressiontransformation)
- [Multientity plugin (multiEntity)](#multientity-plugin-multientity)
- [Bidirectionality plugin (bidirectional)](#bidirectionality-plugin-bidirectional)
- [Autoprovision configuration (autoprovision)](#autoprovision-configuration-autoprovision)
- [Explicitly defined attributes (explicitAttrs)](#explicitly-defined-attributes-explicitattrs)
- [Configuring operation to persist the data in Context Broker (appendMode)](#configuring-operation-to-persist-the-data-in-context-broker-appendmode)
- [Old IoTAgent data migration](#old-iotagent-data-migration)

### Secured access to the Context Broker

For access to instances of the Context Broker secured with a
[PEP Proxy](https://github.com/telefonicaid/fiware-orion-pep), an authentication mechanism based in Keystone Trust
tokens is provided. A trust token is a way of Keystone to allow an user delegates a role to another user for a subservice.
It is a long-term token that can be issued by any user to give another user permissions
to impersonate him with a given role in a given project (subservice). Such impersonation itself is in turn based on a short-term
access token.
tokens is provided. A trust token is a way of Keystone to allow an user delegates a role to another user for a
subservice. It is a long-term token that can be issued by any user to give another user permissions to impersonate him
with a given role in a given project (subservice). Such impersonation itself is in turn based on a short-term access
token.

For the authentication mechanisms to work, the `authentication` attribute in the configuration has to be fully
configured, and the `authentication.enabled` subattribute should have the value `true`.
Expand Down Expand Up @@ -73,13 +72,13 @@ Agent.

Complete info on Keystone trust tokens could be found at:

- [Trusts concept](https://docs.openstack.org/keystone/stein/user/trusts)
- [Trusts API](https://docs.openstack.org/keystone/stein/api_curl_examples.html#post-v3-os-trust-trusts)
- [Trusts concept](https://docs.openstack.org/keystone/stein/user/trusts)
- [Trusts API](https://docs.openstack.org/keystone/stein/api_curl_examples.html#post-v3-os-trust-trusts)

### GeoJSON support NGSI-LD only

The defined `type` of any GeoJSON attribute can be any set to any of the standard NGSI-v2 GeoJSON types - (e.g.
`geo:json`, `geo:point`). NGSI-LD formats such as `GeoProperty`, `Point` and `LineString` are also accepted `type`
The defined `type` of any GeoJSON attribute can be any set to any of the standard **NGSI-v2** GeoJSON types - (e.g.
`geo:json`, `geo:point`). **NGSI-LD** formats such as `GeoProperty`, `Point` and `LineString` are also accepted `type`
values. If the latitude and longitude are received as separate measures, the
[expression language](expressionLanguage.md) can be used to concatenate them.

Expand Down Expand Up @@ -132,6 +131,29 @@ accepted. Provided the `type` is provisioned correctly, the `value` may be defin
}
```

Because GeoJSON types (e.g. `Point`, `LineString` etc.) are native types in **NGSI-LD**, automatic GeoJSON conversion is
switched on for NGSI-LD by default.

With **NGSI-v2**, for backwards compatibility reasons, automatic GeoJSON conversion for types other than `geo:json` is
turned off by default. Add the `autocast` configuration to the attribute to enable GeoJSON conversion. Each GeoJSON
attribute can be provisioned as shown:

```json
Copy link
Member

@fgalan fgalan Apr 27, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This example in documentation is for a configuration group. However, the templates modified in this PR are only about devices:

imagen

The behavior should be as follow:

  • It can be enabled/disabled at configuration group level
  • It can be enabled/disabled at device level. If not defined at device level, the config group setting is used (and if not defined at config group level, then assume false by default)

{
"entity_type": "GPS",
"resource": "/iot/d",
"protocol": "PDI-IoTA-JSON",
..etc
"attributes": [
{
"name": "observationSpace",
"type": "Polygon",
"autocast" : "true"
}
]
}
```

### Metadata support

Both `attributes` and `static_attributes` may be supplied with metadata when provisioning an IoT Agent, so that the
Expand Down Expand Up @@ -179,7 +201,7 @@ following:
- Temporal Properties (e.g. `Datetime`, `Date` , `Time`)
- GeoJSON types (e.g `Point`, `LineString`, `Polygon`, `MultiPoint`, `MultiLineString`, `MultiPolygon`)

Most NGSI-LD attributes are sent to the Context Broker as _properties_. If a GeoJSON type or native JSON type is
Most **NGSI-LD** attributes are sent to the Context Broker as _properties_. If a GeoJSON type or native JSON type is
defined, the data will be converted to the appropriate type. Temporal properties should always be expressed in UTC,
using ISO 8601. This ISO 8601 conversion is applied automatically for the `observedAt` _property-of-a-property_ metadata
where present.
Expand Down Expand Up @@ -269,26 +291,28 @@ updated as shown:

### Autoprovision configuration (autoprovision)

By default, when a measure arrives to the IoTAgent, if the `device_id` does not match with an existing one, then, the IoTA
creates a new device and a new entity according to the group config. Defining the field `autoprovision` to `false`
when provisioning the device group, the IoTA to reject the measure at the southbound, allowing only to persist the
data to devices that are already provisioned. It makes no sense to use this field in device provisioning since it is
intended to avoid provisioning devices (and for it to be effective, it would have to be provisional).
By default, when a measure arrives to the IoTAgent, if the `device_id` does not match with an existing one, then, the
IoTA creates a new device and a new entity according to the group config. Defining the field `autoprovision` to `false`
when provisioning the device group, the IoTA to reject the measure at the southbound, allowing only to persist the data
to devices that are already provisioned. It makes no sense to use this field in device provisioning since it is intended
to avoid provisioning devices (and for it to be effective, it would have to be provisional).

### Explicitly defined attributes (explicitAttrs)

If a given measure element (object_id) is not defined in the mappings of the device or group provision, the measure is stored
in the Context Broker by adding a new attribute to the entity with the same name of the undefined measure element. By adding the
field `explicitAttrs` with `true` value to device or group provision, the IoTAgent rejects the measure elements that are not defined
in the mappings of device or group provision, persisting only the one defined in the mappings of the provision. If `explicitAttrs`
is provided both at device and group level, the device level takes precedence.
If a given measure element (object_id) is not defined in the mappings of the device or group provision, the measure is
stored in the Context Broker by adding a new attribute to the entity with the same name of the undefined measure
element. By adding the field `explicitAttrs` with `true` value to device or group provision, the IoTAgent rejects the
measure elements that are not defined in the mappings of device or group provision, persisting only the one defined in
the mappings of the provision. If `explicitAttrs` is provided both at device and group level, the device level takes
precedence.

### Configuring operation to persist the data in Context Broker (appendMode)

This is a flag that can be enabled by activating the parameter `appendMode` in the configuration file or by using the `IOTA_APPEND_MODE`
environment variable (more info [here](https://github.com/telefonicaid/iotagent-node-lib/blob/master/doc/installationguide.md)).
If this flag is activated, the update requests to the Context Broker will be performed always with APPEND type, instead of the
default UPDATE. This have implications in the use of attributes with Context Providers, so this flag should be used with care.
This is a flag that can be enabled by activating the parameter `appendMode` in the configuration file or by using the
`IOTA_APPEND_MODE` environment variable (more info
[here](https://github.com/telefonicaid/iotagent-node-lib/blob/master/doc/installationguide.md)). If this flag is
activated, the update requests to the Context Broker will be performed always with APPEND type, instead of the default
UPDATE. This have implications in the use of attributes with Context Providers, so this flag should be used with care.

### Data mapping plugins

Expand Down Expand Up @@ -355,8 +379,8 @@ iotaLib.addQueryMiddleware(iotaLib.dataPlugins.compressTimestamp.query);

This plugins change all the timestamp attributes found in the entity, and all the timestamp metadata found in any
attribute, from the basic complete calendar timestamp of the ISO8601 (e.g.: 20071103T131805) to the extended complete
calendar timestamp (e.g.: +002007-11-03T13:18). The middleware expects to receive the basic format in updates and return
it in queries (and viceversa, receive the extended one in queries and return it in updates).
calendar timestamp (e.g.: `+002007-11-03T13:18`). The middleware expects to receive the basic format in updates and
return it in queries (and viceversa, receive the extended one in queries and return it in updates).

##### Attribute Alias plugin (attributeAlias)

Expand All @@ -373,8 +397,8 @@ events in the IoT Agent with the configured type name will be marked as events.

##### Timestamp Processing Plugin (timestampProcess)

This plugin processes the entity attributes looking for a `TimeInstant` attribute. If one is found, for NGSIv2,
the plugin adds a `TimeInstant` attribute as metadata for every other attribute in the same request. With NGSI-LD, the
This plugin processes the entity attributes looking for a `TimeInstant` attribute. If one is found, for NGSIv2, the
plugin adds a `TimeInstant` attribute as metadata for every other attribute in the same request. With NGSI-LD, the
Standard `observedAt` property-of-a-property is used instead.

##### Expression Translation plugin (expressionTransformation)
Expand Down Expand Up @@ -438,7 +462,8 @@ When a device is provisioned with bidirectional attributes, the IoTAgent subscri
change notification for that attribute arrives to the IoTA, it applies the transformation defined in the device
provisioning payload to the notification, and calls the underlying notification handler with the transformed entity.

The following `attributes` section shows an example of the plugin configuration (using IOTA_AUTOCAST=false to avoid translation from geo:point to geo:json)
The following `attributes` section shows an example of the plugin configuration (using IOTA_AUTOCAST=false to avoid
translation from geo:point to geo:json)

```json
"attributes": [
Expand Down
Loading