Scheduling parameterisation: prepare for more than one scheduler

[nhoening]

I'm thinking about making the triggering of scheduling clearer to use.
Problem now: we don't discuss the parameters well and the way in which the trigger endpoint handles them doesn't translate to a situation when we'll have multiple schedulers.

I believe right now for that to be simple and clear, we should:

• Make two parameter groups in the trigger request: I'll call them "flex-context" and "portfolio-context".
• The former describes the sensor's flexibility (for now it contains what I now called "storage-specs").
• The latter describes sensors which also need to be taken into account.

Let's review what stakeholders want:

• As a FlexMeasures user, I want to trigger a schedule for my configured assets, and it should be straightforward. This means, we need: Obvious documentation for parameterisation, good error messaging (preferably from the trigger endpoint, not the job), look job failures up on the asset page, ...), FM attempts to guess the scheduler if you did not say which to use.
• As the author of a (custom) FlexMeasures scheduler, I'd like for parameter values to be pre-checked before my code receives them. I'd also like to define my own parameters. We already use Marshmallow in the trigger endpoint for parameters, but we need to make most optional, as not all schedulers need all flex-contexts. I have no idea how schedule authors could add their own schemas (own flex-contexts) in their plugin and let the trigger endpoint use them. For now, I propose that we govern flex-contexts, and schedules have to deal with what we offer. Open to other ideas.

Here is an example request:

            {
                "start": "2015-06-02T10:00:00+00:00",
                "flex-context": {
                    "storage": {
                        "soc-at-start": 12.1,
                        "soc-unit": "kWh",
                        "soc-targets": [
                            {
                                "value": 25,
                                "datetime": "2015-06-02T16:00:00+00:00"
                            }
                        ],
                        "soc-min": 10,
                        "soc-max": 25,
                        "roundtrip-efficiency": 0.98,
                    }
                },
                "portfolio-context":{
                    "consumption-price-sensor": 9,
                    "production-price-sensor": 10,
                    "inflexible-device-sensors": [13, 14, 15]
                }
            }

My next PR will make the Marshmallow adjustments and the deprecation. And add documentation.

Other issues for the future:

• FM should could check if a scheduler is applicable before a job is made (thus returning an informative error in the trigger endpoint).
• Next to "storage", we can add "shiftable" flex-context soon, when we integrate E-mission's solver.
• Asset pages could try to see if there are failed jobs in the queue with this asset's sensors and thus help a user look them up (and see error messages).

[Flix6x]

We had a productive discussion on this API design, in which we formulated the following wishlist:

• The design for the scheduling trigger endpoint for a single flexible asset translates straightforwardly to the upcoming endpoint for scheduling multiple flexible assets together.
• Field names are easily grasped by API users.
• Explicit fields (such as "flex-type": "storage", "flex-specs": <storage_specs>, and "sensor": 37, "specs": <specs>) are preferred over implicit fields (such as "storage-specs": <storage_specs> and 37: <specs>). In other words, data is defined in the dictionary's values, rather than in its keys.
• Less deeply nested specs are preferable.
• The design is preferably straightforward to implement with Marshmallow.
• Users can leave out fields to let the server fall back to sensible defaults.
• It should be straightforward to extend the specs, like with a new flex type, new flex constraints, or price conditions per device.

[Flix6x]

For multi-device scheduling:

{
    "start": "2015-06-02T10:00:00+00:00",
    "sensor": [
        {
            "id": 37,
            "flex-context": {
                "soc-at-start": 12.1,
                "soc-unit": "kWh",
                "soc-targets": [
                    {
                        "value": 25,
                        "datetime": "2015-06-02T16:00:00+00:00"
                    }
                ],
                "soc-min": 10,
                "soc-max": 25,
                "roundtrip-efficiency": 0.98
            }
        },
        <next flex device>
    ],
    "portfolio-context":{
        "consumption-price-sensor": 9,
        "production-price-sensor": 10,
        "inflexible-device-sensors": [13, 14, 15]
    }
}

For single-device scheduling, the existing scheduling trigger endpoint can be used. Because that url already contains the sensor id, the flex-context can move up one level, which gives:

{
    "start": "2015-06-02T10:00:00+00:00",
    "flex-context": {
        "soc-at-start": 12.1,
        "soc-unit": "kWh",
        "soc-targets": [
            {
                "value": 25,
                "datetime": "2015-06-02T16:00:00+00:00"
            }
        ],
        "soc-min": 10,
        "soc-max": 25,
        "roundtrip-efficiency": 0.98
    },
    "portfolio-context":{
        "consumption-price-sensor": 9,
        "production-price-sensor": 10,
        "inflexible-device-sensors": [13, 14, 15]
    }
}

[nhoening]

Note: We left out "flex-type" here, as well as giving the flex-context a type name (e.g. "schedule"), as we believe that this should be the order of decisions:

1. The scheduler is chosen, either by the asset type (or other asset/portfolio circumstances) or by the user (setting a custom scheduling algorithm)
2. The scheduler determines which flex-context schema is relevant (and should be used to validate parameters against). This should happen with a class variable.

This way, we believe we combine strong in-built support (for ease-of-use, potentially you can get started with no knowledge, error messages guide you) with extensibility (custom schedulers can determine the flex-context they require).

Finally, we should make flex-contexts which

• cover the main flex types (e.g. storage, shiftable, buffer)
• are well-documented in the Docs
• are maybe even backed-up by a standard (e.g. Flex types in EFDM)

[victorgarcia98]

In order to expose the full potential of the device_scheduler, @Flix6x and I had a session to discuss how to describe the scheduling message for multiple devices.

To exemplify how it would look, let us present the outcome of the discussion:

URI: sensors/0/schedules/trigger

Body

{
	"flex-model" : [
			{"sensor" : 0, "power-capacity" : "1MW"},
			{"sensor" : 1, "power-capacity" : "10kW", "soc-maxima" : []},
			{"sensor" : 2, "soc-at-start" : 10.0},
	],
	"flex-context" : {
		"topology" : [
				{"sensor" : 1, "topology" : [
					{"sensor" : 3, "consumption-price-sensor" : 10},
					{"sensor" : 4, "production-price-sensor" : 10},
					{"sensor" : 14, "production-price-sensor" : 10},					            {"sensor" : 15, "production-price-sensor" : 10}
				]},
				{"sensor" : 2},
                {"sensor" : 15, "production-price-sensor" : 10}
		],
		"consumption-price-sensor" : 10,
		"inflexible-device-sensors" : [14, 15]
	}
}

General notes:

• The top level of the message remains the same as before(flex-model and flex-context as the unique keys). This will facilitate backwards compatibility.
• The flex context includes a new key: topology. This field describes a tree of devices which helps us to introduce intermediate power constraints and different price sensors for each of the groups.
  • Each of the terminal nodes describe either a flexible or inflexible device. The inflexible devices must be defined in the inflexible-device-sensors field.
  • If the topology field is defined in the element, it defines a group of devices which, in turn, could define sub-groups.
• In favor of keeping backwards compatibility, the flex model can be of one of the following types:
  • Dict: current description of 1 storage device with the starting soc, soc maxima and minima, etc.
  • List: describe the time-varying parameters such as the starting soc, soc maxima, minima, etc... for each of the devices or groups defined in topology.

How expressive is this schema?

Some constraints that we can define:

• Different prices per group of devices
• Different power constraints for each group of devices.

Some constraints that we can't establish:

• Impose constraints on the aggregate effect of devices that don't belong to the same group. So let's say we have two groups (A,B) and (C,D) and we want to make power(A) + power(B) < max_power. This could be useful in case of delivering ancillary services where two or more batteries in different sites are part of the same group.

This schema also aligns very well with the DataGenerator description.

Follow-up tasks

(feel free to add to this list)

• Decide whether to create a new scheduler (l'll make reference to this as the MultiDeviceScheduler) or reuse the StorageScheduler.
  • On one side, the MultiDeviceSchedulerwill allow to schedule a single storage device so it will cover the existing scope of the StorageScheduler. On the other side, we are relying on the type to trigger the fallback mechanism. Moreover, the new scheduler might come with bugs and we might want to battle test it for a while (e.g. with internal projects) and phase out the StorageScheduler.
• Decide which features will the first version of the MultiDeviceScheduler implement:
  • Different prices per group
  • Intermediate power constraints
  • Fixing binary constraints for elements that we know that will be only consume or produce (e.g. 1 directional chargepoint)
• The MultiDeviceScheduler will return a list of output BeliefDataFrames and the existing infra should support saving the resulting beliefs. This goes inline with what the DataGenerator does.
• Create a new tutorial where the user schedules multiple devices. (Given that the descriptions can we quite long, should we use the FlexMeasures client?)
• Plan how to call the scheduler via the CLI.

What do you think @Flix6x and @nhoening?

[nhoening]

Should this not be a new discussion?

This one contains a lot of outdated ideas and has been locked (weird that comments can still be added)

[nhoening]

Independent of the location of this discussion, I already place a few questions:

1. For multiple sensors, ideally we could place such an endpoint under assets/<id>/schedules/trigger right? That's where I would expect a nested flex-model and flex-context
2. We are working on consumption-price-sensors-per-device and production-price-sensors-per-device. To be clear, would your suggested topology make that PR redundant? (we might re-use it partly here)? It would be a more generic solution, of course.
3. About the actual solver: The most crucial feature we enable here is the ability to schedule multiple assets within one optimization. Why do we have to decide which features to add in a first implementation? That said, I don't know what "Intermediate power constraints" are, and the third idea seems the most relevant to real cases we see in the near future, although I don't know what the positive impact of "Fixing binary constraints" is - can you name them?

[nhoening]

I'm tending to use the term "flex-model" over "flex-context".

[Flix6x]

Perhaps there is a better term than optimization-context, too. Actually, maybe that should be flex-context instead.

[nhoening]

Where do we use optimization-context?

[nhoening]

Or do you mean portfolio-context?

[Flix6x]

Right. Sorry, I had misremembered. I withdraw my comment.

[nhoening]

Actually, flex-context might be a good compagnion to flex-model

[Flix6x]

Linking #537 (comment) so we can continue the discussion here and resolve the PR comment.

Terminology: I'm just realizing you use "flex configuration" as "flex model" + "flex context". It is convenient to have such a definition. I actually missed something like that in the "Flex model" documentation section in notation.rst (first sentence). While wondering whether this definition should be just copied there, I'm having the feeling that there may be a better alternative, for example: "flex state" = "flex model" + "flex context". Here, "state" describes the information available to the agent (reinforcement learning).