Deprecate config.reload.* as pipeline-level settings

[estolfo]

What does this PR do?

These changes handle the case when config.reload.* settings are set as pipeline override settings in pipeline.yml.
They aren't actually applied at the pipeline level so they are considered deprecated as pipeline override settings; they can only be set at the process level.

More specifically, the changes include:

• Don't store config.reload.* settings per pipeline, when set in pipeline.yml
• Remove config.reload.* from pipeline metrics
• Remove config.reload.* from /_node/pipeline(s) endpoint responses
• Remove references to config.reload.* from documentation of /_node/pipeline(s) (see preview below in comments)

Why is it important/What is the impact to the user?

See the related issue for more context. In summary, config.reload.* can be set at the pipeline level but the settings are only actually applied at the process level.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files (and/or docker env variables)
• I have added tests that prove my fix is effective or that my feature works

Author's Checklist

• [ ]

How to test this PR locally

Related issues

• Relates Settings config.reload.* should NOT allow per-pipeline overrides. #18262

Use cases

Screenshots

Logs

[github-actions]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[mergify]

This pull request does not have a backport label. Could you fix it @estolfo? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.
• If no backport is necessary, please add the backport-skip label

[estolfo]

Some questions/comments about these changes:

1. When a config.reload.* setting is attempted to be set via #merge_pipeline_settings, it warns that it’s deprecated and doesn’t actually set it.
2. Should the Settings#to_hash method also remove config.reload.* settings from the hash that is returned? If I'm not misreading the code, the Settings object is also used to track process-level settings, which could include config.reload.*. So Settings#to_hash shouldn't remove config.reload.*. The settings are not actually set as described in 1) so it shouldn't be necessarily to filter them from Settings#to_hash's response.
3. I didn't find any documentation showing that config.reload.* are pipeline-level options so there are no docs changes in this PR. I updated documentation for the _node/pipelines endpoint showing that the config.reload.* options were included in the response.
4. It's not yet clear to me whether removing these lines would be a breaking change. It seems that the two options are included in the response of the api endpoint and perhaps that would be a breaking change, depending on how that endpoint is used. As per the documentation, the config.reload.* settings are included in the response from /_node/pipeline(s) as pipeline level settings. So I've updated the endpoint to not include those settings in the response.

[estolfo]

preview /_node/pipelines documentation with the changes to logstash.yml

and /_node/pipelines/{pipeline_name}

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 50f64bf

History

• 💛 Build #3704 was flaky ff3e87d
• 💔 Build #3703 failed b10d80b
• 💚 Build #3701 succeeded fcf8eda

[kaisecheng]

Overall looks good. Docs and API spec are updated. When Logstash is started with auto reload "-r", the deprecation log by default is printed every 10 seconds. I think it is acceptable.

I have two minor suggestions.

[kaisecheng]

I have two suggestions

• quote the key with "`" rather than "("
• Can we add the pipeline name to the log? Sometimes users have hundreds of pipelines, so it will be great to point them to the right place.