[enhancement]: Add to our documentation the per protection-artifacts rules

[111andre111]

Description

In this blog https://www.elastic.co/blog/continued-leadership-in-open-and-transparent-security
we refer to our as well public protection-artifacts repo
https://github.com/elastic/protections-artifacts

Now it would be very good if we found a way to document these rules.

Similar like we do already for the prebuilt detection rules in Kibana:
https://www.elastic.co/guide/en/security/current/prebuilt-rules.html

Nonetheless that Kibana is not aware of these artifact rules I still would place these rule docs into the security docs:
https://www.elastic.co/guide/en/security/current

Maybe placed under Endpoint Management as a sub structure:
https://www.elastic.co/guide/en/security/current/sec-manage-intro.html

Related links / assets

No response

Which documentation set needs improvement?

ESS and serverless

Software version

8.12.0

Collaborators

PM: @joe-desimone

Timeline / deliverables

Middle term

[Mikaayenson]

👋 @benironside

Background

The prebuilt content is generated from our detection-rules by the repo by the TRaDE team. Currently our process uses automation (that will change) that pushes doc updates to another repo managed by @joepeeples and @jmikell821. This process will change and is essentially deprecated in favor of pushing to docs.elastic.co. We tried to migrate to the new site last April 2023 but the team (@scottybollinger @glitteringkatie and @KOTungseth) was not ready for us to migrate at the time.

Re: endpoint-rules, if we want to add docs for endpoint-rules (externally protections-artifacts/behavior), we should support the docs.elastic.co implementation to prevent rework.

The other note is that not all protections-artifacts are managed by the TRaDE team.

• https://github.com/elastic/protections-artifacts/tree/main/ransomware cc. ayfaouzi
• https://github.com/elastic/protections-artifacts/tree/main/yara cc. @magermark

Where do we go from here?

Speaking for TRaDE only:

1. We still need to migrate to the new repo with the help of @colleenmcginnis prior to any endpoint doc support. This work is tracked here https://github.com/elastic/ia-trade-team/issues/120.
2. Requirements: We need doc designs for how they should look and ideally sample mdx files. Not all docs have to be updated (e.g. landing page, etc.). Some have to be simply updated, and some docs have to be regenerated altogether. It would be great if someone could provide these for us. On the prebuilt side, @approksiu did a lot of the designs and rework for the docs, and we also would need some one to provide this for us (ideally in the form of mdx examples). I imagine all three teams will need this.

It may be good to check in with Colleen on close the team is to other teams being able to migrate to docs.elastic.co to gauge timing and prioritization.

Hope this helps!

[Mikaayenson]

I spoke with @benironside this week, and informed him of the changes in plans. 9.0+ Security docs are maintained by the security-docs (@jmikell821) and docs-engineering teams (@Mpdreamz).

I would recommend reaching out to them and ask to add something like elastic/detection-rules#4507 to the protections-artifacts repo https://github.com/elastic/protections-artifacts/tree/main/ to pickup the different protections in the new docs site.

[benironside]

I synced with @Mpdreamz on the Docs Engineering team, and he confirmed we could automatically generate documentation for the protections artifacts, as we do for detection rules. The way detection rules appear in docs can be seen here — the source files for this content are here. (We will eventually implement filtering for the detection rules page, and could do the same for the protections artifacts.)

• Since there are different owners and formats for the Behavior, Ransomware, and YARA artifacts, we likely will need a different approach for each. The Behavior rules consist of multiple files, with metadata and a description field, and the YARA definitions are also multiple files, with metadata but with no description field — so it seems each of these will have to appear a bit differently in docs.

Questions for the owners of the protections artifacts:

• How does it make sense to organize protections artifacts documentation?

  • We should have separate pages for Behavior and YARA rules, and it seems we should group each of those pages by "Cross platform", "Linux", "MacOS", and "Windows", but within each of those sections, are there any sub-categorizations that would make sense and help keep things organized?

• For Ayoub, how should the Ransomware content be exposed to users in docs, if at all? It seems pretty different from the other two pages in that it's not multiple rules / signatures but instead contains detection logic. Does it make sense to create documentation for this?

• For Mark and Mika, what do you think we should show users about each artifact? Which rule / artifact metadata fields should we display? Are there certain parts of each file that we should put under different headings, or highlight in different ways? Are there any notes, warnings, or best practices we should inform users of when they are reviewing this content? These may not be the right questions, but your input here is key since you're the experts on how these artifacts are used and what users need to know.

CC: @magermark @Mikaayenson @ayfaouzi

[Mikaayenson]

👋 No need to provide docs for Ransomware. However, shipping the Behavior rules organized with the top level folders still makes sense. In the future, it may make sense to organize by MITRE ATT&CK information, but for the first iteration, the top folders will work (macOS, linux, windows, cross-platform).

Regarding fields, all the fields available make sense to ship since they are already a lightweight version of the original artifacts.

For behavior rule organization, you can follow a similar format as best as possible with the fields available for the endpoint detection rules for consistency (example).

• internal can most likely be moved along side the other rule fields.

The logic that pulls the shipped artifacts, and updates these public protections-artifacts is here https://github.com/elastic/protections-cloud/blob/main/tools/artifacts/openness/update_open_artifact.py , so you can see more details about how they are pulled / parsed.

[joe-desimone]

We should have separate pages for Behavior and YARA rules, and it seems we should group each of those pages by "Cross platform", "Linux", "MacOS", and "Windows", but within each of those sections, are there any sub-categorizations that would make sense and help keep things organized?

I think the platform breakdown you outline above works for now (small chance to revisit this in the future).

For Ayoub, how should the Ransomware content be exposed to users in docs, if at all? It seems pretty different from the other two pages in that it's not multiple rules / signatures but instead contains detection logic. Does it make sense to create documentation for this?

We have decided not to expose the ransomware content at all in the documentation. Not useful for this purpose.

For Mark and Mika, what do you think we should show users about each artifact? Which rule / artifact metadata fields should we display?

Mika plans to provide a response for metadata fields to display for endpoint rules. For yara rules, I think the following will work:
rule id, license, creation_date, last_modified date, os, threat_name, reference, and reference_sample. The references won't always exist in every rule, but the other fields should always exist.