Update PPL Command Documentation #4562

ritvibhatt · 2025-10-15T01:24:18Z

This PR standardizes the structure and content of PPL command documentation files to improve consistency and user experience.

Changes Made

Documentation Structure Standardization

Implemented consistent section ordering across all PPL command files:

Description - Clear explanation of command functionality
Syntax - Command syntax with parameter details
Optional sections - Behavior notes, configuration details, or usage guidance as needed
Examples - Practical usage examples with expected output
Limitations - Known constraints or considerations

Content Cleanup and Modernization

Removed version information - Eliminated outdated version references (e.g., "Version:
3.3.0") to keep documentation current and reduce maintenance overhead
Extracted aggregation functions - Moved aggregation function documentation to dedicated
functions file (/functions/aggregations.rst) for better organization

Related Issues

Resolves #[Issue number to be closed when this PR is merged]
#4220

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
New PPL command checklist all confirmed.
API changes companion pull request created.
Commits are signed per the DCO using --signoff or -s.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Ritvi Bhatt <[email protected]>

Signed-off-by: ritvibhatt <[email protected]>

Signed-off-by: Ritvi Bhatt <[email protected]>

RyanL1997

Hi @ritvibhatt , thanks for the change. And I can signoff for rex, regex, and parse. And I just left some minor comments.

RyanL1997 · 2025-10-15T23:46:55Z

docs/user/ppl/cmd/rex.rst

 ================================== ============ ============


 Limitations


I saw that in the regex.rst, there are some formatting changes in the "Limitation" section, and I was wondering do we need to apply the same here?

I think the format is good, I removed the first sentence in limitations but let me know if you think format can be changed to match regex better

LantaoJin · 2025-10-16T03:10:15Z

docs/user/ppl/cmd/append.rst

 ============
-| Using ``append`` command to append the result of a sub-search and attach it as additional rows to the bottom of the input search results (The main search).
+| The ``append`` command appends the result of a sub-search and attaches it as additional rows to the bottom of the input search results (The main search).
 The command aligns columns with the same field names and types. For different column fields between the main search and sub-search, NULL values are filled in the respective rows.


Need a | before head of this line.

LantaoJin · 2025-10-16T03:15:17Z

docs/user/ppl/cmd/appendcol.rst

-Version
-=======
-3.1.0
+| The ``appendcol`` command appends the result of a sub-search and attaches it alongside with the input search results (The main search).


no need to keep the | any more if there is single line in description. | means starting from a new line

Removed all the | where there was only a single line thanks!

LantaoJin · 2025-10-16T03:15:24Z

docs/user/ppl/cmd/dedup.rst

 ============
-| Using ``dedup`` command to remove identical document defined by field from the search result.
-
+| The ``dedup`` command removes duplicate documents defined by specified fields from the search result.


LantaoJin · 2025-10-16T03:19:31Z

docs/user/ppl/cmd/describe.rst

 Syntax
 ============
-describe <dataSource>.<schema>.<tablename>
+describe [dataSource].[schema].<tablename>


should be describe [dataSource.][schema.]<tablename>

LantaoJin · 2025-10-16T03:23:56Z

docs/user/ppl/cmd/expand.rst

+1. It generates one row per element in the specified array field.
+2. The specified array field is converted into individual rows.
+3. If an alias is provided, the expanded values appear under the alias instead of the original field name.
+4. If the specified field is an empty array, the row is retained with the expanded field set to null.


why not bullet mark here?

You're right bullet points make more sense here, updated

LantaoJin · 2025-10-16T03:27:31Z

docs/user/ppl/cmd/fillnull.rst

+======

 fillnull with <replacement> [in <field-list>]
-


if you delete new line, please add | at the beginning of each line.

LantaoJin · 2025-10-16T03:27:33Z

docs/user/ppl/cmd/fillnull.rst

 fillnull with <replacement> [in <field-list>]
-
 fillnull using <field> = <replacement> [, <field> = <replacement>]
-


LantaoJin · 2025-10-16T03:27:37Z

docs/user/ppl/cmd/fillnull.rst


 Example 1: Replace null values with a specified value on one field
-------------------------------------------------------------------
+===================================================================


delete last =

Updated all of the = and - underlines to match

LantaoJin · 2025-10-16T03:27:43Z

docs/user/ppl/cmd/fillnull.rst


 Example 2: Replace null values with a specified value on multiple fields
-------------------------------------------------------------------------
+=========================================================================


ditto and following

LantaoJin · 2025-10-16T03:33:37Z

docs/user/ppl/cmd/join.rst


-For extended syntax (join with field list) in 3.3.0, when duplicate-named fields in output results are deduplicated, the fields in output determined by the value of 'overwrite' option.
-
-Since 3.3.0, join types ``inner``, ``left``, ``outer`` (alias of ``left``), ``semi`` and ``anti`` are supported by default. ``right``, ``full``, ``cross`` are performance sensitive join types which are disabled by default. Set config ``plugins.calcite.all_join_types.allowed = true`` to enable.


this information (how to enable full capability of join types) totally lost

That was mistakenly deleted thanks for catching that! Added back

LantaoJin · 2025-10-16T06:23:11Z

docs/user/ppl/cmd/lookup.rst

-Example 3: no inputField
-========================
+Example 3: No inputField specified
+====================================


delete last two =

LantaoJin · 2025-10-16T06:23:27Z

docs/user/ppl/cmd/lookup.rst

-Example 4: outputField as a new field
-=====================================
+Example 4: OutputField as a new field
+======================================


delete last one =

LantaoJin · 2025-10-16T06:25:37Z

docs/user/ppl/cmd/patterns.rst

-Change default pattern method
-============
+Change the default pattern method
+=============================


LantaoJin · 2025-10-16T06:25:48Z

docs/user/ppl/cmd/patterns.rst

    +-----------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------+

 Simple Pattern Example 3: Extract log patterns with custom regex pattern
 =========================================================


add more =

LantaoJin · 2025-10-16T06:26:31Z

docs/user/ppl/cmd/patterns.rst

 Configuration
 -------------
-With Calcite specific option ``show_numbered_token`` enabled, the output can detect numbered variable tokens from the pattern field.
+With  option ``show_numbered_token`` enabled, the output can detect numbered variable tokens from the pattern field.


delete one space between With and option

LantaoJin · 2025-10-16T06:29:49Z

docs/user/ppl/cmd/sort.rst


-Example 8: Sort with specifying field type
+Example 9: Sort with specifying field type
 ==================================


LantaoJin · 2025-10-16T06:30:05Z

docs/user/ppl/cmd/spath.rst

    +----------+---+

 Example 2: Lists & Nesting
 ============================


= problem

LantaoJin · 2025-10-16T06:31:10Z

docs/user/ppl/cmd/stats.rst

-        "plugins.calcite.enabled" : true
-      }
-    }'
+* aggregation: mandatory. An aggregation function. The argument of aggregation must be field.


The argument of aggregation must be field. can be removed

LantaoJin · 2025-10-16T06:31:48Z

docs/user/ppl/cmd/subquery.rst

 =============
-subquery (aka subsearch)
+subquery
 =============


= problem

LantaoJin · 2025-10-16T06:34:25Z

docs/user/ppl/cmd/timechart.rst

-3.3.0
-
 Syntax
 ============


= problem

Signed-off-by: Ritvi Bhatt <[email protected]>

Signed-off-by: ritvibhatt <[email protected]>

Signed-off-by: Ritvi Bhatt <[email protected]>

aalva500-prog · 2025-10-17T00:18:38Z

Hi @ritvibhatt, thanks for the changes! Quick question: should we also update the documentation in condition.rst for recently added functions like coalesce?

Signed-off-by: Ritvi Bhatt <[email protected]>

dai-chen

Reviewed doc relate to count(eval) and per_second functions. Thanks!

Signed-off-by: Ritvi Bhatt <[email protected]>

ritvibhatt · 2025-10-22T16:35:05Z

Hi @ritvibhatt, thanks for the changes! Quick question: should we also update the documentation in condition.rst for recently added functions like coalesce?

Hi @aalva500-prog, thanks for bringing that up! I have now updated the functions documentation as well

Signed-off-by: ritvibhatt <[email protected]>

ahkcs · 2025-10-28T20:24:16Z

Hi @ritvibhatt , reviewed bin, replace, fillnull commands and I think it looks good, for multisearch command, can you rebase it due to the newly merged PR: #4512? Thanks!

Signed-off-by: ritvibhatt <[email protected]>

Swiddis

crashes my web viewer of the diff, lgtm

Swiddis · 2025-10-29T17:45:23Z

docs/user/ppl/index.rst

+  ==============================================================  ==================  ========================  ==============================================================================================
+  Command Name                                                    Version Introduced  Current Status              Command Description
+  ==============================================================  ==================  ========================  ==============================================================================================
+  `search command <cmd/search.rst>`_                              1.0                 stable (since 1.0)        Retrieve documents from the index.


Nice work on the summaries

Swiddis · 2025-10-29T17:47:36Z

docs/user/ppl/index.rst

+* **Functions**

-  - `top command <cmd/top.rst>`_
+  - `Aggregation Functions <functions/aggregation.rst>`_


question: Should we have descriptions or some further info for these too?

Not clear how to find e.g. the SUM function (is it an aggregation function or a math function?).

Yeah that makes sense to do! Was thinking of a follow up PR to update some of the functions content as well as add to index.rst since this PR is already pretty big

LantaoJin · 2025-10-30T01:39:43Z

docs/user/ppl/cmd/stats.rst

-    | ["Amber","Dale","Hattie","Nanette"] |
-    +-------------------------------------+
+* aggregation: mandatory. An aggregation function.
+* bucket_nullable: optional. Controls whether the stats command includes null buckets in group-by aggregations. When set to ``false``, the aggregation ignores records where the group-by field is null, resulting in faster performance by excluding null bucket. **Default:** determined by ``plugins.ppl.syntax.legacy.preferred``:


@ritvibhatt can you review your preview pages. It seems introduce many unexpected formatting.
For example, lines of bucket_nullable and span-expression in your preview
https://github.com/ritvibhatt/sql/blob/update-docs/docs/user/ppl/cmd/stats.rst are bold.

ritvibhatt added 3 commits October 13, 2025 11:43

remove version info

8445ebb

Signed-off-by: Ritvi Bhatt <[email protected]>

fix defaults

81e2da1

Signed-off-by: Ritvi Bhatt <[email protected]>

update main docs and move aggregate functions

3ea70a5

Signed-off-by: Ritvi Bhatt <[email protected]>

ritvibhatt marked this pull request as ready for review October 15, 2025 04:04

Merge branch 'main' into update-docs

0dc4a62

Signed-off-by: ritvibhatt <[email protected]>

ritvibhatt marked this pull request as draft October 15, 2025 04:13

fix format

3c67613

Signed-off-by: Ritvi Bhatt <[email protected]>

ritvibhatt changed the title ~~Update Documentation~~ Update PPL Command Documentation Oct 15, 2025

ritvibhatt marked this pull request as ready for review October 15, 2025 15:54

RyanL1997 reviewed Oct 15, 2025

View reviewed changes

LantaoJin reviewed Oct 16, 2025

View reviewed changes

LantaoJin added the documentation Improvements or additions to documentation label Oct 16, 2025

LantaoJin reviewed Oct 16, 2025

View reviewed changes

ritvibhatt and others added 6 commits October 16, 2025 11:11

fix heading underlines

7dd7db9

Signed-off-by: Ritvi Bhatt <[email protected]>

fix typos/content

66a3e97

Signed-off-by: Ritvi Bhatt <[email protected]>

Merge branch 'main' into update-docs

06f0f8d

Signed-off-by: ritvibhatt <[email protected]>

fix formatting

657f38e

Signed-off-by: Ritvi Bhatt <[email protected]>

update rex limitations

055030a

Signed-off-by: Ritvi Bhatt <[email protected]>

update underlines and bullet points

a422a2d

Signed-off-by: Ritvi Bhatt <[email protected]>

ritvibhatt added 5 commits October 20, 2025 11:07

update function files

e4f01a0

Signed-off-by: Ritvi Bhatt <[email protected]>

update index.rst with commands table

09da08c

Signed-off-by: Ritvi Bhatt <[email protected]>

update formatting

1165d0f

Signed-off-by: Ritvi Bhatt <[email protected]>

update wording

7470967

Signed-off-by: Ritvi Bhatt <[email protected]>

move syntax

8886563

Signed-off-by: Ritvi Bhatt <[email protected]>

dai-chen previously approved these changes Oct 22, 2025

View reviewed changes

update note

54464e3

Signed-off-by: Ritvi Bhatt <[email protected]>

ritvibhatt dismissed dai-chen’s stale review via 54464e3 October 22, 2025 16:30

Merge branch 'main' into update-docs

50d95b3

Signed-off-by: ritvibhatt <[email protected]>

Merge branch 'main' into update-docs

f43765d

Signed-off-by: ritvibhatt <[email protected]>

Swiddis approved these changes Oct 29, 2025

View reviewed changes

LantaoJin reviewed Oct 30, 2025

View reviewed changes

LantaoJin mentioned this pull request Oct 30, 2025

Support 'usenull' option in PPL top and rare commands #4696

Merged

8 tasks

		================================== ============ ============


		Limitations

		fillnull with <replacement> [in <field-list>]

		fillnull using <field> = <replacement> [, <field> = <replacement>]


		For extended syntax (join with field list) in 3.3.0, when duplicate-named fields in output results are deduplicated, the fields in output determined by the value of 'overwrite' option.

		Since 3.3.0, join types ``inner``, ``left``, ``outer`` (alias of ``left``), ``semi`` and ``anti`` are supported by default. ``right``, ``full``, ``cross`` are performance sensitive join types which are disabled by default. Set config ``plugins.calcite.all_join_types.allowed = true`` to enable.

Update PPL Command Documentation #4562

Are you sure you want to change the base?

Update PPL Command Documentation #4562

Uh oh!

Conversation

ritvibhatt commented Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes Made

Documentation Structure Standardization

Content Cleanup and Modernization

Related Issues

Check List

Uh oh!

RyanL1997 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aalva500-prog commented Oct 17, 2025

Uh oh!

dai-chen left a comment

Choose a reason for hiding this comment

Uh oh!

ritvibhatt commented Oct 22, 2025

Uh oh!

ahkcs commented Oct 28, 2025

Uh oh!

Swiddis left a comment

ritvibhatt commented Oct 15, 2025 •

edited

Loading

LantaoJin Oct 30, 2025 •

edited

Loading