Skip to content

Commit

Permalink
Test all examples from library-user-guide & user-guide docs (#14544)
Browse files Browse the repository at this point in the history 
* add mut annotation

* fix rust examples

* fix rust examples

* update

* fix first doctest

* fix first doctest

* fix more doctest

* fix more doctest

* fix more doctest

* adopt rustdoc syntax

* adopt rustdoc syntax

* adopt rustdoc syntax

* fix more doctest

* add missing imports

* final udtf

* reenable

* remove dep

* run prettier

* api-health

* update doc

* update doc

* temp fix

* fix doc

* fix async schema provider

* fix async schema provider

* fix doc

* fix doc

* reorder

* refactor

* s

* finish

* minor update

* add missing docs

* add deps (#3)

* fix doctest

* update doc

* fix doctest

* fix doctest

* tweak showkeys

* fix doctest

* fix doctest

* fix doctest

* fix doctest

* update to use user_doc

* add rustdoc preprocessing

* fix dir

* revert to original doc

* add allocator

* mark type

* update

* fix doctest

* add doctest

* add doctest

* fix doctest

* fix doctest

* fix doctest

* fix doctest

* fix doctest

* fix doctest

* fix doctest

* fix doctest

* fix doctest

* prettier format

* revert change to datafusion-testing

* add apache header

* install cmake in setup-builder for ci workflow dependency

* taplo + fix snmalloc

* Update function docs

* preprocess user-guide

* Render examples as sql

* fix intro

* fix docs via script

---------

Co-authored-by: Andrew Lamb <[email protected]>
  • Loading branch information
@ugoa @alamb
ugoa and alamb authored Feb 10, 2025
1 parent dee0dc7 commit 3f900ac
Show file tree
Hide file tree
Showing 23 changed files with 1,983 additions and 512 deletions.
2 changes: 1 addition & 1 deletion .github/actions/setup-builder/action.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ runs:
run: |
RETRY=("ci/scripts/retry" timeout 120)
"${RETRY[@]}" apt-get update
"${RETRY[@]}" apt-get install -y protobuf-compiler
"${RETRY[@]}" apt-get install -y protobuf-compiler cmake
- name: Setup Rust toolchain
shell: bash
# rustfmt is needed for the substrait build script
Expand Down
3 changes: 3 additions & 0 deletions Cargo.lock
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions datafusion/core/Cargo.toml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -95,13 +95,15 @@ datafusion-catalog = { workspace = true }
datafusion-catalog-listing = { workspace = true }
datafusion-common = { workspace = true, features = ["object_store"] }
datafusion-common-runtime = { workspace = true }
datafusion-doc = { workspace = true }
datafusion-execution = { workspace = true }
datafusion-expr = { workspace = true }
datafusion-functions = { workspace = true }
datafusion-functions-aggregate = { workspace = true }
datafusion-functions-nested = { workspace = true, optional = true }
datafusion-functions-table = { workspace = true }
datafusion-functions-window = { workspace = true }
datafusion-macros = { workspace = true }
datafusion-optimizer = { workspace = true }
datafusion-physical-expr = { workspace = true }
datafusion-physical-expr-common = { workspace = true }
Expand Down Expand Up @@ -130,6 +132,7 @@ zstd = { version = "0.13", optional = true, default-features = false }
async-trait = { workspace = true }
criterion = { version = "0.5", features = ["async_tokio"] }
ctor = { workspace = true }
dashmap = "6.1.0"
datafusion-functions-window-common = { workspace = true }
datafusion-physical-optimizer = { workspace = true }
doc-comment = { workspace = true }
Expand Down
2 changes: 1 addition & 1 deletion datafusion/core/src/bin/print_functions_docs.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -193,7 +193,7 @@ fn print_docs(
{}
```
```sql
{}
```
"#,
Expand Down
207 changes: 198 additions & 9 deletions datafusion/core/src/lib.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -844,11 +844,17 @@ doc_comment::doctest!("../../../README.md", readme_example_test);
//
// For example, if `user_guide_expressions(line 123)` fails,
// go to `docs/source/user-guide/expressions.md` to find the relevant problem.
//
#[cfg(doctest)]
doc_comment::doctest!(
"../../../docs/source/user-guide/concepts-readings-events.md",
user_guide_concepts_readings_events
);

#[cfg(doctest)]
doc_comment::doctest!(
"../../../docs/source/user-guide/example-usage.md",
user_guide_example_usage
"../../../docs/source/user-guide/configs.md",
user_guide_configs
);

#[cfg(doctest)]
Expand All @@ -859,14 +865,20 @@ doc_comment::doctest!(

#[cfg(doctest)]
doc_comment::doctest!(
"../../../docs/source/user-guide/configs.md",
user_guide_configs
"../../../docs/source/user-guide/dataframe.md",
user_guide_dataframe
);

#[cfg(doctest)]
doc_comment::doctest!(
"../../../docs/source/user-guide/dataframe.md",
user_guide_dataframe
"../../../docs/source/user-guide/example-usage.md",
user_guide_example_usage
);

#[cfg(doctest)]
doc_comment::doctest!(
"../../../docs/source/user-guide/explain-usage.md",
user_guide_explain_usage
);

#[cfg(doctest)]
Expand All @@ -875,20 +887,197 @@ doc_comment::doctest!(
user_guide_expressions
);

#[cfg(doctest)]
doc_comment::doctest!("../../../docs/source/user-guide/faq.md", user_guide_faq);

#[cfg(doctest)]
doc_comment::doctest!(
"../../../docs/source/library-user-guide/using-the-sql-api.md",
library_user_guide_sql_api
"../../../docs/source/user-guide/introduction.md",
user_guide_introduction
);

#[cfg(doctest)]
doc_comment::doctest!(
"../../../docs/source/user-guide/cli/datasources.md",
user_guide_cli_datasource
);

#[cfg(doctest)]
doc_comment::doctest!(
"../../../docs/source/user-guide/cli/installation.md",
user_guide_cli_installation
);

#[cfg(doctest)]
doc_comment::doctest!(
"../../../docs/source/user-guide/cli/overview.md",
user_guide_cli_overview
);

#[cfg(doctest)]
doc_comment::doctest!(
"../../../docs/source/user-guide/cli/usage.md",
user_guide_cli_usage
);

#[cfg(doctest)]
doc_comment::doctest!(
"../../../docs/source/user-guide/sql/aggregate_functions.md",
user_guide_sql_aggregate_functions
);

#[cfg(doctest)]
doc_comment::doctest!(
"../../../docs/source/user-guide/sql/data_types.md",
user_guide_sql_data_types
);

#[cfg(doctest)]
doc_comment::doctest!(
"../../../docs/source/user-guide/sql/ddl.md",
user_guide_sql_ddl
);

#[cfg(doctest)]
doc_comment::doctest!(
"../../../docs/source/user-guide/sql/dml.md",
user_guide_sql_dml
);

#[cfg(doctest)]
doc_comment::doctest!(
"../../../docs/source/user-guide/sql/explain.md",
user_guide_sql_exmplain
);

#[cfg(doctest)]
doc_comment::doctest!(
"../../../docs/source/user-guide/sql/information_schema.md",
user_guide_sql_information_schema
);

#[cfg(doctest)]
doc_comment::doctest!(
"../../../docs/source/user-guide/sql/operators.md",
user_guide_sql_operators
);

#[cfg(doctest)]
doc_comment::doctest!(
"../../../docs/source/user-guide/sql/scalar_functions.md",
user_guide_sql_scalar_functions
);

#[cfg(doctest)]
doc_comment::doctest!(
"../../../docs/source/user-guide/sql/select.md",
user_guide_sql_select
);

#[cfg(doctest)]
doc_comment::doctest!(
"../../../docs/source/user-guide/sql/special_functions.md",
user_guide_sql_special_functions
);

#[cfg(doctest)]
doc_comment::doctest!(
"../../../docs/source/user-guide/sql/sql_status.md",
user_guide_sql_status
);

#[cfg(doctest)]
doc_comment::doctest!(
"../../../docs/source/user-guide/sql/subqueries.md",
user_guide_sql_subqueries
);

#[cfg(doctest)]
doc_comment::doctest!(
"../../../docs/source/user-guide/sql/window_functions.md",
user_guide_sql_window_functions
);

#[cfg(doctest)]
doc_comment::doctest!(
"../../../docs/source/user-guide/sql/write_options.md",
user_guide_sql_write_options
);

#[cfg(doctest)]
doc_comment::doctest!(
"../../../docs/source/library-user-guide/adding-udfs.md",
library_user_guide_adding_udfs
);

#[cfg(doctest)]
doc_comment::doctest!(
"../../../docs/source/library-user-guide/api-health.md",
library_user_guide_api_health
);

#[cfg(doctest)]
doc_comment::doctest!(
"../../../docs/source/library-user-guide/building-logical-plans.md",
library_user_guide_logical_plans
library_user_guide_building_logical_plans
);

#[cfg(doctest)]
doc_comment::doctest!(
"../../../docs/source/library-user-guide/catalogs.md",
library_user_guide_catalogs
);

#[cfg(doctest)]
doc_comment::doctest!(
"../../../docs/source/library-user-guide/custom-table-providers.md",
library_user_guide_custom_table_providers
);

#[cfg(doctest)]
doc_comment::doctest!(
"../../../docs/source/library-user-guide/extending-operators.md",
library_user_guide_extending_operators
);

#[cfg(doctest)]
doc_comment::doctest!(
"../../../docs/source/library-user-guide/extensions.md",
library_user_guide_extensions
);

#[cfg(doctest)]
doc_comment::doctest!(
"../../../docs/source/library-user-guide/index.md",
library_user_guide_index
);

#[cfg(doctest)]
doc_comment::doctest!(
"../../../docs/source/library-user-guide/profiling.md",
library_user_guide_profiling
);

#[cfg(doctest)]
doc_comment::doctest!(
"../../../docs/source/library-user-guide/query-optimizer.md",
library_user_guide_query_optimizer
);

#[cfg(doctest)]
doc_comment::doctest!(
"../../../docs/source/library-user-guide/using-the-dataframe-api.md",
library_user_guide_dataframe_api
);

#[cfg(doctest)]
doc_comment::doctest!(
"../../../docs/source/library-user-guide/using-the-sql-api.md",
library_user_guide_sql_api
);

#[cfg(doctest)]
doc_comment::doctest!(
"../../../docs/source/library-user-guide/working-with-exprs.md",
library_user_guide_working_with_exprs
);
4 changes: 2 additions & 2 deletions dev/update_function_docs.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -236,7 +236,7 @@ WINDOW w AS (PARTITION BY depname ORDER BY salary DESC);
The syntax for the OVER-clause is
```
```sql
function([expr])
OVER(
[PARTITION BY expr[, …]]
Expand All @@ -247,7 +247,7 @@ function([expr])
where **frame_clause** is one of:
```
```sql
{ RANGE | ROWS | GROUPS } frame_start
{ RANGE | ROWS | GROUPS } BETWEEN frame_start AND frame_end
```
Expand Down
3 changes: 3 additions & 0 deletions docs/build.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,4 +25,7 @@ mkdir temp
cp -rf source/* temp/
# replace relative URLs with absolute URLs
sed -i -e 's/\.\.\/\.\.\/\.\.\//https:\/\/github.com\/apache\/arrow-datafusion\/blob\/main\//g' temp/contributor-guide/index.md

python rustdoc_trim.py

make SOURCEDIR=`pwd`/temp html
Loading

0 comments on commit 3f900ac

Please sign in to comment.