Skip to content

Commit 9fc3d5d

Browse files
bbestbbest
committedDec 3, 2024·
portals: gt table portal comparison; diagrams/*.mmd
1 parent 9cd0b66 commit 9fc3d5d

File tree

5 files changed

+171
-121
lines changed

5 files changed

+171
-121
lines changed
 

‎_quarto.yml

+5-1
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,10 @@ project:
88
editor:
99
mode: source
1010

11+
execute:
12+
echo: false
13+
warning: false
14+
1115
book:
1216
title: "CalCOFI.io Docs"
1317
description: "Documentation for Open-Source Products at CalCOFI.io"
@@ -39,7 +43,7 @@ link-citations: true
3943
format:
4044
html:
4145
theme: cosmo
42-
css: lib/style.css
46+
# css: libs/style.css
4347
toc: true
4448
number-sections: true
4549
docx:

‎data/portal_comparison.csv

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
Portal,Full Archive,Versioning,DOI Issued,Query by xyt,Query by taxa,Multiple formats,API Access
2+
EDI,full,full,full,partial,partial,none,partial
3+
NCEI,full,full,full,none,none,none,partial
4+
OBIS,partial,partial,partial,full,full,partial,full
5+
ERDDAP,partial,none,none,full,partial,full,full

‎diagrams/portals_flow.mmd

+61
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
%%{init: {
2+
'theme': 'base',
3+
'themeVariables': {
4+
'primaryColor': '#E0E7FF',
5+
'primaryTextColor': '#312E81',
6+
'primaryBorderColor': '#6366F1',
7+
'lineColor': '#818CF8',
8+
'secondaryColor': '#FEF3C7',
9+
'tertiaryColor': '#ECFDF5'
10+
}
11+
}}%%
12+
13+
graph TD
14+
%% nodes with styles
15+
raw([raw data]):::source
16+
17+
subgraph calcofi[CalCOFI.io]
18+
db[(database)]:::database
19+
web[calcofi.org]:::website
20+
api[APIs]:::api
21+
lib[library]:::code
22+
app[apps]:::code
23+
flow[workflows]:::code
24+
end
25+
26+
subgraph portals[Portals]
27+
edi[EDI]:::portal
28+
obis[OBIS]:::portal
29+
erddap[ERDDAP]:::portal
30+
ncei[NCEI]:::portal
31+
end
32+
33+
subgraph meta[Meta-Portals]
34+
odis[ODIS]:::metaportal
35+
datagov[data.gov]:::metaportal
36+
end
37+
38+
%% edges
39+
raw --> db
40+
db --> api
41+
db --> web
42+
api --> lib
43+
api --> app
44+
api --> flow
45+
flow --> portals
46+
portals -->|sitemap| odis
47+
portals --> datagov
48+
49+
%% Custom styles
50+
classDef source fill:#E0E7FF,stroke:#6366F1,stroke-width:2px
51+
classDef database fill:#FEF3C7,stroke:#D97706,stroke-width:2px
52+
classDef website fill:#F3E8FF,stroke:#9333EA,stroke-width:2px
53+
classDef api fill:#E0E7FF,stroke:#6366F1,stroke-width:2px
54+
classDef code fill:#DBEAFE,stroke:#3B82F6,stroke-width:2px
55+
classDef portal fill:#F0FDF4,stroke:#22C55E,stroke-width:2px
56+
classDef metaportal fill:#FEF2F2,stroke:#DC2626,stroke-width:2px
57+
58+
%% Style subgraphs
59+
style calcofi fill:#F8FAFC,stroke:#CBD5E1,stroke-width:2px
60+
style portals fill:#F8FAFC,stroke:#CBD5E1,stroke-width:2px
61+
style meta fill:#F8FAFC,stroke:#CBD5E1,stroke-width:2px

‎diagrams/portals_metadata.mmd

+38
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
%%{init: {
2+
'theme': 'base',
3+
'themeVariables': {
4+
'primaryColor': '#E0E7FF',
5+
'primaryTextColor': '#312E81',
6+
'primaryBorderColor': '#6366F1',
7+
'lineColor': '#818CF8',
8+
'secondaryColor': '#FEF3C7',
9+
'tertiaryColor': '#ECFDF5'
10+
}
11+
}}%%
12+
graph LR
13+
subgraph inputs[Data & Metadata]
14+
data([data.csv]):::source
15+
eml[metadata.eml]:::database
16+
end
17+
18+
subgraph portals[Data Portals]
19+
edi[EDI]:::portal
20+
erddap[ERDDAP]:::portal
21+
obis[OBIS]:::portal
22+
ncei[NCEI]:::portal
23+
end
24+
25+
data --> eml
26+
eml --> edi
27+
eml --> erddap
28+
eml --> obis
29+
eml --> ncei
30+
31+
%% Custom styles
32+
classDef source fill:#E0E7FF,stroke:#6366F1,stroke-width:2px
33+
classDef database fill:#FEF3C7,stroke:#D97706,stroke-width:2px
34+
classDef portal fill:#F0FDF4,stroke:#22C55E,stroke-width:2px
35+
36+
%% Style subgraphs
37+
style inputs fill:#F8FAFC,stroke:#CBD5E1,stroke-width:2px
38+
style portals fill:#F8FAFC,stroke:#CBD5E1,stroke-width:2px

‎portals.qmd

+62-120
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,4 @@
1-
---
2-
title: "Portals"
3-
---
1+
# Portals
42

53
## Overview
64

@@ -11,85 +9,63 @@ CalCOFI data is available through various portals, each serving different purpos
119
While it would be ideal for CalCOFI data to be available through a single portal, each portal has its strengths and limitations. The following diagram illustrates one possible realization of data flow between CalCOFI data and the portals: from raw data to the integrated database to portals and meta-portals.
1210

1311
```{mermaid}
14-
%%{init: {
15-
'theme': 'base',
16-
'themeVariables': {
17-
'primaryColor': '#E0E7FF',
18-
'primaryTextColor': '#312E81',
19-
'primaryBorderColor': '#6366F1',
20-
'lineColor': '#818CF8',
21-
'secondaryColor': '#FEF3C7',
22-
'tertiaryColor': '#ECFDF5'
23-
}
24-
}}%%
25-
26-
graph TD
27-
%% nodes with styles
28-
raw([raw data]):::source
29-
30-
subgraph calcofi[CalCOFI.io]
31-
db[(database)]:::database
32-
web[calcofi.org]:::website
33-
api[APIs]:::api
34-
lib[library]:::code
35-
app[apps]:::code
36-
flow[workflows]:::code
37-
end
38-
39-
subgraph portals[Portals]
40-
edi[EDI]:::portal
41-
obis[OBIS]:::portal
42-
erddap[ERDDAP]:::portal
43-
ncei[NCEI]:::portal
44-
end
45-
46-
subgraph meta[Meta-Portals]
47-
odis[ODIS]:::metaportal
48-
datagov[data.gov]:::metaportal
49-
end
50-
51-
%% edges
52-
raw --> db
53-
db --> api
54-
db --> web
55-
api --> lib
56-
api --> app
57-
api --> flow
58-
flow --> portals
59-
portals -->|sitemap| odis
60-
portals --> datagov
61-
62-
%% Custom styles
63-
classDef source fill:#E0E7FF,stroke:#6366F1,stroke-width:2px
64-
classDef database fill:#FEF3C7,stroke:#D97706,stroke-width:2px
65-
classDef website fill:#F3E8FF,stroke:#9333EA,stroke-width:2px
66-
classDef api fill:#E0E7FF,stroke:#6366F1,stroke-width:2px
67-
classDef code fill:#DBEAFE,stroke:#3B82F6,stroke-width:2px
68-
classDef portal fill:#F0FDF4,stroke:#22C55E,stroke-width:2px
69-
classDef metaportal fill:#FEF2F2,stroke:#DC2626,stroke-width:2px
70-
71-
%% Style subgraphs
72-
style calcofi fill:#F8FAFC,stroke:#CBD5E1,stroke-width:2px
73-
style portals fill:#F8FAFC,stroke:#CBD5E1,stroke-width:2px
74-
style meta fill:#F8FAFC,stroke:#CBD5E1,stroke-width:2px
12+
%%| label: fig-portals-flow
13+
%%| fig-cap: "Flow of data from raw to database to portals and meta-portals."
14+
%%| file: diagrams/portals_flow.mmd
15+
%%| fig-width: 6
7516
```
7617

7718
In practice, CalCOFI is a partnership with various contributing members, so the authoritative dataset might flow differently, such as from EDI to the database to the other portals. The other portals, such as OBIS or ERDDAP, serve different audiences or purposes. The meta-portals like ODIS and Data.gov then index these portals to provide broader discovery of CalCOFI datasets.
7819

7920
## Portals
8021

81-
| Portal | Full Archive | Versioning | DOI issued | Query by xyt | Query by taxa | Multiple formats | API Access |
82-
|----|----|----|----|----|----|----|----|
83-
| **EDI** | :green_circle: | :green_circle: | :green_circle: | :yellow_circle: | :yellow_circle: | :red_circle: | :yellow_circle: |
84-
| **NCEI** | :green_circle: | :green_circle: | :green_circle: | :red_circle: | :red_circle: | :red_circle: | :yellow_circle: |
85-
| **OBIS** | :yellow_circle: | :yellow_circle: | :yellow_circle: | :green_circle: | :green_circle: | :yellow_circle: | :green_circle: |
86-
| **ERDDAP** | :yellow_circle: | :red_circle: | :red_circle: | :green_circle: | :yellow_circle: | :green_circle: | :green_circle: |
87-
88-
<!-- TODO: get emojis to show in PDF -->
89-
90-
------------------------------------------------------------------------
91-
92-
† Capability: :red_circle: none; :yellow_circle: partial; :green_circle: full
22+
While some portals serve as data repositories, others provide advanced data access and visualization tools. The following sections describe the main portals where CalCOFI data is available and their key features.
23+
24+
```{r}
25+
#| label: tbl-portals
26+
#| tbl-cap: Portal Capabilities.
27+
28+
librarian::shelf(
29+
dplyr, gt, here, readr, scales)
30+
31+
portals_csv <- here("data/portal_comparison.csv")
32+
33+
# Define the capability symbols and colors
34+
cap_symbols <- c(
35+
"none" = "\u2716", # ✖
36+
"partial" = "\u25B2", # ▲
37+
"full" = "\u2714") # ✔
38+
cap_colors <- c(
39+
"none" = "#FFCCCC", # Light red
40+
"partial" = "#FFF2CC", # Light yellow
41+
"full" = "#CCFFCC") # Light green
42+
43+
# Create the gt table
44+
read_csv(portals_csv, show_col_types = F) |>
45+
gt(rowname_col = "Portal") |>
46+
# Apply background colors based on capability levels
47+
data_color(
48+
# columns = everything(),
49+
fn = scales::col_factor(
50+
palette = as.character(cap_colors),
51+
levels = names(cap_colors))) |>
52+
# Center align the text
53+
cols_align(
54+
align = "center",
55+
columns = everything() ) |>
56+
# Apply colored symbols
57+
text_transform(
58+
fn = function(x) {
59+
cap_symbols[x] } ) |>
60+
# Add a caption explaining the symbols and colors
61+
tab_source_note(
62+
source_note = md(
63+
"**Capability Legend**: ✔ = full, ▲ = partial, ✖ = none")) |>
64+
# Adjust the table's style for better aesthetics
65+
tab_options(
66+
table.width = pct(100),
67+
data_row.padding = px(5))
68+
```
9369

9470

9571
### EDI (Environmental Data Initiative)
@@ -147,54 +123,20 @@ In practice, CalCOFI is a partnership with various contributing members, so the
147123
The [Ecological Metadata Language (EML)](https://eml.ecoinformatics.org/) (and using R package [EML](https://docs.ropensci.org/EML/) in workflows) serves as a key standard for describing ecological and environmental data. For CalCOFI, EML metadata files are generated alongside data files, providing structured documentation that enables interoperability across different data portals. This metadata-driven approach allows automated ingestion into various data systems while maintaining data integrity and provenance.
148124

149125
```{mermaid}
150-
%%{init: {
151-
'theme': 'base',
152-
'themeVariables': {
153-
'primaryColor': '#E0E7FF',
154-
'primaryTextColor': '#312E81',
155-
'primaryBorderColor': '#6366F1',
156-
'lineColor': '#818CF8',
157-
'secondaryColor': '#FEF3C7',
158-
'tertiaryColor': '#ECFDF5'
159-
}
160-
}}%%
161-
graph LR
162-
subgraph inputs[Data & Metadata]
163-
data([data.csv]):::source
164-
eml[metadata.eml]:::database
165-
end
166-
167-
subgraph portals[Data Portals]
168-
edi[EDI]:::portal
169-
erddap[ERDDAP]:::portal
170-
obis[OBIS]:::portal
171-
ncei[NCEI]:::portal
172-
end
173-
174-
data --> eml
175-
eml --> edi
176-
eml --> erddap
177-
eml --> obis
178-
eml --> ncei
179-
180-
%% Custom styles
181-
classDef source fill:#E0E7FF,stroke:#6366F1,stroke-width:2px
182-
classDef database fill:#FEF3C7,stroke:#D97706,stroke-width:2px
183-
classDef portal fill:#F0FDF4,stroke:#22C55E,stroke-width:2px
184-
185-
%% Style subgraphs
186-
style inputs fill:#F8FAFC,stroke:#CBD5E1,stroke-width:2px
187-
style portals fill:#F8FAFC,stroke:#CBD5E1,stroke-width:2px
126+
%%| label: fig-portals-metadata
127+
%%| fig-cap: "Metadata in the form of ecological metadata language (EML) is used to describe the dataset in a consistent manner that can be ingested by the portals."
128+
%%| file: diagrams/portals_metadata.mmd
129+
%%| fig-width: 6
188130
```
189131

190132
The EML specification provides detailed structure for describing datasets, including:
191133

192-
- Dataset identification and citation
193-
- Geographic and temporal coverage
194-
- Variable definitions and units
195-
- Methods and protocols
196-
- Quality control procedures
197-
- Access and usage rights
134+
- Dataset identification and citation
135+
- Geographic and temporal coverage
136+
- Variable definitions and units
137+
- Methods and protocols
138+
- Quality control procedures
139+
- Access and usage rights
198140

199141
This standardized metadata enables automated data transformation and ingestion into various portal systems while preserving the original data context and quality information.
200142

0 commit comments

Comments
 (0)