Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

changing parallelization defaults #1654

Merged
merged 6 commits into from
Jan 24, 2025
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions docs/parallel_config.md
Original file line number Diff line number Diff line change
@@ -64,8 +64,8 @@ Validate parameters/structure of configuration data.
* **img_outdir**: (str, default = "."): path/name of output directory where images will be saved.


* **tmp_dir**: (str, default = `None`): path/name of parent folder for the temporary directory, uses system default
temporary directory when `None`.
* **tmp_dir**: (str, default = `"."`): path/name of parent folder for the temporary directory, defaults to the
current working directory.


* **start_date**: (str, default = `None`): start date used to filter images. Images will be analyzed that are newer
@@ -107,15 +107,15 @@ for downstream analysis. The default, `filepath` will create groups of single im
example of a multi-image group could be to pair VIS and NIR images (e.g. `["timestamp", "camera", "rotation"]`). Supported
metadata terms are listed [here](pipeline_parallel.md).

* **group_name** (str, default = `"imgtype"`): either a metadata term used to create a unique name for each image in an
* **group_name** (str, default = `"auto"`): either a metadata term used to create a unique name for each image in an
image group (created by `groupby`), or `"auto"` to generate a numbered image sequence `image1, image2, ...`. The resulting
names are used to access individual image filepaths in a workflow.

* **cleanup**: (bool, default =`True`): remove temporary job directory if `True`.


* **append**: (bool, default = `True`): if `True` will append results to an existing json file. If `False`, will delete
previous results stored in the specified JSON file.
* **append**: (bool, default = `False`): if `False`, will delete previous results stored in the specified JSON file.
If `True` will append results to an existing json file.


* **cluster** (str, default = "LocalCluster"): LocalCluster will run PlantCV workflows on a single machine. All valid
12 changes: 6 additions & 6 deletions docs/pipeline_parallel.md
Original file line number Diff line number Diff line change
@@ -99,7 +99,7 @@ Sample image filename: `cam1_16-08-06-16:45_el1100s1_p19.jpg`
"filename_metadata": ["camera", "timestamp", "id", "other"],
"workflow": "/home/mgehan/pat-edger/round1-python-pipelines/2016-08_pat-edger_brassica-cam1-splitimg.py",
"img_outdir": "/shares/mgehan_share/raw_data/raw_image/2016-08_pat-edger/data/split-round1/split-cam1/output",
"tmp_dir": null,
"tmp_dir": "."",
"start_date": null,
"end_date": null,
"imgformat": "jpg",
@@ -111,7 +111,7 @@ Sample image filename: `cam1_16-08-06-16:45_el1100s1_p19.jpg`
"groupby": ["filepath"],
"group_name": "auto",
"cleanup": true,
"append": true,
"append": false,
"cluster": "HTCondorCluster",
"cluster_config": {
"n_workers": 16,
@@ -175,7 +175,7 @@ in a list to the `filename_metadata` parameter.
"filename_metadata": ["camera", "plantbarcode", "timestamp"],
"workflow": "user-workflow.py",
"img_outdir": "output_directory",
"tmp_dir": null,
"tmp_dir": ".",
"start_date": null,
"end_date": null,
"imgformat": "jpg",
@@ -187,7 +187,7 @@ in a list to the `filename_metadata` parameter.
"groupby": ["filepath"],
"group_name": "auto",
"cleanup": true,
"append": true,
"append": false,
"cluster": "HTCondorCluster",
"cluster_config": {
"n_workers": 16,
@@ -227,7 +227,7 @@ To identify each image within our workflow, we will name them based on the `imgt
"filename_metadata": ["imgtype", "timestamp", "id", "other"],
"workflow": "/home/mgehan/pat-edger/round1-python-pipelines/2016-08_pat-edger_brassica-cam1-splitimg.py",
"img_outdir": "/shares/mgehan_share/raw_data/raw_image/2016-08_pat-edger/data/split-round1/split-cam1/output",
"tmp_dir": null,
"tmp_dir": ".",
"start_date": null,
"end_date": null,
"imgformat": "jpg",
@@ -239,7 +239,7 @@ To identify each image within our workflow, we will name them based on the `imgt
"groupby": ["timestamp"],
"group_name": "imgtype",
"cleanup": true,
"append": true,
"append": false,
"cluster": "HTCondorCluster",
"cluster_config": {
"n_workers": 16,
6 changes: 3 additions & 3 deletions plantcv/parallel/__init__.py
Original file line number Diff line number Diff line change
@@ -21,7 +21,7 @@ def __init__(self):
self.workflow = ""
self.img_outdir = "./output_images"
self.include_all_subdirs = True
self.tmp_dir = None
self.tmp_dir = "."
self.start_date = None
self.end_date = None
self.imgformat = "png"
@@ -31,9 +31,9 @@ def __init__(self):
self.writeimg = False
self.other_args = {}
self.groupby = ["filepath"]
self.group_name = "imgtype"
self.group_name = "auto"
self.cleanup = True
self.append = True
self.append = False
self.cluster = "LocalCluster"
self.cluster_config = {
"n_workers": 1,
6 changes: 3 additions & 3 deletions tests/testdata/workflowconfig_template.json
Original file line number Diff line number Diff line change
@@ -5,7 +5,7 @@
"include_all_subdirs": true,
"workflow": "",
"img_outdir": "./output_images",
"tmp_dir": null,
"tmp_dir": ".",
"start_date": null,
"end_date": null,
"imgformat": "png",
@@ -15,9 +15,9 @@
"writeimg": false,
"other_args": {},
"groupby": ["filepath"],
"group_name": "imgtype",
"group_name": "auto",
"cleanup": true,
"append": true,
"append": false,
"cluster": "LocalCluster",
"cluster_config": {
"n_workers": 1,