Refactor PlantDB client/server: env vars, auth, URLs, deps#81
Merged
jlegrand62 merged 101 commits intodevfrom Jan 26, 2026
Merged
Refactor PlantDB client/server: env vars, auth, URLs, deps#81jlegrand62 merged 101 commits intodevfrom
jlegrand62 merged 101 commits intodevfrom
Conversation
- Updated `pyproject.toml` files to require Python versions greater than or equal to 3.8 but less than 3.13. - Replaced `bcrypt` with `argon2-cffi` in the `commons/pyproject.toml`. - Added `PyJWT>=2.6.0` as a dependency in the `commons/pyproject.toml`.
- Changed all imports of `FSDB`, `Scan`, `Fileset`, and `File` from `plantdb.commons.fsdb` to `plantdb.commons.fsdb.core`. - Updated corresponding docstrings and type hints. - Added type hints for return values in path helper functions.
- Renamed `date_now` to `iso_date_now` for clarity. - Updated import statements with additional `datetime` modules. - Changed return format from custom string to ISO 8601 compliant.
- Updated usages of `date_now` to `iso_date_now`. - Removed unnecessary imports: `datetime`, `timedelta`.
- Introduced authentication module with classes: - `Permission` - `Role` - `User` - `Group` - `UserManager` - Implemented user management functionalities like creation, loading, saving users - Added password hashing and verification using argon2
- Implemented test cases for: - `Permission` - `Role` - `User` - `Group` - `UserManager` - `GroupManager` - `RBACManager` - `SessionManager`
- Changed all imports of `dummy_db`, `FSDB`, `Scan`, `Fileset`, and `File` from `plantdb.commons.fsdb` to `plantdb.commons.fsdb.core`. - Updated corresponding docstrings and type hints. - Ensured consistency across multiple examples within the file.
- Added new session management parameters to `__init__` - Removed user management attributes and methods: - `_load_users`, `_save_users`, `_lock_db`, `_unlock_db` - `create_user`, `user_exists`, `_is_account_locked`, `_record_failed_attempt` - `validate_user`, `connect` (old implementation) - Added session and RBAC manager initialization - Updated `disconnect` method to clean up current session - Modified `get_scans` to use new permission system - Replaced old authentication code in `get_scan`, `create_scan`, `delete_scan` - Added new user management methods: `login`, `logout`, `get_guest_user`, `get_current_user` - Added group management methods: `create_group`, `add_user_to_group`, `remove_user_from_group`
- Added new methods for authentication: - `login` - `logout` - `refresh_token` - Implemented token management and error handling in `_make_authenticated_request` method.
- Added decorator `requires_jwt_auth` for securing endpoints with JWT authentication - Updated existing login logic to use JWT tokens and manage sessions - Implemented new `Logout` and `TokenRefresh` endpoints - Secured multiple resource classes using `@requires_jwt_auth`
- Import necessary modules for session management - Add `Logout` and `TokenRefresh` endpoints - Update secure cookie settings for production environment - Set up session parameters based on debug mode
- Added `create_user` method with detailed documentation - Enhanced `get_guest_user` method with detailed documentation
…ort. - Update the Python version constraint in the conda environment setup from `'python=3.10'` to `'python<3.13'`. - Refactor the import of `FSDB` from `plantdb.commons.fsdb` to `plantdb.commons.fsdb.core`.
- Updated `Scan` and `Fileset` imports from `plantdb.commons.fsdb` to `plantdb.commons.fsdb.core` in various files: - `client/rest_api.py` - `commons/utils.py` - `commons/sshfsdb.py` - `commons/testing.py` - Ensured consistency across docstrings and type hints.
- Renamed `out_path` to `db_path` in the following functions: - `get_test_dataset` - `get_models_dataset` - `get_configs` - `setup_empty_database` - `setup_test_database` - `test_database` - Updated corresponding variable names and usage within each function.
- Updated all occurrences of `Scan` and related classes from `plantdb.commons.fsdb` to `plantdb.commons.fsdb.core`. - Modified corresponding docstrings, type hints, and inline comments for consistency.
- Update server initialization to handle default DB path - Add SSL support for the test REST API server - Improve logging using a dedicated logger instead of print statements - Refactor rest_api function in fsdb_rest_api.py: - Rename `db_location` parameter to `db_path` - Adjust secure cookie settings based on SSL flag
- Add `PLANTDB_API_SSL` environment variable to enable SSL support - Update application initialization to use the new `ssl` parameter
- Added support for `Tuple` type from typing module - Updated comments and log messages for better clarity - Refactored `_user_session_id` to `_user_has_session` - Implemented new method `_create_jwt` for JWT creation - Improved session validation and token management with registered claims - Enhanced error handling in token decoding process - Modified `invalidate_session` to return username along with status
- Added two new decorators: `require_connected_db` and `require_auth_username` - Enhanced various methods to require database connection and user authentication
- Updated docstring to include key features, usage examples, and command-line interface details. - Added inline comments for better code readability and understanding.
- Refactored JWT authentication decorator: `requires_jwt_auth` renamed to `requires_jwt`, updated token retrieval, removed cookie handling, simplified validation - Updated REST API methods for login, logout, refresh, scan creation, metadata management, fileset and file operations: - Changed login response to include access token in JSON format instead of HttpOnly cookie - Refactored logout endpoint to use token from keyword arguments, added error logging - Modified refresh endpoint to return new token in JSON format without updating cookies - Updated scan creation, metadata management, fileset and file operations endpoints to use new `requires_jwt` decorator and token handling - Added utility functions: `is_valid_archive`, improved ZIP file extraction with hash verification, enhanced error handling and cleanup
- Added a new test file: `test_rest_api_server.py`
- Implemented unit tests for various endpoints, including:
- Home endpoint
- Login/logout functionality
- Scans list and table
- Scan metadata retrieval
- Image, point cloud, mesh, skeleton, and sequence data endpoints
- Archive creation and integrity verification
- Enhanced test server initialization with improved health checks
- Updated examples to include server startup/shutdown steps - Improved login method: - Enhanced docstring for clarity - Added JWT token handling using headers - Refactored logout method: - Removed unused cookie handling code - Added exception handling for robustness - Added new `refresh` method for database refresh functionality
- Include `open3d` in the `pip install` command in `docker/Dockerfile` to make the Open3D library available in the build environment.
- Update `requires-python` in `src/server/pyproject.toml` from `>=3.8,<3.13` to `>=3.8` - Update `requires-python` in `src/client/pyproject.toml` from `>=3.8,<3.13` to `>=3.8`
…ad of `plantdb.server.rest_api`.
- Introduced `_get_env_secret` to centralise secret key retrieval and warning logic - Created `_configure_app` for Flask app creation and secure cookie configuration - Added `_configure_api` to attach Flask‑RESTful API and handle proxy prefixes - Implemented `_setup_test_database` to create temporary test DBs with optional data - Moved resource registration into `_register_resources` for cleaner `rest_api` logic - Simplified `rest_api` to use the new helper functions and added cleanup registration - Updated error logging and messages for missing DB paths and temporary DB removal - Updated imports and removed duplicated `typing`/`pathlib` imports - Minor formatting and documentation improvements in `fsdb_rest_api.py`
- Make `secret_key` mandatory in `_configure_app` and update its documentation. - Update `_configure_api` and `_setup_test_database` docstrings and type hints. - Refactor `_register_resources` to call `api.add_resource` with multiline arguments for clarity. - Clarify logger usage in `_get_env_secret`, `_configure_api`, `_setup_test_database`, and `_register_resources`.
- Removed hardcoded `mimetype='image/jpeg'` from the `send_file` call in the image endpoint. - Now rely on Flask’s `send_file` to determine the MIME type automatically.
… package Update all test modules (`test_auth.py`, `test_database.py`) to import `Group`, `GroupManager`, `Permission`, `RBACManager`, `Role`, `SessionManager`, `User`, `UserManager` from `plantdb.commons.auth`. - Replace imports of auth classes and session managers from the old `fsdb.auth` path with the new `plantdb.commons.auth` submodules (`auth.models`, `auth.rbac`, `auth.session`).
- Simplify `_parse_url` signature, remove unused `allow_private_ip` and `validate_host` parameters and associated checks - Add `get_logger` import and remove duplicate import - Introduce `_build_ssl_context` helper to create SSL context, CA certificates and verification mode - Introduce `_configure_http_pool` to build a reusable `urllib3.PoolManager` with retries, timeout and SSL settings - Add `_handle_redirects` to manually follow redirects, validate hostnames and resolve relative locations - Re‑implement `is_server_available` to: - parse URL once, - build retry strategy, - configure SSL, - create HTTP pool, - perform request via `_handle_redirects`, - construct `ServerCheckResult` with `n_redirects` and status handling - Update docstrings and examples to reflect new behaviour and removed parameters
…se.py` - Add `?download=1` query for Zenodo URLs that lack the raw‑file flag. - Wrap `requests.get` with a custom `User‑Agent`, `timeout`, and `raise_for_status`. - Guard against empty keep‑alive chunks and update the progress bar using the actual chunk size. - Verify that the total size matches the downloaded progress; raise `IOError` on mismatch. - Re‑order the `tqdm` import and remove the duplicated import.
- Enable forward references with `from __future__ import annotations` - Import `TYPE_CHECKING` and conditionally load `FSDB`, `Scan`, `Fileset` only for static type checkers - Add explicit `-> None` return types to all `__init__` methods - Annotate constructor parameters with `FSDB`, `Scan`, `Fileset` types for clarity in `src/commons/plantdb/commons/fsdb/exceptions.py`
- Change `PLANTDB_API_*` variables to `PLANTDB_*` throughout `plantdb_client.py` and `rest_api.py` (`PLANTDB_HOST`, `PLANTDB_PORT`, `PLANTDB_PREFIX`). - Update environment‑variable lookup to use `os.getenv` instead of `os.environ.get`. - Adjust default value handling and type conversion for `PLANTDB_PORT`. - Update documentation comments to reference the new variable names. - Remove debugging `print` statements from the `scans_url` helper. - Update docstrings and parameter defaults that previously referred to `PLANTDB_API_PORT` to now refer to `PLANTDB_PORT`.
…ent' pyproject.toml
jlegrand62
force-pushed
the
feature/authentication
branch
from
89f59e5 to
a040982
Compare
- Insert `db.login('admin', 'admin')` in `fsdb_import_images.py` before `db.create_scan` to ensure proper authentication.
- Add `--mount=type=cache,target=/var/cache/apt` and `--mount=type=cache,target=/var/lib/apt` to apt update steps in builder, test, and base stages - Add `--mount=type=cache,target=/root/.cache/pip` to pip install steps in builder and test stages - Update `RUN` commands to include the new cache mounts and maintain lock‑sharing semantics
- Added `UNITTEST_MODE` variable in `docker/run.sh` and defaulted to `false` - Introduced `--unittest` flag to enable unit‑test mode - Implemented `run_unittests` helper that discovers and runs tests in `src/server/tests`, `src/commons/tests`, and `src/client/tests` - Updated `main()` to execute `run_unittests` and exit when `UNITTEST_MODE` is true
- Rename `docker-build` and `unit-tests` jobs into a single `build-and-test` job - Consolidate Docker build, test, and cleanup into one job with sequential steps - Use `ubuntu-22.04` as the runner instead of `ubuntu-latest` - Move image removal into the main job with `if: always()` - Update trigger section to `on: pull_request:` for clarity - Add descriptive comments to each step in `.github/workflows/pull_request.yml`
- Update the `on:` section of `.github/workflows/gh-pages.yml` to trigger on `release` events of type `published`. - Add a `workflow_dispatch` trigger to allow manual runs from the Actions tab. - Remove the previous `push` trigger.
- Install `client/[test]` dependencies after `server/[test]` in `docker/Dockerfile`. - Add `nose2` test run for `client/tests/` after server tests. - Remove the pip cache mount from the test RUN step.
- Update `build-and-test` job in `.github/workflows/pull_request.yml` to build Docker image with `--target test` and tag `plantdb-test:${{ github.sha }}`.
- Remove the now‑redundant unit test step; tests run during the Docker build stage.
- Adjust cleanup step to remove `plantdb-test` image instead of the old `plantdb` image.
- Update the `docker build` command in `.github/workflows/pull_request.yml` to include the `-f docker/Dockerfile` flag.
|
Looks like there are a few issues preventing this PR from being merged!
If you'd like me to help, just leave a comment, like Feel free to include any additional details that might help me get this PR into a better state. You can manage your notification settings |
- Copy the `src/client` directory into the Docker image at `client/` so that client code is available during the build and subsequent test stages.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Key changes
api_endpoints.pyand refactored URL helpers to useada-url. Simplifiedbase_urlsignature and removed the port default.urllib3, added SSRF protection, custom SSL context, redirect handling, and acert_pathoption.*ExistsErrorexceptions for scans, filesets, and files.plantdb.server&open3d) inplantdb.client’spyproject.toml.pyproject.tomlfiles; switched to>=3.8.PLANTDB_API_*environment variables withPLANTDB_*(HOST,PORT,PREFIX) and switched toos.getenv.