[CVAT-M2] Updates for the annotation experiment (humanprotocol#1780)

* Fix job event handling when no assignments exist

* Implement boxes from points task creation

* Refactor cloud storage api

* Implement job downloading

* Estimate max bbox

* Little refactoring

* Copy some shared code from excor to recor

* Refactor task creation and export

* Implement boxes from points validation

* Integrate SDK updates

* Fix escrow access

* [Exchange/Recoring oracles] Add GCS support

* Remove local dev mocks

* Remove some extra changes

* Update tests

* Update some tests

* Remove extra launcher change

* Refactor code

* Add google-cloud-storage dependency

* Fix exception classes

* Set env variables in docker-compose.test.yml

* [CVAT] Points to boxes task (humanprotocol#1560)

* Fix job event handling when no assignments exist

* Implement boxes from points task creation

* Refactor cloud storage api

* Implement job downloading

* Estimate max bbox

* Little refactoring

* Copy some shared code from excor to recor

* Refactor task creation and export

* Implement boxes from points validation

* Integrate SDK updates

* Fix escrow access

* Remove local dev mocks

* Remove some extra changes

* Update tests

* Update some tests

* Remove extra launcher change

* Use virtual hosted bucket style

* Fix creating CS in CVAT

* Fix tests

* Add basic implementation for skeletons from boxes

* Refactor and fix some errors

* Implement job uploading

* Implement downloading

* Fix extra gt boxes in merged results from excor

* Implement skeleton matching

* Fix labels in merged annotations

* Update project events handling in excor

* Improve oks sigma comment

* Remove local testing assets

* Fix label mapping

* Add extra skeleton validations in manifest

* Update .env template

* Update several comments

* Fix failing tests

* Update enum name

* Add more annotation validations for skeletons

* Extend input annotations validation

* Fix quality computation, refactor

* Unify assignment accuracy checks between different job types

* Refactor GT downloading for validation

* Add gcs support - fixes (#1)

* Update code formatting

* Update poetry lock

* Refactor

* Use dict for gcs file contents

* Update class name

* Add backward compatibility for data bucket env var

* Remove extra changes

* Remove extra import

* Fix enum name

* Update bucket access info parsing

* Fix error type in cloud provider parsing

* Update tests

* Add job annotation mode param into assignment links

* Add CD trigger for experimental cvat oracles

* Update code formatting

* Fix test

* Format code

* Fix bucket uses

* Fix result sending

* Remove extra changes

* Fix extra code

* Remove unused code

* Align enum naming convention

* Use relative paths in bucket in task creation

* Improve error messages

* Improve skeleton and bbox validation

* Fix escrow manifest downloading

* Update tests

* Fix tests

* Implement general image bans

* Fix import

* Make default gt ban threshold more strict

* Fix comparison for absent points

* Fix comparison for omitted points in jobs

* Finish escrows with too many unverifiable assignments

* Check if an increased healthcheck interval will fix unhealthy container (humanprotocol#1700)

* Fix validations for boxes from points task creation

* Clean code

* disable roi estimation for unreliable cases

* Fix manifest parsing

* Remove m0 launcher stubs

* Remove m0 rep or stubs

* Make max assignment time optional in manifest

* Make label type fully optional in manifest

* Fix test

* Enable and fix a disabled m1 task creation test

* Don't fail on too many unused GT in boxes from points

* Make joints optional in skeletons from boxes manifests

* Review gt exclusion errors

* Fix RecOr tests

---------

Co-authored-by: maya <[email protected]>
Co-authored-by: Ivan <[email protected]>
Co-authored-by: Dzeranov <[email protected]>

Author: 4 people

packages/examples/cvat/exchange-oracle/src/core/manifest.py

@@ -76,7 +76,7 @@ class SkeletonLabelInfo(LabelInfoBase):
     ]
     """
 
-    joints: List[Tuple[int, int]]
+    joints: Optional[List[Tuple[int, int]]] = Field(default_factory=list)
     "A list of node adjacency, e.g. [[0, 1], [1, 2], [1, 3]]"
 
     @root_validator

packages/examples/cvat/exchange-oracle/src/handlers/job_creation.py

@@ -352,7 +352,7 @@ def _validate_gt_annotations(self):
 
         if (
             excluded_gt_info.excluded_count
-            > excluded_gt_info.total_count * self.max_discarded_threshold
+            > ceil(excluded_gt_info.total_count * self.max_discarded_threshold)
         ):
             raise TooFewSamples(
                 "Too many GT boxes discarded, canceling job creation. Errors: {}".format(
@@ -512,7 +512,7 @@ def _validate_skeleton(skeleton: dm.Skeleton, *, sample_bbox: dm.Bbox):
 
         if (
             excluded_points_info.excluded_count
-            > excluded_points_info.total_count * self.max_discarded_threshold
+            > ceil(excluded_points_info.total_count * self.max_discarded_threshold)
         ):
             raise TooFewSamples(
                 "Too many points discarded, canceling job creation. Errors: {}".format(
@@ -614,7 +614,7 @@ def _prepare_gt(self):
                         sample_id=gt_sample.id,
                         sample_subset=gt_sample.subset,
                     )
-                    excluded_gt_info.excluded_count += 1
+                    # Don't need to count as excluded, because it's not an error in annotations
                     continue
                 elif len(matched_skeletons) == 0:
                     # Handle unmatched boxes
@@ -650,7 +650,7 @@ def _prepare_gt(self):
 
         if (
             excluded_gt_info.excluded_count
-            > excluded_gt_info.total_count * self.max_discarded_threshold
+            > ceil(excluded_gt_info.total_count * self.max_discarded_threshold)
         ):
             raise DatasetValidationError(
                 "Too many GT boxes discarded ({} out of {}). "
@@ -1327,7 +1327,7 @@ def _validate_skeleton(skeleton: dm.Skeleton, *, sample_bbox: dm.Bbox):
 
         if (
             excluded_gt_info.excluded_count
-            > excluded_gt_info.total_count * self.max_discarded_threshold
+            > ceil(excluded_gt_info.total_count * self.max_discarded_threshold)
         ):
             raise TooFewSamples(
                 "Too many GT skeletons discarded, canceling job creation. Errors: {}".format(
@@ -1430,7 +1430,7 @@ def _validate_boxes_annotations(self):
 
         if (
             excluded_boxes_info.excluded_count
-            > excluded_boxes_info.total_count * self.max_discarded_threshold
+            > ceil(excluded_boxes_info.total_count * self.max_discarded_threshold)
         ):
             raise TooFewSamples(
                 "Too many boxes discarded, canceling job creation. Errors: {}".format(
@@ -1580,7 +1580,7 @@ def _prepare_gt(self):
                         sample_id=gt_sample.id,
                         sample_subset=gt_sample.subset,
                     )
-                    excluded_gt_info.excluded_count += 1
+                    # Don't need to count as excluded, because it's not an error in annotations
                     continue
                 elif len(matched_boxes) == 0:
                     # Handle unmatched boxes
@@ -1620,8 +1620,8 @@ def _prepare_gt(self):
             )
 
         if (
-            len(skeleton_bbox_mapping)
-            < (1 - self.max_discarded_threshold) * excluded_gt_info.total_count
+            excluded_gt_info.excluded_count
+            > ceil(self.max_discarded_threshold * excluded_gt_info.total_count)
         ):
             raise DatasetValidationError(
                 "Too many GT skeletons discarded ({} out of {}). "

packages/examples/cvat/recording-oracle/src/core/manifest.py

@@ -76,7 +76,7 @@ class SkeletonLabelInfo(LabelInfoBase):
     ]
     """
 
-    joints: List[Tuple[int, int]]
+    joints: Optional[List[Tuple[int, int]]] = Field(default_factory=list)
     "A list of node adjacency, e.g. [[0, 1], [1, 2], [1, 3]]"
 
     @root_validator