v7 comments (UMD)

[carlosdcastillo]

Here are some comments about JANICE v7:

1. With respect to janice_template_get_feature_vector: we would very strongly prefer if the library didn't add specific semantics to the descriptor (for example the user being able to retrieve the 128-float array for a given UMD descriptor) we would prefer features are not in circulation in a way that the user can feel compelled to modify them or compare them in suboptimal ways. There are ways to extract the descriptor from v6, it is just that they're not API sanctioned, and we prefer the v6 status quo.

2. The core functionality of JANICE needs to be exercised by the harness. This includes:

   • Clustering: which has been commented out from the compilation for a while now. The program can be modified to compile after a few changes but its quality (error checking, command line parsing, multi-gpu parsing, etc) is not the same as janice_enroll_detections, for example.

   • Fine Tuning: which has never really had a JANICE-generation harness program,and the fine tuned models should be able to be used in the other programs of the pipeline.

3. We would prefer a OpenCV 2.4 compatible version of janice_io. We're not saying 2.4 only, just 2.4 compatible, it should compile with 2.4 if it can't find 3.3+ and it finds a 2.4.

4. The spec and the pdf (in the docs/) directory should be updated. The pdf should be updated each time the documentation changes. It should be possible to swing by the JANICE github site and download the latest spec.

5. The divergence between a high grade datasets (like CS4, CS5 and CS6) and the harness should be decreased. The harness should be a high grade component, too. If an image doesn't exist, too bad, it should not stop a 48-hour run, and if it must it should be easy to just continue the run, for example. Running a high grade JANUS dataset with the harness should be a routine transaction, as opposed to a painful one.

6. We've been a long time advocate of keeping the number of calls as low as possible. This has been an unpopular position and there's been a proliferation of calls. Number of calls in janus (v2): 18, janice (v6): 61, janice (v7): 66:

   • Is there still a use case for non-batched calls?
   • Is there really a need for so many serialization calls, to both memory and file?

[JordanCheney]

@carlosdcastillo Thanks for the feedback! Comments below-

1. There is some background for why IARPA would like to see this function explicitly in the API. Could you please follow up with Chris and Lars? I will keep / edit / remove this function based on that conversation
2. Both clustering and fine-tuning will be part of the harness before v7 is released
3. Would you be willing to submit your janice_io implementation with OpenCV 2.4 into the repository? I'd be happy to have 2 (opencv2_io and opencv3_io) with a CMake switch between them
4. This will be fixed before release
5. To clarify, you would like the harness executables to be more tolerant of errors. I can move to a "log and continue" paradigm for error handling in the harness, with a flag for immediate failure on an error (the current behavior). Will that solve this problem?
6. As the complexity of the different use cases has grown (detect+enroll, wild evaluations etc.) more calls through the API have been added. Also, moving from C++ -> C increased the number of required calls. To your specific points,
   • Non-batch calls are the easiest entry point for new users. With the program transitioning, I think that is a significant point for keeping them.
   • The serialization calls in memory are useful for passing JanICE objects into data stores and between machines. Serialization to a file is the easiest for new users and single machine architectures. It also solves the case where a large object (say a gallery) occupies more than half of a systems RAM. In that case, in-memory serialization is probably not possible, but the object can still be written to disk.

[taa01776]

I like the suggestion on #5 for a flag to control error tolerance in the harness executables. When I'm running the test harnesses to get results for a data call, most likely I don't want to tolerate errors, but otherwise I likely do.

I've always assumed that someone building a production system would be coding to the API directly, rather than stringing together harness invocations, which leaves proper error handling up to them. One can imagine cases where that might not be the case, or where the production system started from harness code because that's known to call the APIs correctly. Building in some error tolerance is essential in either case.

[taa01776]

Related to the error tolerance issue, is it too late to provide some method other than the log file to report errors back to the calling application (particularly when processing a batch)? As UMD has pointed out, there will be cases where we really want to continue running, so throwing an exception that's caught by the API user is difficult--we don't have a clean way of returning state so processing could be resumed if desired. Maybe the API user can supply an error function for the API implementation to call? Not exactly sure what the signature would need to be, but this could give the user the ability to terminate processing or not, and to report the error directly, rather than requiring a scan of the log file.

[JordanCheney]

I just pushed an update to error handling for batch functions. The batch functions are now expected to return per-element error codes to indicate success or failure at a more granular level. Users can also specify a batch policy (abort early on errors or always finish) to get their desired behavior. I thought this was cleaner than a callback for error handling. If you both agree I will update the harness accordingly

[JordanCheney]

I think the points raised here have been resolved. Please reopen if you disagree

[carlosdcastillo]

How would we go about giving you our OpenCV 2.4 implementation of janice_io? I believe you already have it, can you distribute that in the main repository?

[JordanCheney]

Hi Carlos, If you can make a PR with a new implementation, opencv2_io, similar to here- https://github.com/Noblis/janice/tree/master/implementations/opencv_io I think that would be best. Noblis can take on maintenance once it's committed.

We do have your source code including your opencv implementation but I'd prefer you or someone on your team commit the relevant portion so there's no confusion about what is supposed to be included vs. what is proprietary.