Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable image recognition for photos added to the plant and moth surveys #292

Open
kitenetter opened this issue Sep 12, 2024 · 6 comments
Open
Milestone

Comments

@kitenetter
Copy link
Collaborator

Image recognition is currently offered for individual records and for the Species List Survey. We'd like to extend this to the Moth Survey and the Plant Survey (unless @sacrevert says otherwise).

I assume that #239 will apply to the Plant Survey.

@sacrevert sacrevert added this to the 6.2.0 milestone Sep 12, 2024
@sacrevert
Copy link
Collaborator

sacrevert commented Sep 12, 2024

I assume that this would be a user-configurable choice anyway.

Personally I would like something more than what I understand has been done for eSurveyor. AFAIK, in that case the image recognition results are presented alongside record cleaner type information. So, e.g., a high probability from the classifier might be presented alongside information that states that the species is unknown from the hectad or whatever.

I would prefer to go further than this, as I think we will still get situations where people just choose the highest values from the classifier probabilities, irrespective of other information. My suggestion is to also use pseudo-frequency information on the taxon in question from the database itself (e.g. as here and combine this with the probability coming from the classifier).

For example, if we had a species that the classifier claimed 0.95 confidence for, but it was only known from 0.001 of relevant records, then we would present the geometric mean of these as the adjusted confidence (i.e. exp(log(0.95)+log(0.001)/2) = 0.03). If we wanted to keep the current presentation, then this number could be presented as an extra result at least.

@kitenetter
Copy link
Collaborator Author

@sacrevert the main focus of this issue is that we offer image recognition in some parts of the app but not others. The questions about what information we give to users are important but separate. For this issue I just want to add consistency for users of the app.

On that basis, would you be happy for image recognition as it currently stands to be available within the plant survey?

@kazlauskis
Copy link
Member

@kitenetter there are two ways to do this.

a) Integrate the PlantNet classifier into the Plant survey, and add the NIA to the Moths survey.

b) #239, I continue using the proxy module and add it to these two new surveys. @JimBacon modifies the proxy module, which currently only uses the NIA results, to detect some suggestions being plants to give it another pass with the PlantNet classifier.

Option b) requires Jim's input, but is the easiest from my perspective and also improves the plant-related results to the other two surveys (single record and list of records) which rely on the NIA service for plant suggestions.

@JimBacon
Copy link
Collaborator

For info, the most recent version of the proxy module, https://github.com/Indicia-Team/drupal-8-modules-indicia-ai, does give access to PlantNet. It doesn't, iirc, do the double pass of using NIA to identify a plant and then Plantnet to refine the result.

@sacrevert
Copy link
Collaborator

Sounds like the proxy module is a good solution (fwiw, the double-pass strategy sounds inefficient anyway, as presumably there will be many photos of insects on plants, and passing these to Plantnet sounds like a waste of energy unless NIA is very good at guessing whether or not the plant is the subject).

@JimBacon
Copy link
Collaborator

And if this is explicitly for a plant survey it makes sense to go directly to PlantNet.

@kitenetter kitenetter modified the milestones: 6.2.0, NCEA app work Jan 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants