Support arbitrary property lookup for creating "resource" type values

[zerocrates]

We have arbitrary-property lookup in the module already, for finding the parent item for a media row, or finding a set to put an item row into.

We can use the same lookup functionality to allow users to create "resource" type values (internal links). Currently this only works via looking up the Omeka S internal ID for a resource, which means an extra step of doing separate imports and correlating IDs.

[pbcGIS]

In case the title of this issue is not clear, an alternate description: It would be useful if a new item created by this module could include among its properties a link to another item. Currently there is an option to link a new item to to an existing item-set. In the existing function, the item-set is itentified by one of its properties using the Resource-Data dropdown.

It would be great to have an additional option in this same Resource-Data dropdown that could specify an Item identifier. The result would be that the new item created by a row in the CSV file would be associated with an existing item as a linked resource.

For example: My Omeka-s site has a lot of items of class: Location each of these describes a building, address or geographic feature. Now I create a CSV import table that creates a bunch of items of class: Document. Each document may be related to a location. The resource template for documents includes a property named "Location" that accepts an existing Omeka item as a linked resource. The CSV table has a column named "Location" which, for each row includes a value which is the DublinCore:Identifier for one of my existing location items. The new item created by my CSV table includes a link to an existing Location item, and the Location item includes a reference to the new document item in its list of Linked Resources.

I believe that this feature could be added to the UI and documentation for the CSV import tool very simply (I think, if the "Item Set (by selected Property) also allowed items to be referred to. It would be up to the user to assure that the property specified contained unique references. In case of a non-unique property, the operation could link to the most recent item, similar to the Advanced Update function, or perhaps it would link to any item having that property.

This would be a very powerful feature for using the CSV import tool to build semantic networks. Thank you.

[Daniel-KM]

The feature is added in #171 / #172 . The linked resource are not added in the image above, but in the data type of the column (wrench). The property is defined in the tab "basic".

Note: It is no more possible to import multiple rows in batch, since the identifiers are created and checked by row. Maybe a future improvement will be to load and check all the csv first.

[zerocrates]

I realize now I haven't actually commented on the several associated PRs here, so I'll fix that now.

I don't think I'm in favor of the general system that's used here, where the existing property setting used for lookup for updates gets overloaded for use in this case also (and therefore promoted from Advanced to Basic). I've always envisioned this feature as working through an additional option in the "wrench" menu that becomes available when you select a "resource" data type, so it could be different for every column.

[pbcGIS]

Dear Zerocrates
I am setting up an application for a client that requires this "Link-to-item" capability. I would like some advice on how I might approach having this capability added to the Omeka team supported CSV Import Plugin. We have money to pay for development but would like specific advice about the Omeka team's design concerns.
We are also looking for a way to save or hard-code mappings for re-use.
Thank you. Our pilot app: CDASH: Cambridge Digital Architectural Survey and History

[zerocrates]

@pbcGIS It's an interesting looking site that you have there.

Can you clarify exactly what the client's looking to do, just so we make sure we're all on the same page that this feature is required for that?

Saving of mappings has been a vague wish-list item for a while also: a few things save but we kept out some partial "saving" code from prior versions with the idea just a "save" and "apply" type system that could set all the options would be better. I've imagined this as something that would just save JSON for the form state and then would possibly reapply it client-side with JS when a user wants to reuse it.

[pbcGIS]

Hi John, After writing... I see that this clarification is on the long side. If it would be easier I'd be more than happy to talk with you on the phone (617) 576-9072 or arrange a video/screen sharing session to show you the amazing potential of extending the CSV Import Plugin to facilitate the development and management of sites like CDASH. . .... At this current (final pilot) stage, the Cambridge Digital Architectural Survey and History (CDASH) <https://chcomeka.azurewebsites.net/omeka-s/s/cdash/page/home> , includes two resource templates, representing Places and Documents. Places can be Monuments, Named Buildings, Trees. 90% of place references are simple street addresses. Document Items are related to Places. In a later phase, CDASH may be extended to include item templates for People and Events -- which will also include references to Places and vice-versa. This sort of multi-dimensional network of items might a be generally useful for local historical commissions. I have been very inspired by the ways that Omaka-S supports this application! As we prepare the production phase, I am taking care to document and publish resources that may help others learn from our experience at github.com/c-dash and a sort of a blog <https://c-dash.github.io/Documentation/home/index.htm>, which one day may help similar orgs build this sort of thing with Omeka. The physical collection behind CDASH is organized in a set of 13 file cabinets; each drawer filled with folders, each marked with a reference to a place. Most of the place references are street addresses. Some are Named Buildings or other Landmarks. Each of the folders contains a lot of miscellaneous ephemera as well as hard-copy survey and research forms created by two systematic building inventories carried out in the 40s and 70s. Mixed into the folders, you will find biographical sketches as well referring to people who lived at the address in question. We came up with a scanning methodology that preserves the folder names and a document-page index. Scanned document pages are sorted into document types by means of a simple drag and drop interface. This procedure creates rought metadata including document titles, document types, keywords, etc. Another semi-automated process assigns Lat-Lon locations and other spatial attributes to each place reference including association with historic districts. About 90% of the place references are street addresses, which geolcate very precisely thanks to Cambridge GIS data, Some addresses refer to streets that have been renamed or re-aligned. . Our semi-automated geolocation assigns rough locations to these with a low precision score. More details on the automatically generated schema of place and document types can be found on the CDASH home page <https://chcomeka.azurewebsites.net/omeka-s/s/cdash/page/home>. A more detailed description of the data model and workflows can be found on our CDASH Slide Show <https://pbcgis.com/cdash/cdash_slideshow.pdf>. Our semi-automated cataloging workflow is carried out on batches of roughly 100 folders / 1000 items per batch. Three CSV files are produced for each batch for Place items, Document Items and Media files. There will be about 200 batches, In these CSV files Document Items are related to Place Items through the DC:Identifier, because in the import stage, the place items do not exist in Omeka. -- Thus the need for Item-to-item relations in the CSV Import Module. The CDASH map <https://chcomeka.azurewebsites.net/omeka-s/s/cdash/page/home> interface provides a marker for each Place item. Clicking a place brings up the Place's show page and a list of the Document Items that refer to this place. When we move to routine maintenance and accession, Historical Commission Staff will adjust existing and add new place locations using the interfaces provided by the Omeka-S Mapping module. With this architecture, the location for a Place may be adjusted in Omeka without worrying about adjusting the locations of each related Document Item. A few years ago, I paid Daniel Berthereau to add the Inter-Item reference capability to the CSV Import module. This allowed us to import the ten batches of places and documents and media for our pilot. It worked well, but we found that filling out the complicated mappings was subject to errors and inconsistencies that were not always discovered right away. There are lots of fields and data types involved. As we move ahead to production, it would be worthwhile to try to implement a means of saving and reloading the mappings, as you described. New documents will be added using the same drag and drop rough-sort procedure resulting in the same CSV manifests and rough metadata for the item types as described above, and will take care of the (often multi-page) media uploads for the document items. It would be especially handy to be able to reload field mappings during these transactions, I am trying to build up my PHP skills by trying to figure out how CSV Import works (working form Daniel's branch) My dream would be to learn enough to submit a pull request that includes the reload feature and addresses your issues with Daniel's implementation of the Relate_Item_By_Identifier. I have learned enough Unix, Docker, Javascript, CSS and Python to create CDASH, but after several weeks od Laminas tutorials and trying to crack Omeka modules I am beginning to wonder if PHP/Laminas may be my Waterloo as a shade-tree hacker. The going is slow -- but I have not given up yet! Maybe I've made a case that the Link-Items-by-Identifier and Reload Mappings features would be generally useful enough to get added to the feature queue for CSV Import. If necessary, I would be willing to spend some of my pay for this project to fund the development. In any case, it would be very helpful to do some planning with you so that the mods could potentially find their way into the supported CSV Import Module. So here we are. . I would be very glad to have your advice on 1. The architecture of our CDASH/Omeka schema and production workflows 2. Recommendations for modifying CSV Import (php viewpoint) 3. Recommendations for getting the mods implemented in the supported module (design / github viewpoint) Maybe the capability of doing repeated batch uploads and creating multi-dimensional networks of items would be worthy of entering into the Omeka-S feature queue! 🙂🙂🙂 I understand that this explanation is long and complicated. I'd be more than happy to talk with you on the phone (617) 576-9072 or arrange a video/screen sharing session to demonstrate the amazing potential of extending the CSV Import Plugin to support networked items! Thanks for reading this and for your excellent ongoing work on the fantastic Omeka project.

…

--pbc Paul B. Cote Geographic Information Services cultivating spatial intelligence [tm] <http://www.pbcgis.com/> pbcGIS.com <http://pbcgis.com/>

On Fri, Jul 15, 2022 at 5:09 PM John Flatness ***@***.***> wrote: @pbcGIS <https://github.com/pbcGIS> It's an interesting looking site that you have there. Can you clarify exactly what the client's looking to do, just so we make sure we're all on the same page that this feature is required for that? Saving of mappings has been a vague wish-list item for a while also: a few things save but we kept out some partial "saving" code from prior versions with the idea just a "save" and "apply" type system that could set all the options would be better. I've imagined this as something that would just save JSON for the form state and then would possibly reapply it client-side with JS when a user wants to reuse it. — Reply to this email directly, view it on GitHub <#162 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABIFNNUVVPUFHTAARNQIOW3VUHHPZANCNFSM4H3WKBTA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[pbcGIS]

Apologies for awkward novice git-hub etiquette. Criticism is welcome.
The client's collection includes items regarding places (e.g. buildings, monuments) that have locations; and items that reflect documents related to place items. Our workflow involves creating thousands of these in batches. At the time of creation, there is no way of knowing the internal ID of the place items. It is difficult to learn/use internal IDs even after the items are created. For more info about the item creation workflow you can look at our slide show.
I would be happy to help modify Daniel's design and pull request if you advise.
I've created another modification to save and re-load mappings that I could submit as another issue or pull request if this would be helpful for the community.
Your good work is very much appreciated!

[zerocrates]

My main concern with the previous PR was that it worked essentially "globally," it introduced a setting that changed how all "link" lookups for the import would be made (if I'm remembering it correctly). Instead, it seemed to me that this should be something that was a property of the mapping (so it could vary from column to column: one might map to dcterms:identifier, another to title, and so on, within the same import). I think the main question on that kind of feature was really more of a UI one, as in where we would unobtrusively put such an option in the mapping interface.

The "global" version does sidestep some of those problems, so it'd also be good to know if that kind of "advanced setting" that just changes all the link mappings to use a single specified property instead of the resource ID to look up would be sufficient for your needs.

We have made some small changes more recently to make getting at the resource IDs easier (for one, they're displayed in more places), but that obviously doesn't eliminate the usefulness of this feature.

As for the save and reload of mappings: that should definitely be discussed on its own issue or PR, so I don't want to get too much into it here. I would say there's definite interest in that kind of feature, though I'm not totally sure about the current method you're using from a quick look at the branch.

[pbcGIS]

Thank you for the clarification. Would the approach illustrated in the 2nd message in this thread address your concern? That approach would put a second property drop-down under the Item-Specific Data panel of the mapping form. So the procedure for mapping any given column in the source table as a foreign key linking to another item (just as the existing one does for item sets.

This may be straight-forward to implement by repeating an existing pattern and also more economical in terms of documentation. In fact, if we were willing to assume that a project that implements this feature would take care to not repeat identifiers between Item Sets and Items, maybe the item linking capability could be implemented using the existing pull-down, and the link would be made to the most recently created resource having the value for the specified property matching the value of the field in the source. This modification would not change the user interface, except for the explanatory tool-tips. And the impact on the documentation would be an extra sentence or two in the explanation of that item-specific data panel.

The following idea may be a stray tangent: I wonder if it makes sense to conceptualize item sets as ordinary items that are targeted by relations from other items. The reason that I designed CDASH Place Items this way is that it allows this type of item to serve the function of item set, but also having the possibility of having a location and associated media of its own. This approach to modeling item sets as items seems to hew closely to simple relational architecture. Note that adopting this idea does not require any change to the architecture of Omeka.

Thanks again. I can't promise that I have the skills to implement this, but agreeing on a design goal would be a useful step in in the process.

[zerocrates]

The second message in this thread, meaning your post with the screenshot?

I'm not quite sure about your description of what you planned exactly, but using that dropdown is what I'm thinking of basically. You can't just put a second dropdown under the Item Set dropdown in Item-specific data though. Because these linkages are specific to a property, the choice needs to be something that comes along with also choosing a property. I guess you could just "map" it as a separate thing that doesn't do anything on its own, but we don't generally do that. Further, it only applies when you're mapping to a "resource" data type, an option you set in the "wrench" menu. One final little issue: this should also work for other resource types so it really shouldn't be in the "Item-specific" area.

So where this should probably end up is inside the "options" sidebar you get to by clicking the wrench icon for a column. That's where other things that just affect the property mappings in particular go, so it's the right place probably.

[zerocrates]

I have an implementation of the "option-based" solution on the branch "resource-identifier-property".

It adds a new option named "Resource identifier property" to the options sidebar. Note that this option only takes effect if you're using the "Omeka resource" data type for the column also.