Standardizing attributions for display on scaife.perseus.org

[jacobwegner]

I've created this issue to track updates to the underlying attribution data that we're now extracting / displaying on scaife.perseus.org

Overview

I've extracted the existing attributions (from respStmt elements) and exported them to a Google Spreadsheet, OGL - First1kGreek Attributions. I can grant access to the appropriate persons within OGL to perform bulk edits to the data.

Once the preferred edits have been made to the spreadsheet, I will use the spreadsheet to bulk update the underlying XML files with the new attribution information and open a pull request.

If this workflow works well, we can do it for other OGL repos (and ideally any other repos contributing texts to scaife.perseus.org)

Desired data model

Here are a few samples of what the updated respStmt elements will look like:

Thibault Clérice, Lead Developer (University of Leipzig) 2015 - 2017

From https://github.com/OpenGreekAndLatin/First1KGreek/blob/master/data/tlg0062/tlg001/tlg0062.tlg001.1st1K-grc1.xml#L28

to:

<respStmt>
  <resp from="2015" to="2017">Lead Developer</resp>
  <persName ref="https://orcid.org/0000-0003-1852-9204">Thibault Clérice</persName>
  <orgName>University of Leipzig</orgName>
</respStmt>

Notes:

• We make use of from and to attrs to denote the timeframe of the resp.
• We set a person's ORCID in persName.ref

Simona Stoyanova, Project Manager (University of Leipzig), 2015, Project Assistant (University of Leipzig), 2013-2014

From https://github.com/OpenGreekAndLatin/First1KGreek/blob/master/data/stoa0146d/stoa001/stoa0146d.stoa001.opp-grc1.xml#L47

to:

<respStmt>
  <resp when="2015">Project Manager</resp>
  <persName>Simona Stoyanova</persName>
  <orgName>University of Leipzig</orgName>
</respStmt>
<respStmt>
  <resp from="2013" to="2014">Project Assistant</resp>
  <persName>Simona Stoyanova</persName>
  <orgName>University of Leipzig</orgName>
</respStmt>

Notes:

• We move from a single respStmt containing two resp elements to a 1:1 relationship between respStmt and resp
• when and from|to attrs denote the resp. timeframe

Gregory Crane, Leonard Muellner, Bruce Robertson, Published original versions of the electronic texts, Open Greek and Latin

From

First1KGreek/data/tlg0093/tlg005/tlg0093.tlg005.1st1K-grc1.xml

Line 12 in 3f5519b

<respStmt>

to:

<respStmt>
  <resp>Published original versions of the electronic texts</resp>
  <persName role="principal">Gregory Crane</persName>
  <orgName ref="https://www.opengreekandlatin.org">Open Greek and Latin</orgName>
</respStmt>
<respStmt>
  <resp>Published original versions of the electronic texts</resp>
  <persName role="principal">Leonard Muellner</persName>
  <orgName ref="https://www.opengreekandlatin.org">Open Greek and Latin</orgName>
</respStmt>
<respStmt>
  <resp>Published original versions of the electronic texts</resp>
  <persName role="principal">Bruce Robertson</persName>
  <orgName ref="https://www.opengreekandlatin.org">Open Greek and Latin</orgName>
</respStmt>

Notes:

• We move from a single respStmt containing multiple persName elements to a 1:1 relationship between respStmt and persName.
• We also include orgName in each respStmt

Implementation

Extraction process

Each row in the attributions-data worksheet corresponds to a set of URNs extracted from the underlying XML files.

There are "key" and "urn" fields which should not be modified and will be used to perform the bulk update.

Editing attribution data in the spreadsheet

I went through and made an initial pass to clean up the data. This involved fixing small typos in organization names, normalizing names (Mt. Allison vs Mount Allison, etc) and restructuring data to fit the desired model (discussed below).

The unique-* worksheets show uniquevalues for the resp, orgName and persName.

Ideally, we can standardize on "Proofreading" vs "proofreader" vs "Proofreading and CTS conversion" as appropriate. If proofreading and CTS conversion are two distinct responsibilities for a given text, I would suggest:

1. Adding an additional row beneath "Proofreading and CTS conversion"

2. Edit the original resp to Proofreading

3. Set the resp in the new row to CTS conversion

4. Copy the other relevant fields (resp, orgName and persName) to the new row

5. Leave a comment on the row so I can ensure that the urn and key fields are also populated.

There are also several instances where slight variants in a person's name are used, or resp possibly contains data better suited for orgName .

We should not delete any rows; if there are duplicate rows in the spreadsheet, we'll use the urn and key fields to de-duplicate data.

Bulk update process

Once edits have been finalized in the spreadsheet, I'll use the urn and key fields to map the edits back to the desired data model (see below)

I will also perform a reordering of the desired "proofreading / conversion" role(s) so that they are weighted before any other roles.

I'll open up a PR and link it back to this issue. The PR can be merged and then the updated attributions will be made available on scaife.perseus.org

Closing thoughts

I'm not sure if there is "template" for future XML files, but I would also be happy to take the examples in Desired data model above and integrate them into that template.

As long as the XML files have respStmt with resp and one of persName or orgName, we can extract attributions for display on scale.perseus.org.

[lcerrato]

Hi @jacobwegner
This is such a large undertaking, I hesitate to offer the first comments.

The working header template is here: https://docs.google.com/document/d/16fZThJUuTwJJKFiJgi1cLW0ONKeVnE3MdqMHVdCcmDM/edit#heading=h.c5otxj8nwtfd

Overall the header consistency is a challenge that extends beyond the credits. We know there is a lot of work to be done on the older file headers. As you note, we have inconsistent labels. Some describe roles, others describe tasks. (The practice of using the dates is not something we want to preserve.)

We need better labels that fully capture the contributions of the students, so we have to settle on the vocabulary. I don't know that we can do much with the older work. There are just some roles/labels there that we no longer use. I think much of the older stuff was boilerplate.

I prefer a lighter and streamlined header, so I would lean against splitting up a description such as "Proofreading and CTS conversion." I also like keeping Greg, Lenny, and Bruce together in their grouping — it makes for less bulk and more readability. If it doesn't work, that's fine.

I presume the file headers should read in order of the desired presentation, if so, that impacts how files are exported from Lace. I do not know if that affects Zenodo display — I believe we added the three top names as part of the bibliographic cleanup so that we would have a consistent set of names to appear at the top of our releases.

Will people be displayed as role/name/org or name/org/role? If it's the former, perhaps we could toggle the roles, so that

[jacobwegner]

@lcerrato Thanks for your feedback and the link to the existing header template–very helpful!

We need better labels that fully capture the contributions of the students, so we have to settle on the vocabulary. I don't know that we can do much with the older work. There are just some roles/labels there that we no longer use. I think much of the older stuff was boilerplate.

Part of my goal in providing the spreadsheet was to demonstrate some of the variance between labels and headers; I know the "hard" part is building that vocabulary, as you've said, but hopefully the bulk update process helps to simplify updating them once the vocabulary is established.

I prefer a lighter and streamlined header, so I would lean against splitting up a description such as "Proofreading and CTS conversion." I also like keeping Greg, Lenny, and Bruce together in their grouping — it makes for less bulk and more readability. If it doesn't work, that's fine.

I don't intend to enforce a "Scaife-specific" convention on the headers, and the readability issue is a good one to bring up.

In the Scaife attributions model, we tie each attribution to a person and or organization.

See for example, tlg0093.tlg005.1st1K-grc1:

If the desire is to have those principals and OGL as the first respStmt, would this be agreeable?

<respStmt>
  <resp>Published original versions of the electronic texts</resp>
  <orgName ref="https://www.opengreekandlatin.org">
    Open Greek and Latin
    <persName role="principal">Gregory Crane</persName>
    <persName role="principal">Leonard Muellner</persName>
    <persName role="principal">Bruce Robertson</persName>
  </orgName>
</respStmt>

I think we could account for that type of nested respStmt in our attribution model, and likely even "group" the principals into a single entry.

I presume the file headers should read in order of the desired presentation, if so, that impacts how files are exported from Lace.

That's correct, but I could also see us making use of the n attribute on a respStmt to control "weight", e.g.:

<respStmt>
  <resp>Published original versions of the electronic texts</resp>
  <orgName ref="https://www.opengreekandlatin.org">
    Open Greek and Latin
    <persName role="principal">Gregory Crane</persName>
    <persName role="principal">Leonard Muellner</persName>
    <persName role="principal">Bruce Robertson</persName>
  </orgName>
</respStmt>
<respStmt n="1">
  <resp>Proofreading</resp>
  <orgName>Mt. Allison University</orgName>
  <persName>Kirsten Mason</persName>
</respStmt>
<respStmt n="2">
  <resp>CTS conversion</resp>
  <orgName>Center for Hellenic Studies</orgName>
  <persName>Michael Konieczny</persName>
</respStmt>

(Where a respStmt without n would be sorted to the bottom of those numbered, etc).

So ideally I'd like to come up with a convention that works for you / OGL to allow the Scaife Viewer environment to extract attributions from the files directly, but in the case of the "principals" header and sorting of proofreading / CTS conversions, I could also see having a repository-level mapping configuration file (much like .zenodo.json where we could enforce the mapping).

Another way to say this is that we want to place as much control as possible of the display of attributions with the providing data source (so the Scaife dev team is not a bottleneck on updating mappings, etc).

With either the convention or the mapping file, I'd expect to end up with something more like this in Scaife:

(With the near-term goal of being able to click on Kristen's or Michael's name to see other works they have contributed to)

I do not know if that affects Zenodo display — I believe we added the three top names as part of the bibliographic cleanup so that we would have a consistent set of names to appear at the top of our releases.

I don't know for certain either, but it looks like .zenodo.json controls the Zenodo display directly.

Will people be displayed as role/name/org or name/org/role? If it's the former, perhaps we could toggle the roles, so that

I think you may have got cut off here, but currently we display name + org and then role:

[lcerrato]

@jacobwegner
Yes, I didn't finish my thoughts — too many revisions.

I would like other input on this but I like the middle display and the notion of adding the weighted attribute. My only concern on that would be that is an easy thing to miss in file review and it wouldn't be in the testing regime.

[jacobwegner]

@lcerrato I'd be happy to change the scope of this bulk update to just cleaning up / standardizing the existing attributions.

If they're standardized, I'd be happy to write small config file that could handle the "weighting" (e.g.):

• For resp values like Published original versions of the electronic texts, weight lower
• For resp values like Proofreading or Proofreading and CTS conversion or CTS conversion, weight higher
• For other resp values, keep the order in the file

That would keep you / other repo maintainers in control of the weighting without having to rely on keeping the attributes up to date.

[lcerrato]

@jacobwegner
Thanks. I've brought this to the group and we have some good progress on language and management. I'll update soon.

[lcerrato]

Hi @jacobwegner!
@ThomasK81 @brobertson @lmuellner @LucieSty @mkonieczny9805 @AlisonBabeu @gregorycrane @jtauber
So first resolution is that proofreading/ proofreader should read "Digital conversion and editing". Those with this designation should be displayed first.

I believe that others who have CTS conversion or something similar in the line should be "Digital editor" — (not entirely sure we settled on that). This would be the second level of display.

I can get to work spotting other issues.

One thing that @ThomasK81 mentioned is that while the group preferred the middle display example, (with Kirsten Mason first), the OGL itself is in parentheses with the three director names. Note on the last screen shot, the Org name appears first.

Would it be possible to have<orgName ref="https://www.opengreekandlatin.org">Open Greek and Latin</orgName> with a live link at the top of the list while still preserving the rest of the weighted order we propose? Is that practical in this stage or should it wait until we delve further into the header exposure? (We know that sidebar space is precious at this point and that there will need to be a separate jump off for other file info.)

If I/we want to regularize other information, it should be on the first tab, yes? (I saw some minor inconsistencies and started fixing them.) Edit: I see your notes above. I will make notations to my changes and then go from there.

Let me know the next steps.

[lcerrato]

@jacobwegner
Let's strike the "from" and "to" attributes.

[jacobwegner]

(As I mentioned to @lcerrato in chat, outside of the urns and key fields, we can just do the edits directly in the document, so I ended up just deleting the when, to and from columns)

[lcerrato]

@jacobwegner Thanks! Will resume next week when I'm less distracted.

[jtauber]

Should we just use the publication statement to get the OGL info rather than "hack" the OGL into the responsibility statement?

[jacobwegner]

Here's the latest draft of the widget:

• Publication statement is extracted and used as the first entry
• We regroup names underneath each "role" (resp in TEI terms)

[lcerrato]

@ahanhardt @brobertson
There is a Janan Assaly listed in these files as being affiliated both with CHS and Mount Allison. Which is correct?

[lcerrato]

@jacobwegner
I can't insert any rows. I just need to add some attributions to 5 files. All the same person.

[jacobwegner]

@lcerrato I have just removed the locks so you can add rows.

I hadn't been thinking about adding new entries from the sheet (just updating), but as long as you populate the urns cell for each entry and leave the key blank, that should work.

Happy to hop on a chat or call to discuss further.

[lcerrato]

@jacobwegner
I can add manually if that is best done in the next phase of work. I was actually just wondering if I wanted to do that anyhow in case I spot other things.