unexpected (incorrect) info in the search summary table and citation record table

[puzzledhedgehog]

Hi folks, great tool, I am having fun exploring it. I ran into something weird though. Maybe the solution will be obvious to experts? ;)
I was using an in-progress project to go through the source analysis vignette. It looked like everything was ok re the file import and tagging (once I realized that I couldn't use underscores in the cite_sources tag), but the number of results in my "final" tag seems to be different in different spots.
93 in the ris file
93 in the report about the import
But 89 in the citation record table download
61 in the search summary table total records (which are footnoted as being after deduplication)
And I got a warning (btw very useful to highlight it) that there were 28 records in the "final" tag that didn't match with records in the "search" tag.
I doublechecked -- I'm sure that I imported all search results files that ended up getting screened, and that the "final" file I used has only the records identified via the searches, not the additional ones from citation chaining. So where the 28 "new" records appeared from, I'm not sure.
I have a clue, though -- the search summary table reports that no final records came from Embase or Medline or ProQuest Dissertations and Theses, which seemed odd to me, so I checked.
I found one record (and stopped looking after that) that definitely is present in the Embase search results file which I imported, and present in none of the other search results ris files, and also present in the final/included ris file. Maybe there is a subtle difference that prevented it from being identified as a match? I attach the two matching records.
Similarly, some of the records which are easily identified as mysterious "new" ones in the citation record table are clearly from the ProQuest Dissertations and Theses export file, which I imported and tagged, but are not identified as matching.
Some of cases of "new" (ie unmatched) records were ris records for clinical trials -- maybe from trials registries or maybe from CENTRAL/Cochrane Library, I haven't done the legwork to figure out. I wouldn't be surprised if these trials records didn't have enough info/the right info for duplicate pairs to be identified....
Maybe I did something wrong with the import? Maybe Covidence stripped out/changed some metadata during screening and that's why so many pairs of "search" and "final" records didn't match?
Any insights would be welcome!
Concepcion_record_final.txt
Concepcion_record_search.txt

Thanks
Kate Nyhan

[TNRiley]

Hi Kate,

It sounds like the metadata associated with the final citations was different enough from the source citation metadata, which definitely could throw things off. It's a bit difficult to tell though and I'd need to see it myself and look at some of the data. Would you be interested/able to either providing the .ris or doing a quick troubleshooting video call so I can see the outputs? You can DM me on Twitter @tlibriley

The 89 in the record level table seems odd, as that should be pulling anything with the cite_label you call. As for the search summary table, we're working to change that. If you look at the "source analysis across screening phases" vignette, there is a new precision_sensitivity_table. There is also a record_summary_table. Take a look at those, it will still be off for your data but may be helpful.

[puzzledhedgehog]

Hi Trevor, I can definitely help troubleshoot. Or at least provide the materials for you to troubleshoot ;) I'm not on Twitter anymore (instead, ***@***.***) but you can reach me by email at ***@***.*** All the ris files are in this Box folder: https://yale.box.com/s/c4ve4jmiyj5d8tn9irc4x8mdzfc2fc7c. It's set for everyone with the link to be able to download. I will add a readme. The only output I saved was some figures and the citation record table (I think that's the right name -- the csv of the table where each included document is a row and you can see which of the input files that document was present in). I'd be happy to do a videochat -- I bet I would learn a lot from seeing you troubleshoot in real time. I use this scheduling link ***@***.*** but considering the time zone issue I could be flexible beyond what's offered there. Kate

…

On Wed, May 24, 2023 at 7:02 AM Trevor Riley ***@***.***> wrote: Hi Kate, It sounds like the metadata associated with the final citations was different enough from the source citation metadata, which definitely could throw things off. It's a bit difficult to tell though and I'd need to see it myself and look at some of the data. Would you be interested/able to either providing the .ris or doing a quick troubleshooting video call so I can see the outputs? You can DM me on Twitter @tlibriley — Reply to this email directly, view it on GitHub <#144 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIBYJN264LYUNTX5XAIRZ4TXHXTDNANCNFSM6AAAAAAYK72QHI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[TNRiley]

@puzzledhedgehog bringing them all into EndNote first can also help combine the WoS1 and WoS2, same with Scopus, and then bring them into CiteSource as a single .ris. I assume those are raw .ris exports that were limited by export size? Also, one last thing, take a look at the plot sandbox vignette. I recently added some new tables that you should like, they are an improvement over the citation summary table you used.

[puzzledhedgehog]

@TNRiley Yes, the final .ris, review_258535_included_endnote_20230523003130.ris, came out of Covidence.
About your suggestion "bring everything into EndNote first, grouped by source, and then re-export each individual source .ris. You can then try rerunning things in CS with these new .ris" -- when you say "everything" do you mean all the files that I tagged "search," or do you also mean the "final" file too? I'm not immediately seeing how comparing ris-files-exported-from-EndNote would make for better matching against the existing Covidence output.
Or, wait, do you mean, on the next project, do that EndNote step before deduplication and screening, in order to have better CiteSource performance later on, the screeners have eventually gotten down to the final included set?
Another question about this suggestion: is merely importing the files into EndNote, then exporting EndNote-generated ris, "enough"? Or should I also run the EndNote "update references" command?

"If you do this, add a new folder in your structure with all the .ris and I can grab them as well."
OK, that's pretty clear that you're talking about this troubleshooting exercise, not just future best practices. Here's a folder with all the EndNote-output ris: https://yale.box.com/s/ks3fngoityvhvmeyskic5jhoozq87kv6

Re the multiple WoS and Scopus files -- some of them were a workaround for export size limits, and some of them are actually different search strategies.

20220714 Web of Science results Choy file1.ris
20220714 Web of Science results Choy file2.ris
= one results set split into 2 files for downloading

20221005 Web of Science results file 1.ris
20221005 Web of Science results file 2.ris
20221005 Web of Science results file 3.ris
= one results set split into three files for downloading

But those two sets represent a first and second draft, so to speak, of the Web of Science search strategy. Between July and October we reflected on the research question and decided to conceptualize/operationalize one of the concepts a bit more broadly.

Thanks for all the input!

Kate

[TNRiley]

I ran the files you had in the endnote folder, keeping the WoS and Scopus variable searches separate, but you could also combine them. The new tables are in the next comment.

From the looks of it, I don't see any immediate issues. The rest of the plots looked good as well. I can revisit them again here soon, but see if you can run everything and check it out :) - This is the Rmd you can use, just replace file paths https://drive.google.com/file/d/1KUVbXGfCPa776FCNLVoQLru5uxcTPPQE/view?usp=share_link

[TNRiley]

Full tables

[TNRiley]

Running the citation_record_table to further analyze things. It looks as if there were 8 records that were in the final .ris that were not identified in a source .ris. - here is the .csv - https://drive.google.com/file/d/1uR_iAhbCviP7RrkMcVFM_BjK2mQ3ai8r/view?usp=share_link

Here are two of the eight - I believe that 7 of them are the clinical trials you mentioned and one is a dissertation. In terms of CiteSource, I believe it's working as intended, though I'll look into the single dissertation that isn't being caught - I'll also be sure to add to our documentation that records should be imported into EndNote/Zotero/Etc. and re-exported to synchronize variable metadata fields. Variables in metadata is always going to cause issues when deduplicating/matching, so If you would like to use these tables or other plots in a publication, slides, etc. it should be pretty easy to just ensure that these records have the most complete metadata as possible.

Aflague, T. A. F. (2015). Addressing childhood obesity through novel assessments, enhancement of summer enrichment programs, and surveillance in Guam. . https://www.proquest.com/dissertations-theses/addressing-childhood-obesity-through-novel/docview/2315248481/se-2?accountid=15172

NA, NA. (NA). Pacific Kids DASH for Health - Full Text View - ClinicalTrials.gov. . https://clinicaltrials.gov/ct2/show/NCT00905411

[TNRiley]

.ris files should first be imported into a record management software, combined if necessary for each source. Metadata should be checked and as accurate/complete as possible before exporting from the record management software and then imported into CitSource. The record_level_table can be used to check for any matches that were not caught when comparing source citations to screened or final citations.