Editorial Management, Review & Submission

[gabestein]

Hi Folks,

We're starting to think deeply about the next steps for submission and review processes on PubPub, which are coming up soon on our roadmap. Given how complicated the subject is, we figured we'd try starting an open thread on it to collect your feedback and ideas. We'll be reaching out proactively to setup real-time discussions with a few communities we know are interested. If you'd like to be among them, please let us know here.

We have some initial ideas for what it could look like, but we'd actually rather hear from you first, so we don't accidentally bias your responses. What we'd love from you is answers to any (or all!) of the following questions that you're willing to share:

• What is the name and URL of your community?
• How do you describe yourself? (e.g. a journal, a preprint server, a blog, etc.)
• Briefly describe your community's submission/review process.
• What tools do you currently use to manage your submission/review/editorial management process?
• What are the top 2-3 most painful things about your current process, and why are they so painful?
• In an ideal world, what are the top 2-3 things that would make your process not just less painful, but exciting and maybe even...fun?

You can spend as much or as little time responding as you have time for, and any response is a good one — even if it's incomplete or doesn't respond to any of the above.

Thank you so much. We're really looking forward to building this out with you and continuing our mission to make PubPub an end-to-end solution for community publishing of all kinds!

[devinberg]

I am operating The Journal of Open Engineering (https://www.tjoe.org). We are very low volume, but have been going at it for about five years on PubPub.

Currently, we are using the annotation system for reviews. Reviewers can use individual highlights/comments to point to a specific item and can type a general review in the page comment area. This works ok, but can become cluttered when there are a high number of comments. Things that would help here:

1. The ability to archive comments (currently exists), but then resurface them again. Maybe hidden rather than archived so that the review comments could still be publicly accessible, but not in-your-face by default.
2. The ability to distinguish comments from invited reviewers from the general public. Again, maybe with a toggle to turn them on and off at the top of the page.

I also currently separately manage the invitation and status of reviewers separately (in a Google Doc :-( ) because I've found the current implementation of "Reviews" to be confusing and a little too much still in the air to be able to keep up with what works and what doesn't at different times. My requests for a process or workflow would include:

1. Allow an author to request a review
2. I approve the review process to go forward (this creates a release) [right now creating a release seems to close the review from what I can tell, which is not what I want]
3. I can invite reviewers from within PubPub (emails go out per a template) and author can suggest reviewers
4. reviewers get the request with an easy Accept/Decline/Suggest-alternate button
5. if reviewer accepts, reviewer comment on the article with comments somehow indicated as belonging to a review
6. author can reply to review comments and make edits (perhaps also tied to that comment to clearly show the author's reply and what they changed in response, maybe via a diff)
7. an editor can mark a comment as resolved, hiding it but not deleting it
8. when satisfied, the editor can mark a review as complete and submit a decision

Some of this is nearly possible with the current system, but not quite.

[writingpoli]

As pretty heavy users of PubPub who receive endless notes from potential contributors unclear how to use the platform to submit their work, I'm (along with the rest of the SAR Editorial Collective) excited to work with y'all on figuring this out.

One of the biggest issues is notifications -- both our receiving them when a new submission comes in and notifying contributors that we've followed up on their submission via comments.

I also started drafting this Pub a while back (which may be a bit higher-level than what you're looking for) and need to add to it (others are welcome to add notes/request access to edit and add their thoughts).

https://www.sareview.org/pub/xhin6k71

About San Antonio Review:

• San Antonio Review (SAR) - sareview.org (sar.pubpub.org)
• International literary, arts and ideas journal (and book publisher, on the side)
• Currently, potential contributors are directed to import/upload/copy-and-paste their work (text, images, files) into the submission portal with basic information allowing us to follow-up with them (currently, we also try to collect this information through an embedded datasheet on our submissions page, which most contributors miss while also failing to leave contact data in the submission itself so we have no way to contact them).
• Outside Tools:

1. AirTable: For submission datasheet form and data and contributor emails (not my preferred approach to security)
2. Gmail: To communicate with contributors about submissions and receive complaints about platform (thus, Gmail turns into our submission platform because people attach them there instead)
3. Open Collective: To collect a fee for the additional labor of importing submissions (again, far outside the ideal)
4. Mailchimp: Another effort to get contributor info and store emails
5. Backblaze/blot.im: Off-site data storage

• Top Pain Points:

1. Communication breakdown between contributors and editors
2. Reviews section is useless to me. Not even sure how it functions.
3. Ability to edit source of blocks - poets like to format stuff weird, and the ability to manipulate some of the HTML directly would be very helpful.

More to come, I'm sure.

[writingpoli]

We also need a way to mass delete Pubs. We have a billion abandoned, empty Pubs and it's a terror going through to kill them all and clean up our space.

[writingpoli]

It occurs to me I didn't really describe our review process in detail:

Since there's no notification of new submissions, we know there's a new submission waiting for us because:

• an email from AirTable notifies us (if someone completes the datasheet on our submissions page)
• the contributor emails us directly letting us know they've submitted or
• an editor is checking our Pending Submissions collection (to which all new subs are automatically assigned).

After receipt of submission:

1. An editor moves Pub from Pending Submissions general collection to genre-specific Pending Submissions --(Prose/Poetry/Art/etc.) collection to organize space.
2. After editor's first read, they will generally discuss piece in email/chat with other editors to decide on acceptance.
3. If accepted, editor will move Pub to Accepted/In Review collection and remove author's access to maintain a static version of the Pub (otherwise, authors tend to go in an add/make edits while we're trying to review and it breaks down) while we review. We generally make suggestions for changes (including small spelling, punctuation, grammar, etc.) in annotations along the side for the other editors and author to review. (I know there are many who would love a Word-like track changes option.) Sometimes, I'll make the edit and annotate noting what I changed.
4. If rejected, we move to Rejected/Archives collection and notify author.
5. First reviewing editor requests a second read by another member of the editorial collective prior.
6. Second reviewer leaves any annotations needed.
7. Department editor usually contacts author after re-adding them to the Pub including link to draft for them to review our suggestions.
8. Communications with authors generally occur over email but every once in a while they will engage inside the Pub in the comments/annotations, which is nice.
9. I embedded an AirTable as our editorial calendar on a dashboard-like page I created for members of the editorial collective to try to keep track of our publication schedule, but it's really awful constantly switching between platforms. Currently, if a core member of the editorial collective dies between publication dates (Weds and Sundays), the remaining members have to hope they used our shared SAReditors email for or CCed it their discussions with pending contributors and search to find agreed-upon publication dates.
10. As there's no way we're aware of scheduling a Pub to automatically publish at a certain date and time, one of our Europe-based editorial collective members publishes the latest piece early in the morning.
11. Editor creates shareable image in Canva or elsewhere with quote from Pub for marketing.

Also, somewhere in there one of the editors will find an image (or transfer a submitted artwork) to serve as the header image of the Pub. Sometimes the image loosely relates to the content of the Pub, other times not.

Here are images of the dashboard pages I created a while back (note they haven't been updated recently):

[tefkah]

First of all, it's great to see that this is being worked on and that you are so open to feedback! I am from the Journal of Trial and Error (https://archive.jtrialerror.com), a journal which at the moment publishes approximately 10-15 articles a year.

Currently, our submission/review process is a bit of a mess which involves a lot of emailing back and forth. In general we "just" do closed double-blind peer review, i.e. the authors send us a completed manuscript in .docx or .tex format, which gets reviewed by reviewers, comments get sent back to the authors, they revise and then possibly get reviewed again. All this happens very... manually though.

Process

At the moment we are using OJS as a submission portal at https://submit.jtrialerror.com, although some papers are just being handled manually through email exchange. Rough outline of how his works in OJS:

1. Author creates an account
2. Author gets presented with a list of submission requirements they have to check off
3. Author uploads the relevant files
4. Editors get a notification that an article has been submitted
5. Journal manager/editor assigns article to specific editor, granting them control over the submission/review process
6. Editor can invite reviewers by typing in their name and email, sending them an automatically generated email with login information and information about the article.
7. Reviewer can accept or decline
8. If reviewer accepts, editors get a notification and article is moved to "under review"
9. If reviewer does not respond, there is a possibility to send a follow up reminder email
10. Reviews are collected, editor can send notification of acceptance or otherwise to author, along with reviewer comments
11. Repeat steps 6-10 for the number of rounds of review
12. Paper gets accepted (hopefully), and moves on to copyediting/typesetting.

Pain

I will reach out to the rest of my team for more input on these, but as far as I can tell the most painful things are related to what happens after step 12:

1. The current review/submission process is entirely disconnected from PubPub, so we cannot take advantage of the annotation system or otherwise incorporate a more open peer review style, which we would like to have. At the moment we "solve" this by publishing the reviews of papers separately, but that's no real solution, see this as an example.
2. The disconnect also causes us to do a lot of work twice, such as setting all the citations both in TeX and in PubPub. If this would have already been done in PubPub by the authors it would save us a lot of time.

What would make things more exciting/less painful:

1. Although the process looks quite streamlined, it involves SO MUCH emailing back and forth, which makes things take a long time. Having something more akin to this, a discussion platform, built into the review system streamline the whole thing immensely. OJS sort of has that, but it's faaar from ideal.
2. A way to implement different forms of open peer review in house would be amazing. I'm mainly thinking of either fully open peer review where a preprint is published and comments are being made afterwards, or having this happen beforehand and showing the whole peer review process afterwards in the form of annotations. The difficulty we found with implementing this is deciding which version of the document to show: if you show the "post peer-review" finalized article, the comments don't make sense, but showing the pre-review version with comments obviously does not showcase the final product. A way to easily toggle between "final version" and "version with peer review comments" would be the ideal solution.

In any case, thank you for opening up this discussion, I'm looking forward to seeing PubPub get even better!

[oliviodare]

Hi there! another member of the journal. I have a simpler and perhaps more abstract thought: it would be great if the way our community on pubpub is structured would allow for different levels of engagement. Not all of us engage to the same level, and right now it seems as thought you are either in (editors) or out (readers), with little in between. It'll be ideal if authors and reviewers could be somehow involved but not with full privileges, and that this would work in an individual sing-in basis. The current system is based on community administrators using the 'Members' page to invite and manage collaborators, but would be great if people can simply join without needing our invitation. This would be very interesting in order to find reviewers for our papers, in which people in the community add their expertise and their willingness to review. And in case of open post-publication review, articles who haven't been reviewed could perhaps show a sign that they need the community's attention and even send a notification to relevant reviewers/editors.

[jeffpooley]

We have two linked communities: One, mediastudies.press for monographs, and a second, History of Media Studies, a journal to launch in September.

I'm really enjoying this thread, and excited that you're reaching out to the community. I will post with more specifics soon, after my co-editors and I meet later this week, but I wanted to offer a more general comment first: I endorse the basic philosophy that you outlined back in June, that took as its premise that PubPub doesn't want to replicate existing review workflows (as stitched into, for example, ScholarOne or OJS).

Flexibility to accommodate review workflows that don't hinge on double-blind peer review seems important here. (I'm thinking of post-publication review, signed reviews, and community reviewing, in line with @ThomasJorna and @oliviodare ’s comments.)

More broadly, I was excited to see, back in a June meeting, PubPub’s willingness to rethink review—to not be bound by inherited conventions. This strikes me as in keeping with PubPub's history and ethos, going back to Travis's dissertation. One idea floated in June, that has great potential, is affordance for a bundle of review types, including incorporating outside materials, that could be mixed and matched according to community preferences.

This would mean sacrificing some of the path-rigid intricacy of existing systems (which get convoluted, in my opinion), as a conscious trade-off in favor of flexibility and support for fresh approaches.

[mlafl]

Hi all, I'm chiming in a bit late but would echo the appreciation that others have voiced about being consulted as PubPub focuses on building out submission and review. I'm affiliated with Exertions (https://saw.americananthro.org/exertions), the short-form web publication of the Society for the Anthropology of Work. We're using PubPub in a slightly unconventional way, as the homepage for the society itself and with Exertions set up as a tag-based collection. This may or may not have been the best way to go.

The majority of the content we publish at Exertions goes through editorial review only and, in this sense, the publication is more like a blog: this includes book reviews, standalone posts, and guest edited series like this recent one on Policing and Labor. But we also give authors the option of open peer review: so far, we've published one piece that has gone through this process and we have another that is currently under revision.

A few reflections on how this has gone (and here I'll echo some of the points made above): annotation-based review has its value, but it does channel reviewers toward thinking at the sentence level rather than commenting in a more synthetic way about a submission's structure, use of evidence, etc. So I would love to see a mixed system that keeps the option of annotation but also allows for extended-format reviews that get their own publication metadata and DOI: this is super important for crediting reviews as scholarly work in their own right (while maintaining a legible link to the piece that anchors them). +1 to @devinberg's points about having the option to display annotations more prominently (having to archive them in the first place just doesn't fit with my mental model) and to clearly show how later versions evolved in response.

I'll knock off there for now, but would be happy to talk further and/or offer Exertions as a site for experimentation with new features 👾

[gabestein]

Just chiming in to say thank you all so much for your feedback. It's incredibly helpful. We're planning to do a few rounds of loose design sprints, and we'll be back next week with some specific questions and follow-ups on this thread, as a start.

[gabestein]

Hi folks! Again, thank you so much for your feedback and ideas. We're ready to start walking through some of our initial designs to hear finer-grained feedback from you all.

If you're interested in seeing what we're up to and providing feedback, please sign up for one of our slots next week, via this link: https://calendly.com/gabekfg/review-feedback?month=2021-06

There will be more in a few weeks (we're working around staff vacations), so not to worry if none of these work for you. We'll also be reaching out to you directly if we have specific questions for you.

Again, thank you so much for your help making PubPub better!

[vanessarosa710]

Hello! Just checking in to see if there’s any progress on this. I have no idea how to use the current reviews tab on PubPub. It seems like the authors can request a review and then people can be added as reviewers to the contributors list and add comments and files. But this is different than using the connections tab to write a review.

There’s also no way I could find to communicate between editors and reviewers or track changes, which makes this whole process harder. If there are any guides for these things or if anyone knows what to do to use the tools in PubPub, please let me know.

[writingpoli]

My wife, Misty, came up with a possible solution to our submitters' oft-expressed confusion:

Instead of squishing all the upload/import/metadata workflow on a single page with the WYSIWYG editor, what if:

1. On the initial submission page, submitters upload their document and complete the metadata;
2. On the next page, they are presented with a preview of their submission as it would appear on PubPub along with metadata confirmation leading to two options:
3. SUBMIT -- Everything looks okay, send it to the journal for consideration OR
4. EDIT -- This brings up the PubPub WYSIWYG for the first time and allows the submitter to make any necessary formatting or other changes and a SAVE or SUBMIT option.

I think that mostly makes sense -- it's early.

Essentially, her hypothesis is that the editor ultimately confuses people at the initial stage of submission because they aren't in the creative process during the period they're submitting work. They aren't looking to create/edit their work (they've been doing that elsewhere for days or weeks or months or years). They're looking to submit it -- sometimes to multiple places as quickly as possible. But, as I explained, because no import/conversion process works perfectly anywhere, we have to present submitters with a preview to allow them to fix any errors introduced (see my soon-to-be added thread "Poetry on PubPub") -- otherwise, we may as well just let them email it to us and do all the work ourselves.

Misty is going to sketch up some mockups, I think. I'll share them when she does.