Trouble with large collections

[vgranger2]

Hi!

We have had some problems importing and processing our larger collections. During the appraisal module, the collection will become corrupted, and I have to start all over with importing that collection only for it to become corrupted again down the line. The last one I tried unsuccessfully was 78.7 GB and I have an upcoming collection that is about 4x larger than that, I'm just wondering if anybody else has this problem and if I should be doing something differently.

Thank you!

[jfarwer]

Hi,

Thanks for reaching out. In what way does the collection get corrupted? Are you getting error messages?

Thanks

[vgranger2]

I don't recall there being any error messages. Usually how it goes is that I'll save my work at the end of the day, come back the next morning and I'll find that ePadd has closed. When I open it back up the collection just isn't there, instead it's back on the About this Archive page ready for import. It seems like this has happened the last few times we've worked on email collections that have had around 300,000+ emails.

[jfarwer]

Thanks. Could you please send the files epadd.log and epadd.log.warnings from the epadd-settings folder?

[vgranger2]

So sorry for the delay! Here are the files:

epadd.log.zip

epadd.zip

Unrelated to this issue- I've also been experiencing some problems trying to save my work when processing a collection. For days after clicking save, the window saying "'Working...Saving Archive..." will just go and go and go. It usually doesn't take that long, wondering if you may have some insight to that as well. Thanks so much!

[jfarwer]

Thanks, that's helpful.

Would it be possible for you to share the archive for testing, or are there privacy restrictions? If needed, you can send it to me via email instead of uploading it here on GitHub.

Does the other issue occur with a different collection, is it with the same one or is it a general problem?

[vgranger2]

Unfortunately, there are some privacy restrictions in the archive so I won't be able to send it to you. Is there anything else I can do that would be helpful to you at all?

The saving issue seems to be a general problem.

[jfarwer]

Could you please try using this version: epadd-custom? It includes a few recent fixes and improvements.