Consider how to import from Language Forge non-SR projects

[rmunn]

There are some Language Forge projects that do not use the Send/Receive system, and thus don't have a project code on Lexbox. We don't know how many there are yet.

First step is figuring out how many there are. Then, use the LfClassic API to import then into a Lexbox CRDT project with a FieldWorks project as well (a Lexbox hg repo for Send/Receive). At that point we can delete the "old" LF project and replace it with an LF project that uses S/R.

I'll update this issue with the actual numbers once I have them.

[rmunn]

Initial findings: 1252 total projects in Language Forge, 536 of which are on the Send/Receive system and 716 of which are not. But of those 716 projects, 571 of them had a lexicon table whose BSON representation, after gzipping, was less than 4 KiB in size (strongly suggesting either an empty project or else a test project where only a few entries were ever created). Another 70 non-SR projects had a lexicon whose gzipped representation was between 4 KiB and 64 KiB (small projects, might also be test data), and 75 non-SR projects had a lexicon more than 64 KiB in size after being gzipped.

So there are at most 150 non-Send/Receive projects that we might need to convert. Many of them might also be abandoned: my initial research did NOT include looking at each project's last-modified date.

[rmunn]

There are several non-SR projects with real data, so this is definitely worth doing. I won't copy project codes in here just in case any of them are sensitive, but the largest non-SR project has about 24,000 entries; second largest, about 22,000; and third largest, about 20,000.

[rmunn]

Once #2439 is complete and working properly, it might be useful for this. The import process would:

• Create a new, empty FLEx project in Lexbox with the appropriate vernacular / analysis writing systems
• Clone that empty FLEx project into Language Forge with a temporary project code
• Copy the Mongo database(s) we need from the real LF project into the temporary one
  • Will definitely want lexicon and comments
  • Don't forget LinkedFiles and so on, to get pictures and audio files
• Do a Send/Receive on that temporary LF project
• Verify that what ends up in LexBox has the right number of lexicon entries and so on
• (During testing): Delete the temporary LF project so it doesn't clutter things up

That would allow us to back up all the non-SR LF projects, and eventually offer them in FW Lite, without having to spend a lot of time working on the LfClassicMiniLcmApi adapter.

[rmunn]

Testing with real data has revealed a missing LfMerge feature we'll need to implement: at least one large project (17.5k entries) has several custom fields that LfMerge was skipping, with the message "Custom field customField_entry_General_word from LF skipped, because we're not yet creating new custom fields in LCM". (There were several custom fields, not just "General word", and they were heavily used in that dictionary project). That was fine when the target was Send/Receive projects, where we could tell users "Just create the custom field in FieldWorks, then do a Send/Receive". But now that the migration process is creating the FW project itself, we will need an automated way of creating custom fields. This should probably be done after the empty project is created, i.e. not as part of #2439. I'll probably create a new issue in LfMerge to track this need.

[rmunn]

There are actually very few (possible as few as 2) non-SR projects with real data (as opposed to testing projects) that had custom fields. Perhaps instead of a complicated LfMerge feature, we could just create the .fwdata file for those two projects by hand: create the empty project in FLEx, add all the needed custom fields, and then do the Send/Receive migration. The manual work involved in doing that would still be smaller than adding an LfMerge feature we'd never use again.