Find a better syncing solution for script files

[JohnSundell]

When the user starts editing a script (using marathon edit Myscript.swift), we start syncing the file between the internal copy that Marathon maintains (in order to be able to generate packages for the script) and the original script file (located wherever the user wants).

Currently, this sycning mechanism is done through a simple timer, that every 3 seconds copies the original script file back into Marathon's script folder. This is not ideal for a few reasons:

1. It requires the user to keep Marathon open in order for this timer to be able to run.
2. If any edits are made in the Xcode project after Marathon was closed, those edits won't be synced and will be easily lost.
3. Timers are ugly 😬

Before I implemented the syncing this way, I tried two other solutions:

Symlinking the script file

Marathon makes heavy use of symlinks (for example for packages, to "trick" SPM that packages are already compiled, so it won't re-compile them for every script). So my initial idea was to use symlinks for scripts as well. The problem is that Xcode doesn't like symlinks, and would not be able to save the file back (basically it tried to save into the symlink itself, which would cause an error dialog in Xcode). So I abandoned that idea.

Hooking into file system events

The next thing I tried was using GCD to hook into the file system to get notified whenever the file was changed. While this worked really well if the script was edited using an editor like Atom, it didn't work for Xcode, which doesn't seem to generate the same events every time the file is saved.

What we want

So, given the above context, what we want is a more robust syncing solution that fullfils the following requirements:

1. Files are always kept in sync, whenever the original script file is updated - Marathon is also updated.
2. Edits are made directly into the original file, meaning that if the Xcode project is later opened (using "Recently opened") and edits are made, they should be auto-synced.
3. The main Marathon CLI should not have to run in order to sync files.

Perhaps we could hook into the Xcode project itself somehow? Looking for ideas from the community here 😄

[JohnSundell]

@kareman @darthpelo @clayellis @pixyzehn @cojoj @alexaubry @garricn Any ideas on how this can be solved in a better way? 🙂

[clayellis]

Just firing off ideas in the hopes that one of them sticks —
In the same way that ScriptManager.makeScriptPathList() "takes the opportunity to clean up cache data", what if marathon run | edit | create --tests "took the opportunity" to copy the original script file back to Marathon? The way I see it, Marathon only needs to be in sync when the user:

1. runs the script
2. Opens the script through a command for editing (either edit [--tests] or create --tests)
   In any other case, it doesn't matter too much to the user if Marathon is in sync with the original file or not. Right? The implementation would be pretty simple, too.

Or maybe that's basically what you described about Symlinking?

[alexaubry]

One solution could be to create a small daemon that uses the Kernel Queues API to listen to file changes in the background.

🌎 From the Documentation:

The kernel queues API provides a way for an application to receive notifications whenever a given file or directory is modified in any way, including changes to the file’s contents, attributes, name, or length.

Whenever the daemon would receive a file change notification from the kernel, it would copy the modified file back to its intended location, and keep both files in sync almost immediately; all this with:

• little memory footprint
• minimal disk read/writes
• no need for the the main Marathon CLI to stay running

Furthermore, the implementation would be fairly simple 😄

[JohnSundell]

@clayellis The problem is that when the user edits the script in Xcode, the original script file is not the file being edited. Meaning that if some edits are made after Marathon stops syncing, they won't be copied back to the original. So the copying needs to go MarathonCopy -> Original, not the other way around. And we can't really copy Marathon's version into the original file, because the user might've made edits without Marathon (we basically need to treat the original as the master copy).

@alexaubry Very interesting idea. That could totally be a way forward. Need to investigate it, but looks very interesting 👍

Keep the ideas coming guys, this is going to require some brainstorming for sure 😄

[clayellis]

D'oh! I was bored in a meeting when I drummed that up. I had the files backwards. I'll keep thinking. I like @alexaubry's idea — time for some reading.

[cojoj]

AppleScript? 🤔 (Blind guess, without any proven background)

[cojoj]

But we should look for something not tied to Apple's ecosystem.

[clayellis]

This isn't a permanent fix, nor is it very reliable one, but it's something to add to the list of ideas.

open -W script

From man open:

-W: Causes open to wait until the applications it opens (or that were already open) have exited.

This works as a replacement of the current system except when Xcode was already open before we open a script for editing. In which case open -W will wait until those windows are closed as well. So that means it won't be very reliable. This does not fulfill all of our "wants" above. For these reasons, we probably shouldn't implement it. Just posting this to document ideas.

[kareman]

The problem is that when the user edits the script in Xcode, the original script file is not the file being edited.

Can we make Xcode edit the original script file and place a symlink in marathons copy to it? So marathon edit creates an Xcode project like it does now, then alters it to use the original script. I have no idea how to do this programmatically though.

[JohnSundell]

@cojoj The fix can totally be tied to Apple's ecosystem, as this is for Xcode editing only. I don't know how AppleScript could solve this problem though 😄

@clayellis Like you said, open -W script also has significant downsides, so don't think it's a solution. But awesome that you're keeping the ideas coming, and like you also said, good to document every solution considered 🚀

@kareman The problem is that the file need to be named main.swift in order to work with Xcode as a command line tool, and we don't want to force all scripts to be named main.swift 😢

Great ideas so far guys, keep 'em coming! 👍

[ghost]

The problem is that the file need to be named main.swift in order to work with Xcode as a command line tool, and we don't want to force all scripts to be named main.swift 😢

I think users of Marathon would understand why script files are named main.swift. At the moment, the parent folder is the name of the script, so you could just keep this convention going forward as a means of storing / identifying the script.

• The create command could be modified to create a main.swift file, nested in a named folder.
• The install command could be modified to create a main.swift file, nested in a named folder.
  And so on

You could regard each of these nested files as a script bundle. In the future, these script bundles could store other resources for the script to consume? Like a playground?