Skip to content

Commit 7c49b26

Browse files
ig11987ig11987
committed
Add continous transfer mode
Allows the transfer to happen continuously. Files can change on the source and will be picked up and transferred as long as fastsync is running. This is also means fastsync now supports incremental transfers i.e. stopping the operation and resuming it later. Closes: #16
1 parent 52e49e1 commit 7c49b26

File tree

5 files changed

+756
-93
lines changed

5 files changed

+756
-93
lines changed

‎README.md‎

Lines changed: 38 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -15,9 +15,6 @@ Fastsync targets the following use case:
1515
* **Compression is handled externally.** Fastsync does not compress the stream.
1616
If the data volume benefits from compression, then compress the files ahead
1717
of time with e.g. `lz4`, `brotli`, or `zstd`.
18-
* **A full transfer is necessary.** Fastsync always sends all files. If some
19-
files are already present at the receiving side, or similar data is already
20-
present, `rsync` might be a better fit.
2118

2219
## Building
2320

@@ -49,6 +46,44 @@ On the receiving end, suppose we download with 32 TCP connections:
4946
cd /some/path
5047
fastsync recv 100.71.154.83:4440 32
5148

49+
File modification timestamps are preserved during all transfers.
50+
51+
## Continuous mode
52+
53+
Fastsync supports continuous mode with the `--continuous` flag. When enabled, fastsync will:
54+
55+
1. Compare files by name, size, and modification timestamp
56+
2. Skip files that already exist at the destination with matching size and timestamp
57+
3. Transfer only files that are missing or have different size/timestamp
58+
4. Keep syncing in rounds with a 2-second delay between rounds
59+
5. After each transfer round, wait and then re-scan the source files
60+
6. If changes are detected in the new scan, start another sync round
61+
7. Stop when 5 consecutive rounds finish with no changes detected
62+
63+
Both sender and receiver must use the `--continuous` flag:
64+
65+
# Receiver
66+
fastsync recv <ip>:<port> 32 --continuous
67+
68+
# Sender
69+
fastsync send --continuous <ip>:<port> ./data
70+
71+
**Note:** Continuous mode only handles file additions and modifications. File deletions from the sender are not propagated to the receiver - files that exist at the destination will remain even if they're removed from the source.
72+
73+
## Testing
74+
75+
To run all tests:
76+
77+
cargo test
78+
79+
To run only unit tests:
80+
81+
cargo test --lib
82+
83+
To run integration tests:
84+
85+
./integration_tests/test_continuous.sh
86+
5287
## Known issues
5388

5489
* It's too spammy.

‎integration_tests/test_continuous.sh‎

Lines changed: 304 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,304 @@
1+
#!/bin/bash
2+
3+
# This test will start sender and receiver
4+
# processes on the local machine.
5+
# It will then create/delete files in the `/tmp/`
6+
# directory of the machine.
7+
#
8+
# Note: this test will also kill any
9+
# fastsync process running on ports 8899, 8901
10+
# as part of cleanup.
11+
# If you are running real fastsync operations,
12+
# it's advisable to not run this test in parallel.
13+
14+
15+
set -e
16+
17+
RED='\033[0;31m'
18+
GREEN='\033[0;32m'
19+
YELLOW='\033[1;33m'
20+
BLUE='\033[0;34m'
21+
NC='\033[0m' # No Color
22+
23+
# Get the directory where this script is located
24+
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
25+
PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
26+
FASTSYNC_BIN="$PROJECT_ROOT/target/release/fastsync"
27+
28+
kill_fastsync_processes() {
29+
for port in 8899 8901; do
30+
lsof -ti:$port 2>/dev/null | while read pid; do
31+
# Get full command line to ensure it's really fastsync
32+
if ps -p $pid -o args= 2>/dev/null | grep -q fastsync; then
33+
echo "Killing fastsync process $pid on port $port"
34+
kill -TERM $pid 2>/dev/null || true
35+
fi
36+
done
37+
done
38+
}
39+
40+
# Build the project if binary doesn't exist
41+
if [ ! -f "$FASTSYNC_BIN" ]; then
42+
echo "Building fastsync..."
43+
cd "$PROJECT_ROOT"
44+
cargo build --release
45+
fi
46+
47+
echo -e "${YELLOW}=== Testing Continuous Mode ===${NC}"
48+
49+
echo -e "${GREEN}Killing any fastsync processes already running${NC}"
50+
kill_fastsync_processes
51+
52+
53+
# Test 1: Basic continuous mode functionality
54+
echo -e "\n${YELLOW}=== Test 1: Basic Continuous Mode Functionality ===${NC}"
55+
56+
# Clean up any previous test directories
57+
rm -rf /tmp/fastsync_test_src /tmp/fastsync_test_dst
58+
mkdir -p /tmp/fastsync_test_src /tmp/fastsync_test_dst
59+
60+
# Create initial test files
61+
echo "Creating initial test files..."
62+
echo "File 1 content" > /tmp/fastsync_test_src/file1.txt
63+
echo "File 2 content" > /tmp/fastsync_test_src/file2.txt
64+
echo "File 3 content" > /tmp/fastsync_test_src/file3.txt
65+
66+
# Start receiver in continuous mode first
67+
echo -e "\n${YELLOW}Starting receiver in continuous mode...${NC}"
68+
cd /tmp/fastsync_test_dst
69+
echo "y" | "$FASTSYNC_BIN" recv 127.0.0.1:8899 2 --continuous 2>&1 | tee /tmp/fastsync_receiver.log &
70+
RECEIVER_PID=$!
71+
echo "Receiver PID: $RECEIVER_PID"
72+
sleep 2
73+
74+
# Start sender in continuous mode (using . to monitor current directory)
75+
echo -e "\n${YELLOW}Starting sender in continuous mode...${NC}"
76+
cd /tmp/fastsync_test_src
77+
"$FASTSYNC_BIN" send 127.0.0.1:8899 --continuous . 2>&1 | tee /tmp/fastsync_sender.log &
78+
SENDER_PID=$!
79+
echo "Sender PID: $SENDER_PID"
80+
81+
# Wait for initial sync
82+
echo -e "\n${YELLOW}Waiting for initial sync...${NC}"
83+
sleep 5
84+
85+
# Check initial sync with retry logic
86+
echo -e "\n${YELLOW}Checking initial sync...${NC}"
87+
SYNC_SUCCESS=false
88+
for i in {1..10}; do
89+
if [ -f /tmp/fastsync_test_dst/file1.txt ] && \
90+
[ -f /tmp/fastsync_test_dst/file2.txt ] && \
91+
[ -f /tmp/fastsync_test_dst/file3.txt ]; then
92+
echo -e "${GREEN}✓ Initial sync successful${NC}"
93+
SYNC_SUCCESS=true
94+
break
95+
fi
96+
echo -n "."
97+
sleep 1
98+
done
99+
100+
if [ "$SYNC_SUCCESS" = false ]; then
101+
echo -e "\n${RED}✗ Initial sync failed after 10 seconds${NC}"
102+
kill_fastsync_processes
103+
exit 1
104+
fi
105+
106+
# Test 1a: Modify a file
107+
echo -e "\n${YELLOW}Test 1a: Modifying file2.txt...${NC}"
108+
echo "File 2 modified content" > /tmp/fastsync_test_src/file2.txt
109+
touch /tmp/fastsync_test_src/file2.txt
110+
sleep 3
111+
112+
if [ "$(cat /tmp/fastsync_test_dst/file2.txt)" = "File 2 modified content" ]; then
113+
echo -e "${GREEN}✓ File modification synced${NC}"
114+
else
115+
echo -e "${RED}✗ File modification not synced${NC}"
116+
echo "Expected: 'File 2 modified content'"
117+
echo "Got: '$(cat /tmp/fastsync_test_dst/file2.txt)'"
118+
fi
119+
120+
# Test 1b: Add a new file
121+
echo -e "\n${YELLOW}Test 1b: Adding file4.txt...${NC}"
122+
echo "File 4 new content" > /tmp/fastsync_test_src/file4.txt
123+
sleep 3
124+
125+
if [ -f /tmp/fastsync_test_dst/file4.txt ] && \
126+
[ "$(cat /tmp/fastsync_test_dst/file4.txt)" = "File 4 new content" ]; then
127+
echo -e "${GREEN}✓ New file synced${NC}"
128+
else
129+
echo -e "${RED}✗ New file not synced${NC}"
130+
fi
131+
132+
# Test 1c: Delete a file (should not be deleted on receiver)
133+
echo -e "\n${YELLOW}Test 1c: Deleting file1.txt from sender...${NC}"
134+
rm /tmp/fastsync_test_src/file1.txt
135+
sleep 3
136+
137+
if [ -f /tmp/fastsync_test_dst/file1.txt ]; then
138+
echo -e "${GREEN}✓ File correctly retained on receiver${NC}"
139+
else
140+
echo -e "${RED}✗ File incorrectly deleted on receiver${NC}"
141+
fi
142+
143+
# Test 1d: Modify multiple files
144+
echo -e "\n${YELLOW}Test 1d: Modifying multiple files...${NC}"
145+
echo "File 3 modified content" > /tmp/fastsync_test_src/file3.txt
146+
echo "File 4 modified content" > /tmp/fastsync_test_src/file4.txt
147+
sleep 3
148+
149+
if [ "$(cat /tmp/fastsync_test_dst/file3.txt)" = "File 3 modified content" ] && \
150+
[ "$(cat /tmp/fastsync_test_dst/file4.txt)" = "File 4 modified content" ]; then
151+
echo -e "${GREEN}✓ Multiple file modifications synced${NC}"
152+
else
153+
echo -e "${RED}✗ Multiple file modifications not synced${NC}"
154+
fi
155+
156+
# Kill processes
157+
echo -e "\n${YELLOW}Stopping sender and receiver...${NC}"
158+
kill_fastsync_processes
159+
160+
echo -e "${GREEN}✓ Test 1: Basic continuous mode functionality passed${NC}"
161+
162+
163+
# Test 2: Auto-exit after 5 rounds with no changes
164+
echo -e "\n${YELLOW}=== Test 2: Auto-Exit After 5 Rounds ===${NC}"
165+
166+
# Clean up any previous test directories
167+
rm -rf /tmp/fastsync_test_exit_src /tmp/fastsync_test_exit_dst
168+
mkdir -p /tmp/fastsync_test_exit_src /tmp/fastsync_test_exit_dst
169+
170+
# Create initial test files
171+
echo "Creating initial test files..."
172+
echo "Initial content 1" > /tmp/fastsync_test_exit_src/file1.txt
173+
echo "Initial content 2" > /tmp/fastsync_test_exit_src/file2.txt
174+
175+
# Start receiver in continuous mode first
176+
echo -e "\n${YELLOW}Starting receiver in continuous mode...${NC}"
177+
cd /tmp/fastsync_test_exit_dst
178+
echo "y" | "$FASTSYNC_BIN" recv 127.0.0.1:8901 2 --continuous 2>&1 | tee /tmp/fastsync_receiver_exit.log &
179+
RECEIVER_PID=$!
180+
echo "Receiver PID: $RECEIVER_PID"
181+
sleep 2
182+
183+
# Start sender in continuous mode
184+
echo -e "\n${YELLOW}Starting sender in continuous mode...${NC}"
185+
cd /tmp/fastsync_test_exit_src
186+
"$FASTSYNC_BIN" send 127.0.0.1:8901 --continuous file1.txt file2.txt 2>&1 | tee /tmp/fastsync_sender_exit.log &
187+
SENDER_PID=$!
188+
echo "Sender PID: $SENDER_PID"
189+
190+
# Wait for initial sync
191+
echo -e "\n${YELLOW}Waiting for initial sync...${NC}"
192+
sleep 5
193+
194+
# Check initial sync with retry logic
195+
echo -e "\n${YELLOW}Checking initial sync...${NC}"
196+
SYNC_SUCCESS=false
197+
for i in {1..10}; do
198+
if [ -f /tmp/fastsync_test_exit_dst/file1.txt ] && \
199+
[ -f /tmp/fastsync_test_exit_dst/file2.txt ]; then
200+
echo -e "${GREEN}✓ Initial sync successful${NC}"
201+
SYNC_SUCCESS=true
202+
break
203+
fi
204+
echo -n "."
205+
sleep 1
206+
done
207+
208+
if [ "$SYNC_SUCCESS" = false ]; then
209+
echo -e "\n${RED}✗ Initial sync failed after 10 seconds${NC}"
210+
kill_fastsync_processes
211+
exit 1
212+
fi
213+
214+
# Test 2a: Modify a file to reset the skip counter
215+
echo -e "\n${YELLOW}Test 2a: Modifying file1.txt to reset skip counter...${NC}"
216+
sleep 3 # Let one round pass with all files up to date
217+
echo "Modified content 1" > /tmp/fastsync_test_exit_src/file1.txt
218+
sleep 3
219+
220+
if [ "$(cat /tmp/fastsync_test_exit_dst/file1.txt)" = "Modified content 1" ]; then
221+
echo -e "${GREEN}✓ File modification synced${NC}"
222+
else
223+
echo -e "${RED}✗ File modification not synced${NC}"
224+
kill_fastsync_processes
225+
exit 1
226+
fi
227+
228+
# Test 2b: Now let it run without changes to test the 5-round exit
229+
echo -e "\n${YELLOW}Test 2b: Testing auto-exit after 5 rounds with no changes...${NC}"
230+
echo -e "${BLUE}Round timings (2 seconds per round):${NC}"
231+
echo -e "${BLUE} Round 1: 0-2 seconds${NC}"
232+
echo -e "${BLUE} Round 2: 2-4 seconds${NC}"
233+
echo -e "${BLUE} Round 3: 4-6 seconds${NC}"
234+
echo -e "${BLUE} Round 4: 6-8 seconds${NC}"
235+
echo -e "${BLUE} Round 5: 8-10 seconds${NC}"
236+
echo -e "${BLUE} Expected exit: ~10-11 seconds${NC}"
237+
238+
START_TIME=$(date +%s)
239+
240+
# Monitor for up to 15 seconds (should exit around 10-11 seconds)
241+
for i in {1..15}; do
242+
if ! kill -0 $SENDER_PID 2>/dev/null; then
243+
END_TIME=$(date +%s)
244+
ELAPSED=$((END_TIME - START_TIME))
245+
echo -e "\n${GREEN}✓ Sender exited after $ELAPSED seconds${NC}"
246+
247+
# Check if it exited in the expected time window (9-13 seconds)
248+
if [ $ELAPSED -ge 9 ] && [ $ELAPSED -le 13 ]; then
249+
echo -e "${GREEN}✓ Exit timing correct (expected ~10-11 seconds)${NC}"
250+
else
251+
echo -e "${RED}✗ Exit timing unexpected (got $ELAPSED seconds)${NC}"
252+
fi
253+
254+
# Check the log for the expected message
255+
if grep -q "Continuous sync completed - no changes detected for 5 consecutive iterations" /tmp/fastsync_sender_exit.log; then
256+
echo -e "${GREEN}✓ Found expected exit message in logs${NC}"
257+
else
258+
echo -e "${RED}✗ Expected exit message not found in logs${NC}"
259+
fi
260+
261+
# The continuous mode exit logic works internally without specific skip count messages
262+
263+
break
264+
fi
265+
266+
echo -n "."
267+
sleep 1
268+
done
269+
270+
# Check if sender is still running (it shouldn't be)
271+
if kill -0 $SENDER_PID 2>/dev/null; then
272+
echo -e "\n${RED}✗ Sender still running after 15 seconds (should have exited)${NC}"
273+
kill $SENDER_PID 2>/dev/null || true
274+
FAILED=1
275+
else
276+
echo -e "${GREEN}✓ Sender correctly exited after no changes${NC}"
277+
fi
278+
279+
kill_fastsync_processes
280+
281+
echo -e "${GREEN}✓ Test 2: Auto-exit functionality passed${NC}"
282+
283+
284+
echo -e "\n${YELLOW}=== Test Summary ===${NC}"
285+
echo -e "${GREEN}✓ Basic continuous mode functionality${NC}"
286+
echo -e "${GREEN}✓ File modification sync${NC}"
287+
echo -e "${GREEN}✓ New file detection${NC}"
288+
echo -e "${GREEN}✓ File retention on deletion${NC}"
289+
echo -e "${GREEN}✓ Multiple file modification sync${NC}"
290+
echo -e "${GREEN}✓ Auto-exit after 5 rounds with no changes${NC}"
291+
292+
echo -e "\n${YELLOW}All continuous mode tests completed successfully!${NC}"
293+
echo "Logs available at:"
294+
echo " /tmp/fastsync_sender.log"
295+
echo " /tmp/fastsync_receiver.log"
296+
echo " /tmp/fastsync_sender_exit.log"
297+
echo " /tmp/fastsync_receiver_exit.log"
298+
299+
# Exit with success if no failures
300+
if [ -z "$FAILED" ]; then
301+
exit 0
302+
else
303+
exit 1
304+
fi

‎rust-toolchain.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
[toolchain]
2-
channel = "1.74.0"
2+
channel = "1.75.0"
33
components = ["rustc", "cargo", "rustfmt", "clippy"]
44
targets = [
55
# Default regular build.

0 commit comments

Comments
 (0)