Stdout often gets truncated (reporducable in both golem-js and yapapi) #3450
Description
The stdout output is often truncated. From my testing, this issue can occur on all kinds of providers, but it happens most frequently on those with a low thread count. We managed to consistently reproduce this with both yapapi and golem-js (tested with both polling batch and streaming batch).
A temporary workaround is to append && sleep 5 to the end of the executed script, but that's obviously not ideal.
This is what we get from yagna:
[{"index":0,"eventDate":"2025-08-14T09:07:49.856122959Z","result":"Ok","stdout":"<INCOMPLETE_STDOUT>","stderr":null,"message":null,"isBatchFinished":true}]Here, the stdout field contains only about 50-80% of the expected output. So the batch is marked as finished, but not everything is sent over.
Here's the minimal reproduction in golem-js:
import { GolemNetwork } from "@golem-sdk/golem-js";
import { pinoPrettyLogger } from "@golem-sdk/pino-logger";
const order = {
demand: {
workload: {
imageTag: "golem/alpine:latest",
},
},
market: {
rentHours: 15 / 60,
pricing: {
model: "linear",
maxStartPrice: 0,
maxCpuPerHourPrice: 0.001,
maxEnvPerHourPrice: 0.001,
},
offerProposalFilter: (offer) =>
offer.properties["golem.inf.cpu.threads"] < 3,
},
};
(async () => {
const glm = new GolemNetwork({
payment: {
network: "polygon",
},
logger: pinoPrettyLogger({
level: "info",
}),
});
const linesToPrint = 1000;
try {
await glm.connect();
const pool = await glm.manyOf({ order, poolSize: 10 });
await Promise.allSettled(
Array(10)
.fill(null)
.map(() =>
pool.withRental(async (rental) => {
await rental
.getExeUnit()
.then((exe) =>
exe.run(
`seq ${linesToPrint} | awk '{printf "%-120s\\n", $0}' | sed 's/ /-/g'`,
),
)
.then((res) =>
console.log(
"Received",
String(res.stdout).split("\n").filter(Boolean).length,
"lines of stdout, but expected",
linesToPrint,
),
);
}),
),
);
} catch (err) {
console.error("Failed to run the example", err);
} finally {
await glm.disconnect();
}
})().catch(console.error);Run with YAGNA_APPKEY=appkey node script.js. This script runs seq 1000 | awk '{printf "%-120s\n", $0}' | sed 's/ /-/g', which prints 1000 lines of output, each 120 characters wide. When you run the script, you will see that we often don't receive the entire stdout:
Received 727 lines of stdout, but expected 1000
Received 1000 lines of stdout, but expected 1000
Received 1000 lines of stdout, but expected 1000
Received 881 lines of stdout, but expected 1000
Received 1000 lines of stdout, but expected 1000
Received 1000 lines of stdout, but expected 1000
Received 1000 lines of stdout, but expected 1000
Received 800 lines of stdout, but expected 1000
Received 586 lines of stdout, but expected 1000
Yapapi:
#!/usr/bin/env python3
import asyncio
from typing import AsyncIterable
from yapapi import Golem, Task, WorkContext
from yapapi.log import enable_default_logger
from yapapi.payload import vm
linesToPrint = 1000
async def worker(context: WorkContext, tasks: AsyncIterable[Task]):
async for task in tasks:
script = context.new_script()
future_result = script.run(
"/bin/sh", "-c", f"seq {linesToPrint} | awk '{{printf \"%-120s\\n\", $0}}'"
)
yield script
task.accept_result(result=await future_result)
async def main():
package = await vm.repo(image_tag="golem/alpine:latest")
tasks = [Task(data=None)] * 100
async with Golem(
budget=0.0005, subnet_tag="public", payment_network="polygon"
) as golem:
async for completed in golem.execute_tasks(worker, tasks, payload=package):
numLines = len(list(filter(None, completed.result.stdout.split("\n"))))
print(f"Received {numLines} lines of stdout, expected {linesToPrint}")
if __name__ == "__main__":
enable_default_logger(log_file="hello.log")
loop = asyncio.get_event_loop()
task = loop.create_task(main())
loop.run_until_complete(task)