Still not finaly fixed: web based upgrade and hanging / LZMA read errors with serial Flashes E1200 & co.

[maverick3005]

In Reference to:
#23

There is still an issue when remote updating a few devices.

Based of my finding from the last days, today i have repeated new flashings. and the same device, which hangs last week hangs again,
shows today the old "lzma read errors" in tri-state.

This device has one changed option:
The WIRELESS Module wl0 on this device is actualy on DISABLED. The other devices have Wifi enabled.

So basicly when wireless is anyway not enabled i asume another "timing" in preparation for web based flashing,
maybe it´s faster.

We should better handle the situation when in middle of a slow serial flashing an LZMA-read errors comes in the way,
because of not completly clean flushed buffers and "unmount", eventualy rise the actual "sleep 2" a little higher,
or let the flashing runs with a check loop or use a watchdog.

At the moment, the lzma read error comes up, the reboot phase will never be reached. So all daemons are down,
interfaces are still up and device is only pingable and nothing else. Needs hard reboot and fired up with a good flashed new image.

In my case mostly when the new image differ in the filesize, this problems shows up. Flashing the same image
a few times in a row, reboot works basicly (because the byte-positions in the LZMA/Filesys are not changing)

There must still be something running, which try to read in middle of flashing from the already unmounted rootfs ?

[pedro0311]

httpd?

[maverick3005]

httpd is anyway all offline at the moment when the LZMA read error shows up on serial console. When i remember correctly as i had devices on workdesk, i saw a kernel oops, too. I have no clue, why so often still some "kernel-processes" try to access
with "read" the already unmounted squashfs at specific locations - which are changing at the moment when a complete new serial write is running. I´am still not sure with the 100% load an no response on the serial-line until the write process has ended.
Parellel-Flashing has never such issues when i remember correctly - it´s only SPI slow flashing which brings this random
problems...

[goofuns]

When I flash a Linksys E3000 with 2025.4 from the webif, it hangs, requiring a manual reboot using the power button. The flash does complete, and after power button on and off it does return with 2025.4, but it won't boot up normally without manually power cycling. I've since rolled back the unit to 2025.3

[maverick3005]

There are still issues with read access from the running system to the before mounted LZMA sqazashfs.
You can see this only on the serial console in realtime corectly (because often with a segfault or kernel oops, so
logging/remote logging brings nothing, httpd(s)+ssh+telnet have to be off when the reflashing starts....

The Scritpting itself shows a clean sync, sleep, unmount, but this shoulds not be enough.
When i add echo "3" > /proc/sys/vm/drop_caches && sleep 2 && sync && sleep 2, there is much more "flushed"
before the unmount, but this finaly doesn´t solve the issue.

I asume some running kernel lowlevelprocesses still need random access to the before mounted squashfs, so it´s not
really a daemon - they all got killed, except snmp cleanly, when used (still missing pid / start / stop support)

The console shows that the running kernel expect some data at a specific position in the flash - but this becomes changed
when flashing. So flashing the exactly same version usualy the unit reboots cleanly. but when there is a small difference in
the new binary, like smaller or bigger, that the typical errors with LZMA read error comes back.

On ARM i never saw this.

What i can redroduce is, when WIFI is disabled, the devices usualy hangs mostly, when it´s enabled, it is more or less random.
So i asume a timing issue: When WIFI is disabled, the process killings are a little faster. My conclusion:
some "sleeps" needs to be revised. Wait a little bit longer, something like.

What i don´t understand is the fact, that when the filesystem is cleaning disamounted, why there is still a kernel read-access ?

My idea is, that still some daemon is open when flashing is running (httpd/s) ? ssh ? telnet ?

The original Flashing Routines usualy kill all running tasks, then made a complete flash-erase, except the bootboolader, and finaly
reprogramm it with the entire flashfile. I see this in the Stock Firmware via the serial console.

anyway my E3000+E3200 doesn´t have this issue, but good to know, that parallel flash devices are affected, too :-(

[HommeOursPorc]

@maverick3005 do you do "dirty" upgrade or you erase nvram each time? I'm asking because I'm wondering if it is trying to reach some nvram instead of the Linux/os partition?

[goofuns]

The latest mars 2025.4 build for the E3000 does the same. Flashed from the webif, everything does as it should, until it reboots, once it reboots and comes back up, it's not accessible until a manual power cycle. This is from a clean nvram. And I noticed that as well, sometimes yes it hangs, sometimes it doesn't, for me it didn't matter if WiFi was enabled or not.

[maverick3005]

dirty flashing in my case, because when just a few lines are changed and nothing really new with new nvram variables, complete
reset is not needed here.

I have only the hard-resetting power/on problems when initial flashing from dd-wrt -> freshtomato.
arp clear on the local machine is anyway needed (or wait +300 seconds for invalidated ARP-cache....)
(but that is a one time prepare thing, which i handle anyway directly on the workdesk, so no problem)

I can tell you that the very latest E3000 i can always flash by web and it always reboots. On the other side,
i have a very castrated profile created and not a full featured VPN-AIO. So many things i tooked out,
which i never need. My Image is just 6 MB in size. Same i made for the E3200.

anyway...

Because of no local access to a few devices and ONLY ssh was still working i made a quickfix:

i use the trx-file which starts with "HDR0" - so not the trailed build.
put this file as "code.bin" to a webserver (http) and downloded the good image to /tmp/code.bin via wget
(my mips device has no scp or sftp active in dropbear - the arm have... so the workaround with the wget-method)

Then i prepared the device with some shell commands:

• killed all running daemons excluding the ssh shell
• echo "3" > /proc/sys/vm/drop_caches
• sync

and then:

mtd-write -w -i /tmp/code.bin -d linux

after 1 minute the prompt was back and i closed the thing with "reboot".
2 minutes later devices was alive again (not really all, 1 device lost the ssh connection, same way as the web-flashing
situation is. So an LZMA error as usual must be came up or an kernel oops). So Hard Power-ON/OFF was (again) needed.

My Conclusion: mtd-write itself works not reliable as other "mtd write" methods, like used on dd-wrt or openwrt actualy.

...not the correct method this way, but i had no other choice from the far end to bring back the devices quickly..

[maverick3005]

The same after fresh reboots again. web flashed a little changed new firmware, and always the same router is crashing.
But... all devices are identicly including cfe-version, same flash chips and same config, except IPs+1 and other MACs.

So i really bet - and with some parallel flashings there is the same effects... we should review the "mtd-write"
and some other preparations. Maybee a compare with openwrt "mtd" can be useful?

And as i used the ssh/shell/fix flash method before, there was an other device which defuncted after flashing.

The usual tftp cfe-method, including a mini-version from dd-wrt, is not working on the E1200v2, cfe in serial console
always write "out of memory". It works only when the download is directly initiated by cfe commands - which are different
from the 3 seconds tftp timeout method. So... no serial console - no quick tftp solution - when devices on the far end....

[maverick3005]

Just yesterday:

From 5 devices E1200, one device again was not rebooting and hungs. It´s much better after the last commits, but finaly not
reliable when the device is on a remote/unwatched side.

[HommeOursPorc]

What if you stop all wireless services before an upgrade? Will this makes things better?

[maverick3005]

Not really. Have this testet long time before. It is just random for the same type of hardware with identical firmware.

A clean reboot before flashing results in the same random problems. So when "mass updating" the same type of Router (E1200v2),
usualy 1 device by random hangs. Still the lzma-read error when upgrading. This is curios, because the filesystem is dismounted before, but there must still be something buffered in the kernel-space which need read-access to a specific position at the image, which is overwritten and no longer active or available. Visible is this only with serial console directly. A logfile/entry is never written, because of complete dismounted filesystem and/or stopped remote-syslog.

A simple test is to overwrite the same version more times, so basicly the byte position of the filesystem is equal.
A good trigger is a little bigger or smaller new image, which have complete other datapositions, like 100kb more or less in filesize.