Tcp_bind() fails after object delete/re-create RP2040 Philhower Core

[BetterComputing]

I am using the Philhower RP2040 core, version 4.4.3, and a WIZNet W5500 Ethernet chip. Rather than creating a global reference to Wiznet5500lwIP, I am using a pointer that I allocate and free as the system starts and stops. (In case you are wondering why, it's because I also plan to send commands to the WIZNet W5500 Ethernet chip to turn it off/put it in low power mode and then back on). I have run into a situation where tcp_bind() fails and I can't properly control the cleanup of the internal data structures that cause the problem.

Attached is sample code that should demonstrate the issue. I am using an HTTP server based on HTTP by Giuseppe Masino. I have included an earlier version of that library because I couldn't get the most recent (7 years old) to compile. So, don't install the HTTP library into Arduino IDE, just use the included code.

MAC address and IP settings are near the beginning of the .ino file - adjust for your network. I hard coded the IP address settings because the network connection starts faster.

To run the example code you'll need a Raspberry Pi Pico, a WIZnet W5500 board, and a pushbutton. The WIZnet is connected to CS 17 and SPI0. The button can be connected to any available pin - I used pin 28. To debug, I use a PicoProbe.

The project setup is mostly defaults. Optimize is turned off to allow set of breakpoint at tcp_bind() result:
· Board: Raspberry Pi Pico
· Debug Level: "None"
· Debug Port: "Disabled"
· C++ Exceptions: "Disabled"
· Flash Size: "2MB (no FS)"
· CPU Speed: "133 MHz"
· IP/Bluetooth Stack: "IPv4 Only"
· Optimize: Disabled (-O0)
· Profiling: "Disabled"
· RTTI: "Disabled"
· Stack Protector: "Disabled"
· Upload Method: "Picoprobe/Debugprobe (CMSIS-DAP)"
· USB Stack: "Pico SDK"

After the code is built and running, press the button to start the Ethernet chip and servers. Notice that ping of DemoOfFailedTCPBind.local works and a browse to http://demooffailedtcpbind.local/ returns the demo web page.

If the button is pressed again, the servers are stopped and the Ethernet object is closed and deleted. Now ping and the demo web page are no longer available.

Here is the problem: If the button is pressed again, which starts Ethernet and the webserver again, the tcp_bind() that is called by the webserver begin() will fail. To see this, set a breakpoint in arduino15/packages/rp2040/hardware/rp2040/4.4.3/libraries/WiFi/src/WiFiServer.cpp, line 68.

Note these observations: tcp_bind() will always succeed if (a) I never refresh the web page http://demooffailedtcpbind.local/, (b) I restart the debugger, or (c) if I wait EXACTLY 2 minutes or more after the web client closes.

Can anyone help me solve this issue? Am I using the client code correctly? Is there a re-init function for the W5500lwIP code?

I also don't see how the translation layer lwip_wrap.cpp in the Philhower core is built - the functions with names like __real_lwip_init() and __real_tcp_bind(). I also cannot debug it. I did try using PlatformIO but those functions are just assembly language code.

Any feedback would be appreciated.

DemonstrateFailedTCPbindTelnet.ino.zip

[earlephilhower]

Can you pare down things to just a simple sketch that doesn't have its own HTTP server implementation and other things going on? Just start the Ethernet device, start the WiFiServer, wait for that button (or a character on the Serial port), disable it, wait for one more event, then start a new WiFiServer?

I'm doubt that LeaMDNS will work properly when interfaces are added after the initial startup, off the bat. It's not getting notifications that interfaces are added so unless you manually restart everything it'll not be listening on the new device.

AIUI, the way LWIP works I think a bind to 0.0.0.0 really does a bind across all netifs and isn't a real wildcard that's parsed every connection. When you stop/start the Ethernet device you're making a new netif which isn't going to be on any bind lists. (And if your manual HTTP server implementation is not completely destroying and newing/begining a complete new WiFiServer that's not going to work with your deletion of netifs)

The __real and __wrap functions are using GCC -Wl,--wrap=xxx (see lib/core_wrap.txt) to implement the LWIP lockout since LWIP doesn't play nice with concurrency of any kind.

[earlephilhower]

Oh, and this bit is unneeded with unsigned subtraction:

    if (currentMillis < previousMillis)
    {
        // Overflow has occurred
        timeDifference = UINT32_MAX - previousMillis + currentMillis;
    }
    else
    {
        // No overflow
        timeDifference = currentMillis - previousMillis;
    }

Just subtract the two and the magic of 2's complement will make it all just work

timeDifference = currentMillis - previousMillis;

[BetterComputing]

I have reduced the code down to just a Telnet server and can reproduce the problem (attached).

MAC address and IP settings are near the beginning of the .ino file - adjust for your network. I hard coded the IP address settings because the network connection starts faster.

To run the example code you'll need a Raspberry Pi Pico, and a WIZnet W5500 board. The WIZnet is connected to CS 17 and SPI0. To debug, I use a PicoProbe which uses Serial1 - I placed a define PREFERRED_SERIAL so you can easily change to another.

After the code is built, open a serial terminal and run the code. You should see "Demonstrate Failed TCP bind" and a "command:" prompt. Enter the command "on" and press enter. You should see "Turn ON requested." and "MDNS responder started".

Open a Telnet client and connect to "DemoOfFailedTCPBind.local". The code will send 80 characters followed by CRLF and then close the connection.

Back at the serial terminal, enter the command "off" and press enter. ,You should see "Turn OFF requested." Finally, enter the command "on" and press enter. You will see the normal on messages and then "TCP bind failed." Notice that the Telnet server at "DemoOfFailedTCPBind.local" is no longer available.

The error within the bind failure is due to the address already in use - returns -8 (ERR_USE). This was determined by modifying the local copy of ~/.arduino15/packages/rp2040/hardware/rp2040/4.3.1/libraries/WiFi/src/WiFiServer.cpp at or about line 67 and observing the return value of tcp_bind(). I do not know how to build and debug this code properly so I cannot offer a fix.

Note these additional observations: tcp_bind() will always succeed if (a) I never make the Telnet connection, (b) if I wait EXACTLY 2 minutes or more after the web client closes, or (c) I restart the debugger.

DemonstrateFailedTCPbindTelnet.ino.zip

P.S. LeaMDNS is working as expected in this sample code.

[earlephilhower]

(b) if I wait EXACTLY 2 minutes or more after the web client closes

There's your problem, welcome to TCP connection states.

I've been bitten by this since the 90s writing PERL web services on SUN workstations where I needed to run the same script multiple times while fixing errors. First run is great, crashes, and on restart it can't bind to the port unless I wait a bit. I'm not a TCP expert, but AIUI, TCP sockets get into a TIME_WAIT state (making sure any outstanding packets have time to clear), and you can't re-bind that port for a fixed period. It's just the way the network stack works: https://stackoverflow.com/questions/15198834/bind-failed-address-already-in-use

MDNS is UDP and doesn't have any connection states, so the concept of a time delay for remaining packets to clear out doesn't exist.

On Linux/POSIX there's SO_REUSEADDR to avoid the issue. For LWIP it seems to be a compile-time option for all connections, but I can only find something for socket mode (we use RAW so it should be a no-op). https://www.nongnu.org/lwip/2_0_x/group__lwip__opts__socket.html#gaf3822feed320cf8439b083ee525e4942

You could try adding #define SO_REUSE 1 to tools/libpico/lwipopts.h and running ./make-libpico.sh to rebuild the libs and see if it does have an effect.

If you fork this repo you can run a GH action to build libpico on a branch manually. Fork, then in your fork do the change in a branch and push to GH (or use the online editor to make the edit/branch). Then go to the Actions menu on top of the repo web page, select libpico builder on the left, then use the menu popup on the right to manually start a run on the branch. It'll upload a ZIP of the libs you can overwrite your copies with. Also, please be sure to copy the edited lwipopts.h in ./include to match your branch's changes...