Remove peers_watcher dependency on pubsub

[aritrbas]

This PR refactors the BGP peer management architecture to (1) eliminate the pubsub/event channel usage from PeerWatcher and SecretWatcher and (2) achieve a clear separation of the watching logic from the handling logic.

Earlier,

• PeerWatcher contained both watching logic and business logic (selectPeers() and state reconciliation).
• PeerHandler had an event loop, making it a "server" instead of a pure "handler".
• SecretWatcher sent events through pubsub instead of direct calls.
• PeerHandler maintained its own BGPConf (duplicate state) instead of using the shared Felix cache.

Now,

• All pubsub/event channel usage have been removed from PeerWatcher. PeerWatcher now directly calls handler.ProcessPeers(peers).
• PeerWatcher only watches Calico BGPPeer resources, all business logic (selectPeers() and state reconciliation) have been moved to PeerHandler.
• SecretWatcher no longer sends events (no pubsub), instead it directly calls registered handlers.
• PeerHandler has pure business logic now, all event looping and callback registrations have been removed. PeerHandler now exposes methods that are called directly by RoutingServer's event loop.
• RoutingServer's event loop handles PeerNodeStateChanged events. BGPWatcher event loop receives the event and calls peerHandler.OnPeerNodeStateChanged(old, new).
• PeerWatcher → SecretWatcher relationship has been preserved. PeerWatcher still manages secret sweep for referenced secrets and calls secretWatcher.SweepStale(activeSecrets) after processing peers.

Architecture Changes

Before (Event-Based):

SecretWatcher ──BGPSecretChanged event──> routingServerEventChan
                                              │
                                              ├──> PeerWatcher (has state, event loop)
                                              │    ├── ProcessPeers(peers, state, secrets)
                                              │    └── selectPeers() logic
                                              │
                                              └──> BGPWatcher

After (Direct Calls, No Event Loop in Handler):

SecretWatcher ──RegisterHandler──> PeerWatcher ──calls──> PeerHandler (owns state, business logic)
      │                                                        │
      │                                                        ├── ProcessPeers(peers)
      │                                                        ├── selectPeers() logic
      └────────────GetSecret()──────────────────────────────>  └── getSecretValue()

FelixServer:
  - Initializes PeerHandler in NewFelixServer()
  - PeerHandler uses shared cache (cache.BGPConf)

PeerWatcher:
  - Watches peers from Calico
  - Sends peer list to Handler
  - Manages secret watcher sweep

PeerHandler:
  - Owns all state and business logic
  - Gets secrets directly when needed via SecretGetter interface

RoutingServer (bgp_watcher.go):
  - Has event loop (WatchBGPPath)
  - Handles PeerNodeStateChanged by calling PeerHandler.OnPeerNodeStateChanged()

Made the changes on top of the nsk-split-svc branch for now, will rebase on top of master once those commits for single thread agent are merged.

[sknat]

Many thanks for taking a stab at this !
A couple comments inline, I think there is still room for more untangling of the logic.
Tell me if anything sounds unclear

[sknat]

Instead of passing a handler here, could we emit events ?
i.e.

felixServer.GetFelixServerEventChan() <- &common.ProcessPeers{peers.Items}

and in the felixServer main loop explicitly calling handlers

func (s *Server) handleFelixServerEvents(msg interface{}) (err error) {
...
case *common.ProcessPeers:
		s.peerHandler.ProcessPeers(evt)

[sknat]

This would then be an event

[sknat]

Same thing here, we could make OnSecretChanged be an event,
that way we wouldn't have to keep a reference to the handler here

[sknat]

I think this shouldn't be needed anymore :
Before it was more convenient to call resyncPeers() as this was doing both scanning, and constituting the local cache of Peers+Secrets.
But now that the reconstitution is done in the handlers, we can do everything there, and maybe call the same code in both cases (peerChanged & secretChanged)

[sknat]

If we rely on the fact that the peerWatcher does not need to be informed when the secret changes, and we make a common.OnSecretChanged towards the felix server, we could remove the need to have keep a sw.handler reference.

We could simply

eventChan <- &common.OnSecretChanged{...}

[sknat]

We could probably remove the reference to the secretWatcher by adding a secret store in the peerHandler. We could create events emitted by the secretWatcher

common.SecretAdded{...}
common.SecretDelete{...}

that would be received in the felixServer event loop,
call felix.peerHandler.OnSecret{Added,Deleted}(evt)
which would update a local cache map.

Below h.secretWatcher.GetSecret(secretName, secretKey) just becomes a lookup in the map in question

[sknat]

I think for simplicity, if we remove the need for handler's registration by introducing events
we could instantiate the secretWatcher directly here
i.e.

secretWatcher: watchers.NewSecretWatcher(k8sclient, log.WithFields(logrus.Fields{"component": "secret-watcher"}))

[sknat]

Could we make this w.secretWatcher.OnPeerListUpdated(peers.Items) and have the sweep logic in the secret watcher ?

[sknat]

Following on the comment above, we could directly start watching for secrets when OnPeerListUpdated() is called with a new Peer that has secrets attached.

And emit events when we receive the secret's contents.

[sknat]

Now that we have an isolated watcher, we could implement the watch logic :

• we would cache the list of peers locally on the peerWatcher
• when update/delete/create happens, we update the list, and send a copy as an event over the eventChan

We only resort to listing again if an error happens

[sknat]

Further simplification would be to send separated events for ADD/DEL/UPD over the eventchan, but this will require changing the handler's logic which might be beyond the scope of this PR