Investigate K8s resource management / rate limiting

[willcl-ark]

Avoid taking down whole network when attacking, instead rate-limit attackers?

[josibake]

per my comment #465 (comment), perhaps using resource quotas at the namespace level would work? cc @m3dwards

[josibake]

Adding this as a milestone for Tabconf. We will need to either a) be sure this is not an issue or b) have some idea of how to limit resources in a namespace to prevent users from borking the cluster.

[m3dwards]

We can give commanders (and tanks) resource limits which I think would cover us?

No of course that won't work as you could just spin up many commanders. Happy to look into namespace resource limits.

[willcl-ark]

I still don't understand particuarly what other thing we are looking for here that's not #465?

Just some other native k8s way of doing it? Or are you guys trying to only limit cpu/memory resources (which we already have, no?)

[josibake]

Right, I think this issue is a bit confusingly named. So starting with the problem: we won't want people to bring down the network by doing expensive attacks. To me, the most obvious way to do this is limit the resources any single node has access to (via our current methods of requests/limits). You can't really blast out tons of network calls if your CPU is being throttled. Secondly, we can place resource limits on the namespace, such as "blue team can only have up to X total CPU and memory in this namespace."

This feels more straightforward to me than trying to implement networking policies on the namespaces and more reflective of the real world. I am very uncomfortable with any solution that involves fiddling with network traffic as it's much harder, imo, to reason about and can have weird side effects.

@willcl-ark perhaps you can provide a better justification for why you think network rate limiting is the right approach here and why namespace limits wouldn't solve the problem?

[willcl-ark]

My understanding was that k8s already natively handles CPU and memory resources, everything is vCPU (and I guess "vRAM" too), so we already can't exhaust a "real" cluster. The story may be different on a simulated single-machine cluster. Additionally we could artificially resource contstrain nodes too, if desired.

Therefore I was trying to address someone saturating network bandwidth, as AFAIK this is unconstrained.

My worry if we resource (CPU/memory) throttle nodes, one could lock them up by simply sending them so much work to do that they will never finish it. This would not be the spirit of the "crash" we are attempting to have folks demonstrate.

If we alternatively throttle bandwidth, it is the attacker who will find themselves constrained, not the node.

Hope this clears up my perspective.

[josibake]

The story may be different on a simulated single-machine cluster

Not sure I follow? Whatever resource/limits we specify should work regardless of how it is deployed since we are using docker desktop/minikube/k8s.

My worry if we resource (CPU/memory) throttle nodes

To be clear, my proposal is to set limits at the namespace level so that between all of their nodes, attackers can't request a disproportionate amount of CPU/RAM. Individual nodes will already need resource/limits set on them to a) provide better hints to k8s on scaling and b) better reflect what a real user would be running.

If we alternatively throttle bandwidth, it is the attacker who will find themselves constrained, not the node.

This feels fragile to me and doesn't reflect real world usage. In a real attack, you do have CPU and RAM limits (i.e., the size of the machine you are running the node on) and you do not have network throttling.

If you could demonstrate how the network throttling is needed by setting up a network where nodes have "realistic" (but on the generous side!) resources and show that you are still able to bring down the the whole network, I'd be more convinced? But as of now I'm not seeing the need for network throttling and worry it creates too artificial of an environment.

[willcl-ark]

The story may be different on a simulated single-machine cluster

Not sure I follow? Whatever resource/limits we specify should work regardless of how it is deployed since we are using docker desktop/minikube/k8s.

If I deploy 50 nodes on one machine then pin my CPU to 100% using one or two commanders then I effectively lock the whole machine. We have seen this numerous times before. This, I am assuming, will behave differently when the nodes are on seperate (v)machines or on a real hypervisor.

My worry if we resource (CPU/memory) throttle nodes

To be clear, my proposal is to set limits at the namespace level so that between all of their nodes, attackers can't request a disproportionate amount of CPU/RAM. Individual nodes will already need resource/limits set on them to a) provide better hints to k8s on scaling and b) better reflect what a real user would be running.

If we alternatively throttle bandwidth, it is the attacker who will find themselves constrained, not the node.

This feels fragile to me and doesn't reflect real world usage. In a real attack, you do have CPU and RAM limits (i.e., the size of the machine you are running the node on) and you do not have network throttling.

There are always network limits? It's the network you are running on.

If you could demonstrate how the network throttling is needed by setting up a network where nodes have "realistic" (but on the generous side!) resources and show that you are still able to bring down the the whole network, I'd be more convinced? But as of now I'm not seeing the need for network throttling and worry it creates too artificial of an environment.

I'm not about to demonstrate it, nor will I be there at TABConf :(

I'm only considering if you have a 100 node network with normal network activity, and additionally 20 folks all running multiple scripts trying to spam 4 billion addr messages, hugh INVs, huge GETDATAs, low difficulty headers etc. that it may be the network that packs up first?

Sure, in the real world network bandwidth is bigger, but node resources are bigger too. Artificially constraining them (both) in a system like this for testing makes sense to me. But it seems naive to give a node raspi-level resources and let someone unleash some crazy 1GBps (per commander) attack on it (IDK what folks will invent). Perhaps it's useful though to see if a node will crash when resource-constrained under TCP/UDP flood, some amplification attack, or being spammed by open/close connection requests. To be fair, not all of those require large amounts of bandwidth necessarily.

IMO it seems more reasonable to limit nodes to x RAM, y CPU, and constrain bandwidth to e.g. zMBps on the network. A small-scale simulation all around.

That said, I don't particularly enjoy needing to touch the CNI...

[josibake]

If I deploy 50 nodes on one machine then pin my CPU to 100% using one or two commanders then I effectively lock the whole machine

Is this without specifying any resource/limits for the pods? This is exactly why I worked on adding the ability to specify resource/limits so would be very surprised if you can brick your machine with those in place.

But it seems naive to give a node raspi-level resources and let someone unleash some crazy 1GBps (per commander) attack on it (IDK what folks will invent)

I feel like you're not understanding my proposal. We can set namespace level resources, in addition to node level resource limits. It does not matter how many commanders you want to spin up, they will need to all share the limits set at the namespace level. Thus, I don't see how you can launch such a massive attack without having your CPU/RAM throttled.

and constrain bandwidth to e.g. zMBps on the network

It's not at all clear to me what effects this will have on the running nodes. It also sounds like you're not interested in testing this or demonstrating its need, so I'd suggest we leave networking alone until we are sure we need to touch it.

[willcl-ark]

Sure lets leave this

[m3dwards]

I'm in favour of this remaining open as I think anything we can do to minimise the chances of things grinding to a halt at Tabconf could be worth it.

Things that have been mentioned are:

• Rate limiting on network
• tank / commander cpu and memory limits
• CPU / memory limits applied to namespaces so that one team can't exhaust resources from their competitors.

This feels like polish though if everything else is working well.