more flexible replica select strategy

[glorv]

Feature Request

Is your feature request related to a problem? Please describe:

In tidb-cloud, we want to use follower read to minimize the cross AZ traffic cost. Currently, tidb and client-go provide dynamic replica selection base on store label. If we simple send all read/write traffic to the nearest store(by set tidb_replica_read = 'closest-replicas') When the traffic is not evenly distributed across each AZ, the resources of some stores may be used up while other stores are free. So we need a more flexible replica select strategy that prefers some restriction but also take the backend stores' state into consideration.

Describe the feature you'd like:

Add a new config in client-go to let StoreSelector also consider store running statistics when select a peer. RegionRequestSender need to run a background job to fetch the realtime statistics of each store either from pd or directly from each store periodically.
Add a new field perferedLabels in storeSelectorOp, these labels affect the store selection in a best-effort way, stores match the most labels with the highest priority. if there are no certain stores available or the store is unhealthy, we can fallback to other store.

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:

[glorv]

@disksing @youjiali1995 PTAL

[disksing]

I think it is a good idea to extend storeSelectorOp to support the prefer strategy. My concern is that the maintenance cost of periodically pulling statistics may be too high.
I was wondering if it might be possible to carry some extra information in the request sent to tikv, and then put the judgement of whether to process the message to tikv. If it can't process it, return an error to client-go for it to try to connect to the next tikv.