The metric tidb_tikvclient_batch_recv_latency is not reasonable

[you06]

Found by @zyguan

Start a cluster and wait a while, then run some workload, we got "Batch Receive Average Duration" panel like this.

The way we observe the receiving time is not reasonable. In an idle cluster, this function is waiting for responses a long time while there are no in-fly requests, so the receiving time is high. In a busy cluster, the responses are pending in the client's buffer and we just loop reading them from memory, so the receiving time is extremely low(avg 547us).

client-go/internal/client/client_batch.go

Lines 427 to 449 in 79a0fc5

	func (s *batchCommandsStream) recv() (resp *tikvpb.BatchCommandsResponse, err error) {
	now := time.Now()
	defer func() {
	if r := recover(); r != nil {
	metrics.TiKVPanicCounter.WithLabelValues(metrics.LabelBatchRecvLoop).Inc()
	logutil.BgLogger().Error("batchCommandsClient.recv panic",
	zap.Reflect("r", r),
	zap.Stack("stack"))
	err = errors.New("batch conn recv paniced")
	}
	if err == nil {
	metrics.BatchRecvHistogramOK.Observe(float64(time.Since(now)))
	} else {
	metrics.BatchRecvHistogramError.Observe(float64(time.Since(now)))
	}
	}()
	if _, err := util.EvalFailpoint("gotErrorInRecvLoop"); err == nil {
	return nil, errors.New("injected error in batchRecvLoop")
	}
	// When `conn.Close()` is called, `client.Recv()` will return an error.
	resp, err = s.Recv()
	return
	}

[disksing]

maybe change it to another title? such as 'Batch Wait for response duration'?