Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[coproc] Async engine testng #2369

Closed
1 task done
VadimPlh opened this issue Sep 21, 2021 · 7 comments
Closed
1 task done

[coproc] Async engine testng #2369

VadimPlh opened this issue Sep 21, 2021 · 7 comments
Assignees
@VadimPlh
Labels
area/coproc Legacy WASM coprocessor, please use area/wasm instead area/tests kind/enhance New feature or request
Milestone
Wasm Async Engine...

Comments

@VadimPlh
Copy link
Contributor

VadimPlh commented Sep 21, 2021 •
edited
Loading

Limits

Setup:

  • 1machine (c5.9xlarge) the machine has:
  • 36 vcpus
  • 3 TB storage (external volume)
  • 72GB RAM (2GB/vcpu)
  • On this one machine, 3 redpanda processes and 3 wasm engine sidecars deployed.
  • 20 partitions
  • record size: 500 bytes
  • messages count: 50000
  • -c 5 -g 10G
  • Throuputs 1000 record/sec
  • Coproc perfomance tests VadimPlh/redpanda#8

Testing

Coproc count Redpanda settings Node CPU Read info Notes
5 default 30 fetch.time.ms: 47367, fetch.MB.sec: 0.5033, fetch.nMsg.sec: 1055.5872
15 default 50-60 fetch.time.ms: 46853,, fetch.MB.sec: 0.5089,, fetch.nMsg.sec: 1067.1675
30 default 80-90 fetch.time.ms: 4732,, fetch.MB.sec: 0.5039,, fetch.nMsg.sec: 1056.8143 2 rpc - transport.cc:233 - Request timeout and 2 coproc - event_listener.cc:184 - Wasm engine failed to reply to heartbeat within the expected interval
60 default 80-90 fetch.time.ms: 96771,,, fetch.MB.sec: 0.2464,,, fetch.nMsg.sec: 516.6837 > 30 Request timeout and heartbeat problem
100 default 100-110 11184 timeouts, 32 heartbeats problem

Chaos testing

  • done
@VadimPlh VadimPlh self-assigned this Sep 21, 2021
@VadimPlh VadimPlh mentioned this issue Sep 21, 2021
Closed
41 tasks
@jcsp jcsp added area/coproc Legacy WASM coprocessor, please use area/wasm instead area/tests kind/enhance New feature or request labels Sep 22, 2021
@VadimPlh
Copy link
Contributor Author

#2482

@VadimPlh
Copy link
Contributor Author

VadimPlh commented Oct 5, 2021 •
edited
Loading

Testing v2

  • COPROC_COUNT=60
  • RECORD_SIZE=500
  • PARTITIONS=20
  • MESSAGES_COUNT=20000000

After ~50 minutes work

input

[root@ip-172-31-4-84 coproc-test]# ../vbuild/go/linux/bin/rpk topic describe one_to_one_10 -p
PARTITION  LEADER  REPLICAS  LOG-START-OFFSET  LAST-STABLE-OFFSET  HIGH-WATERMARK
0          0       [0 1 2]   0                 230426              230394
1          1       [0 1 2]   0                 237661              237661
2          0       [0 1 2]   0                 230825              230825
3          1       [0 1 2]   0                 240424              240424
4          1       [0 1 2]   0                 242600              242600
5          1       [0 1 2]   0                 241873              241873
6          0       [0 1 2]   0                 231050              231018
7          2       [0 1 2]   0                 235904              235904
8          2       [0 1 2]   0                 236864              236832
9          2       [0 1 2]   0                 235936              235936
10         2       [0 1 2]   0                 235904              235904
11         2       [0 1 2]   0                 235937              235937
12         2       [0 1 2]   0                 240097              240097
13         2       [0 1 2]   0                 236000              235968
14         1       [0 1 2]   0                 245664              245664
15         2       [0 1 2]   0                 235968              235968
16         2       [0 1 2]   0                 235936              235936
17         2       [0 1 2]   0                 235520              235520
18         2       [0 1 2]   0                 237312              237312
19         2       [0 1 2]   0                 234200              234200

output

[root@ip-172-31-4-84 coproc-test]# ../vbuild/go/linux/bin/rpk topic describe one_to_one_10._output_ -p
PARTITION  LEADER  REPLICAS  LOG-START-OFFSET  LAST-STABLE-OFFSET  HIGH-WATERMARK
0          0       [0 1 2]   0                 45632               45568
1          1       [0 1 2]   0                 45377               45377
2          0       [0 1 2]   0                 45507               45507
3          1       [0 1 2]   0                 45244               45244
4          1       [0 1 2]   0                 45280               45280
5          1       [0 1 2]   0                 45506               45506
6          0       [0 1 2]   0                 45440               45440
7          2       [0 1 2]   0                 45952               45952
8          2       [0 1 2]   0                 45952               45952
9          2       [0 1 2]   0                 45888               45888
10         2       [0 1 2]   0                 46016               46016
11         2       [0 1 2]   0                 46017               46017
12         2       [0 1 2]   0                 45953               45953
13         2       [0 1 2]   0                 45952               45952
14         1       [0 1 2]   0                 45249               45249
15         2       [0 1 2]   0                 46016               46016
16         2       [0 1 2]   0                 46016               46016
17         2       [0 1 2]   0                 45952               45952
18         2       [0 1 2]   0                 45952               45952
19         2       [0 1 2]   0                 45890               45890

@VadimPlh
Copy link
Contributor Author

VadimPlh commented Oct 5, 2021

Got in wasm deploy

ERROR 2021-10-05 13:52:02,852 [shard 2] cluster - partition.cc:165 - Can't process an idempotent request to {kafka/coprocessor_internal_topic/0}. Compacted topic doesn't support idempotency.
ERROR 2021-10-05 13:52:03,365 [shard 2] cluster - partition.cc:165 - Can't process an idempotent request to {kafka/coprocessor_internal_topic/0}. Compacted topic doesn't support idempotency.
ERROR 2021-10-05 13:52:03,679 [shard 2] cluster - partition.cc:165 - Can't process an idempotent request to {kafka/coprocessor_internal_topic/0}. Compacted topic doesn't support idempotency.
ERROR 2021-10-05 13:52:04,059 [shard 2] cluster - partition.cc:165 - Can't process an idempotent request to {kafka/coprocessor_internal_topic/0}. Compacted topic doesn't support idempotency.
ERROR 2021-10-05 13:52:04,472 [shard 2] cluster - partition.cc:165 - Can't process an idempotent request to {kafka/coprocessor_internal_topic/0}. Compacted topic doesn't support idempotency.
ERROR 2021-10-05 13:52:05,276 [shard 2] cluster - partition.cc:165 - Can't process an idempotent request to {kafka/coprocessor_internal_topic/0}. Compacted topic doesn't support idempotency.
ERROR 2021-10-05 13:52:05,632 [shard 2] cluster - partition.cc:165 - Can't process an idempotent request to {kafka/coprocessor_internal_topic/0}. Compacted topic doesn't support idempotency.
ERROR 2021-10-05 13:52:06,109 [shard 2] cluster - partition.cc:165 - Can't process an idempotent request to {kafka/coprocessor_internal_topic/0}. Compacted topic doesn't support idempotency.
ERROR 2021-10-05 13:52:06,440 [shard 2] cluster - partition.cc:165 - Can't process an idempotent request to {kafka/coprocessor_internal_topic/0}. Compacted topic doesn't support idempotency

@VadimPlh
Copy link
Contributor Author

VadimPlh commented Oct 5, 2021 •
edited
Loading

In wasm generate got error twice

Can not parse json from npm search: {}, Error: {} [[{"name":"@vectorizedio/wasm-api","scope":"vectorizedio","version":"21.9.3","description":"wasm api helps to define wasm function","date":"2021-10-04T19:51:46.430Z","links":{"npm":"https://www.npmjs.com/package/%40vectorizedio%2Fwasm-api"},"publisher":{"username":"vectorizedio","email":"[email protected]"},"maintainers":[{"username":"vectorizedio","email":"[email protected]"}]} ] ] unexpected end of JSON input
Can not parse json from npm search: {}, Error: {} [[{"name":"@vectorizedio/wasm-api","scope":"vectorizedio","version":"21.9.3","description":"wasm api helps to define wasm function","date":"2021-10-04T19:51:46.430Z","links":{"npm":"https://www.npmjs.com/package/%40vectorizedio%2Fwasm-api"},"publisher":{"username":"vectorizedio","email":"[email protected]"},"maintainers":[{"username":"vectorizedio","email":"[email protected]"}]} ] ] unexpected end of JSON input
npm created project in /home/fedora/data/redpanda/coproc-test/test

@VadimPlh
Copy link
Contributor Author

VadimPlh commented Oct 5, 2021

  • COPROC_COUNT=60
  • RECORD_SIZE=500
  • PARTITIONS=20
  • MESSAGES_COUNT=20000000

input

[root@ip-172-31-4-84 coproc-test]# ../vbuild/go/linux/bin/rpk topic describe one_to_one_22 -p
PARTITION  LEADER  REPLICAS  LOG-START-OFFSET  LAST-STABLE-OFFSET  HIGH-WATERMARK
0          0       [0 1 2]   0                 109184              109184
1          0       [0 1 2]   0                 109120              109088
2          0       [0 1 2]   0                 109182              109182
3          0       [0 1 2]   0                 109277              109245
4          1       [0 1 2]   0                 115897              115897
5          0       [0 1 2]   0                 109222              109222
6          0       [0 1 2]   0                 109153              109121
7          1       [0 1 2]   0                 113412              113412
8          0       [0 1 2]   0                 109179              109147
9          0       [0 1 2]   0                 109216              109216
10         2       [0 1 2]   0                 100761              100761
11         0       [0 1 2]   0                 109216              109184
12         2       [0 1 2]   0                 100884              100884
13         1       [0 1 2]   0                 115810              115810
14         2       [0 1 2]   0                 98527               98495
15         0       [0 1 2]   0                 109217              109217
16         0       [0 1 2]   0                 109184              109152
17         1       [0 1 2]   0                 111921              111921
18         0       [0 1 2]   0                 109120              109120
19         1       [0 1 2]   0                 113759              113727

output

[root@ip-172-31-4-84 coproc-test]# ../vbuild/go/linux/bin/rpk topic describe one_to_one_22._output_ -p
PARTITION  LEADER  REPLICAS  LOG-START-OFFSET  LAST-STABLE-OFFSET  HIGH-WATERMARK
0          0       [0 1 2]   0                 49984               49984
1          0       [0 1 2]   0                 49728               49728
2          0       [0 1 2]   0                 49950               49886
3          0       [0 1 2]   0                 50013               50013
4          1       [0 1 2]   0                 50023               50023
5          0       [0 1 2]   0                 49990               49990
6          0       [0 1 2]   0                 49729               49729
7          1       [0 1 2]   0                 49920               49856
8          0       [0 1 2]   0                 50011               50011
9          0       [0 1 2]   0                 49664               49664
10         2       [0 1 2]   0                 48511               48511
11         0       [0 1 2]   0                 49728               49728
12         2       [0 1 2]   0                 48173               48173
13         1       [0 1 2]   0                 49984               49984
14         2       [0 1 2]   0                 48409               48409
15         0       [0 1 2]   0                 50049               50049
16         0       [0 1 2]   0                 49728               49728
17         1       [0 1 2]   0                 49867               49803
18         0       [0 1 2]   0                 49984               49984
19         1       [0 1 2]   0                 49970               49970

@emaxerrno
Copy link
Contributor

emaxerrno commented Oct 5, 2021 •
edited
Loading

need.

  1. FIx cluster size to 3 nodes. 32 cores. 64G of ram. 1TB of disk. Make sure all tests are in distributed mode. (i.e. replication count 3 and partition count 20 or so)
  2. Data cardinality - i.e.: 5TB
  3. Need throughput - i.e: 1GB/s
  4. Message size
  5. Need latency graphs
  6. Coproc registered functions
  7. Coproc materialized topics (i.e.: how many child topics derived from parent as a variable)

@dswang
Copy link
Contributor

dswang commented Dec 16, 2021

Calling this done as per @graphcareful

@dswang dswang closed this as completed Dec 16, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@VadimPlh VadimPlh

Labels
area/coproc Legacy WASM coprocessor, please use area/wasm instead area/tests kind/enhance New feature or request
Projects
None yet
Milestone
Wasm Async Engine Tech Preview
Development

No branches or pull requests
4 participants
@emaxerrno @jcsp @dswang @VadimPlh