segmentfault

[wooddeep]

Cannot access memory at address 0x8
(gdb) bt
#0 0x00000000004583f3 in ngx_rbtree_min (sentinel=, node=0x0) at src/core/ngx_rbtree.h:74
#1 ngx_rbtree_delete (tree=0xdacce0 <ngx_event_timer_rbtree>, node=node@entry=0x7f1eedbd05e8) at src/core/ngx_rbtree.c:176
#2 0x00000000005220ed in ngx_event_del_timer (ev=0x7f1eedbd05c0) at src/event/ngx_event_timer.h:49
#3 ngx_http_upsync_clear_all_events (cycle=) at /home/lee/share/test/mgsgw/openresty-1.11.2.2/nginx-upsync-module/src/ngx_http_upsync_module.c:3612
#4 0x00000000005221e4 in ngx_http_upsync_need_exit () at /home/lee/share/test/mgsgw/openresty-1.11.2.2/nginx-upsync-module/src/ngx_http_upsync_module.c:3557
#5 0x0000000000527366 in ngx_http_upsync_begin_handler (event=0x7f1ef6ebda30) at /home/lee/share/test/mgsgw/openresty-1.11.2.2/nginx-upsync-module/src/ngx_http_upsync_module.c:2577
#6 0x0000000000469162 in ngx_event_expire_timers () at src/event/ngx_event_timer.c:97
#7 0x0000000000468dbb in ngx_process_events_and_timers (cycle=cycle@entry=0x7f1ef6ba22f0) at src/event/ngx_event.c:256
#8 0x000000000046f0d1 in ngx_worker_process_cycle (cycle=0x7f1ef6ba22f0, data=) at src/os/unix/ngx_process_cycle.c:753
#9 0x000000000046dae0 in ngx_spawn_process (cycle=cycle@entry=0x7f1ef6ba22f0, proc=proc@entry=0x46f050 <ngx_worker_process_cycle>, data=data@entry=0x0,
name=name@entry=0x69a985 "worker process", respawn=respawn@entry=-4) at src/os/unix/ngx_process.c:198
#10 0x000000000046f2a4 in ngx_start_worker_processes (cycle=cycle@entry=0x7f1ef6ba22f0, n=1, type=type@entry=-4) at src/os/unix/ngx_process_cycle.c:358
#11 0x000000000046fe3a in ngx_master_process_cycle (cycle=0x7f1ef6ba22f0, cycle@entry=0x7f1ef834f290) at src/os/unix/ngx_process_cycle.c:243
#12 0x000000000044bef4 in main (argc=, argv=) at src/core/nginx.c:367
(gdb) l

[xiaokai-wang]

@wooddeep 是master 分支？场景能描述一下吗

[gfrankliu]

I saw the same problem, with the latest master.

I think it is related to @xiaokai-wang last commit 75b4a12
Every time when we reload or stop nginx, nginx slaves crashes with signal 11 at https://github.com/weibocom/nginx-upsync-module/blob/master/src/ngx_http_upsync_module.c#L3617

@xiaokai-wang can you take a look?

(gdb) backtrace full

#0  0x000000000042b653 in ngx_rbtree_min (sentinel=<optimized out>, node=0x3e1)

    at src/core/ngx_rbtree.h:76

No locals.

#1  ngx_rbtree_delete (tree=0x1013c80 <ngx_event_timer_rbtree>, 

    node=node@entry=0x10fb578) at src/core/ngx_rbtree.c:176

        red = <optimized out>

        root = 0x1013c80 <ngx_event_timer_rbtree>

        sentinel = 0x7f6c80 <ngx_event_timer_sentinel>

        subst = <optimized out>

        temp = <optimized out>

        w = <optimized out>

#2  0x0000000000543eb5 in ngx_event_del_timer (ev=0x10fb550)

    at src/event/ngx_event_timer.h:38

No locals.

#3  ngx_http_upsync_clear_all_events (cycle=<optimized out>)

    at nginx-upsync-module-2.1.0/src/ngx_http_upsync_module.c:3617

        i = 0

        head = 0x11bbe98

        c = <optimized out>

        upsync_type_conf = <optimized out>

        has_cleared = 1

        next = 0x10fb550

        queue_event = 0x10fb550

---Type <return> to continue, or q <return> to quit---     

        upsync_server = <optimized out>

        cycle = <optimized out>

        has_cleared = 1

#4  0x0000000000544024 in ngx_http_upsync_need_exit ()

    at nginx-upsync-module-2.1.0/src/ngx_http_upsync_module.c:3562

No locals.

#5  0x00000000005468e8 in ngx_http_upsync_recv_handler (event=0x7f25a4ada4f0)

    at nginx-upsync-module-2.1.0/src/ngx_http_upsync_module.c:2777

No locals.

#6  0x0000000000430065 in ngx_close_idle_connections (

    cycle=cycle@entry=0x10f2d70) at src/core/ngx_connection.c:1333

        i = 13

        c = <optimized out>

#7  0x0000000000448b73 in ngx_worker_process_cycle (

    cycle=cycle@entry=0x10f2d70, data=data@entry=0x0)

    at src/os/unix/ngx_process_cycle.c:767

        worker = 0

#8  0x000000000044701c in ngx_spawn_process (cycle=cycle@entry=0x10f2d70, 

    proc=proc@entry=0x448a60 <ngx_worker_process_cycle>, data=data@entry=0x0, 

    name=name@entry=0x574ef3 "worker process", respawn=respawn@entry=-3)

    at src/os/unix/ngx_process.c:199

        on = 1

        pid = 0

---Type <return> to continue, or q <return> to quit---

        s = 0

#9  0x00000000004481e4 in ngx_start_worker_processes (

    cycle=cycle@entry=0x10f2d70, n=4, type=type@entry=-3)

    at src/os/unix/ngx_process_cycle.c:359

        i = 0

        ch = {command = 1, pid = 0, slot = 0, fd = 0}

#10 0x0000000000449762 in ngx_master_process_cycle (

    cycle=cycle@entry=0x10f2d70) at src/os/unix/ngx_process_cycle.c:131

        title = 0x11f1014 "master process /opt/nginx/sbin/nginx -c /opt/nginx/conf/nginx.conf"

        p = <optimized out>

        size = <optimized out>

        i = <optimized out>

        n = <optimized out>

        sigio = <optimized out>

        set = {__val = {0 <repeats 16 times>}}

        itv = {it_interval = {tv_sec = 0, tv_usec = 0}, it_value = {

            tv_sec = 0, tv_usec = 0}}

        live = <optimized out>

        delay = <optimized out>

        ls = <optimized out>

        ccf = 0x10f4ec0

#11 0x000000000041dd22 in main (argc=<optimized out>, argv=<optimized out>)

---Type <return> to continue, or q <return> to quit---

    at src/core/nginx.c:382

        b = <optimized out>

        log = 0x7f4540 <ngx_log>

        i = <optimized out>

        cycle = 0x10f2d70

        init_cycle = {conf_ctx = 0x0, pool = 0x10f2040, 

          log = 0x7f4540 <ngx_log>, new_log = {log_level = 0, file = 0x0, 

            connection = 0, disk_full_time = 0, handler = 0x0, data = 0x0, 

            writer = 0x0, wdata = 0x0, action = 0x0, next = 0x0}, 

          log_use_stderr = 0, files = 0x0, free_connections = 0x0, 

          free_connection_n = 0, modules = 0x0, modules_n = 0, 

          modules_used = 0, reusable_connections_queue = {prev = 0x0, 

            next = 0x0}, reusable_connections_n = 0, listening = {elts = 0x0, 

            nelts = 0, size = 0, nalloc = 0, pool = 0x0}, paths = {elts = 0x0, 

            nelts = 0, size = 0, nalloc = 0, pool = 0x0}, config_dump = {

            elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, 

          config_dump_rbtree = {root = 0x0, sentinel = 0x0, insert = 0x0}, 

          config_dump_sentinel = {key = 0, left = 0x0, right = 0x0, 

            parent = 0x0, color = 0 '\000', data = 0 '\000'}, open_files = {

            last = 0x0, part = {elts = 0x0, nelts = 0, next = 0x0}, size = 0, 

            nalloc = 0, pool = 0x0}, shared_memory = {last = 0x0, part = {

              elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc = 0, 

            pool = 0x0}, connection_n = 0, files_n = 0, connections = 0x0, 

---Type <return> to continue, or q <return> to quit---

          read_events = 0x0, write_events = 0x0, old_cycle = 0x0, conf_file = {

            len = 26, data = 0x7ffcc0ad17ad "shutting down"}, conf_param = {

            len = 0, data = 0x0}, conf_prefix = {len = 16, 

            data = 0x7ffcc0ad17ad "shutting down"}, prefix = {len = 11, 

            data = 0x56f655 "/opt/nginx/"}, lock_file = {len = 0, data = 0x0}, 

          hostname = {len = 0, data = 0x0}}

        cd = <optimized out>

        ccf = <optimized out>

(gdb) 

[xiaokai-wang]

Thanks @gfrankliu .
Only happens when reloading/stoping?

[gfrankliu]

Correct

…

On Aug 15, 2019, at 3:47 AM, xiaokai ***@***.***> wrote: Thanks @gfrankliu . Only happens when reloading/stoping? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[gfrankliu]

Were you able to reproduce?

If you reload, most of time, you should see the worker process crashing, something like below in your nginx error log:

2019/08/19 03:39:23 [alert] 23219#23219: worker process 23222 exited on signal 11

BTW, It's not 100% always crashing. In case not crashing and you won't see those error logs, you will see a different problem: old worker process will stay in the "nginx: worker process is shutting down" for 30 minutes until going away. It doesn't matter what "upsync_timeout" value you set.

[gfrankliu]

BTW, if I revert commit 75b4a12 , the segfault will go away, but the other issue of "nginx: worker process is shutting down" hanging for 30 minutes still there.

[xiaokai-wang]

@gfrankliu Thanks for your infos, it's greate. I'll have a look.

[pbrzica]

Hi,

Any news regarding this issue?

[tcp9]

@gfrankliu Thanks for your infos, it's greate. I'll have a look.

我看了下，应该是从queue上拿数据的时候，指针指错了。#267 @xiaokai-wang

[xiaokai-wang]

@tcp9 thanks great. Do you have a test?

[gfrankliu]

@tcp9 thanks!
@xiaokai-wang do we need the same fix in nginx-stream-upsync-module?

[tcp9]

@tcp9 thanks great. Do you have a test?

yes, I tested it in local env, use http://tengine.taobao.org/download/tengine-2.3.2.tar.gz with

./configure --add-module=modules/ngx_http_upstream_check_module --add-module=/path/to/nginx-upsync-module/    --with-debug --with-cc-opt="-O0" 

seem to be sloved, any more test could i do?

[xiaokai-wang]

@gfrankliu yeah, I think it's necessary.
@tcp9 please sync the change to the nginx-stream-upsync-module.