RMR version "ric message routing library on SI95 p=4560 mv=3 flg=00 id=a (d5197f5 4.8.0 built: Dec 2 2021"
rmr message sending triggers the routing entry statistics printout every ~5 minutes. When traversing entries for statistics print the mutex lock is missing . So if the routing update happens in same time, then this will cause the crash. Following is the trace (incomplete printout when printing the entry statistics):
1640562413268 1/RMR [INFO] sends: ts=1640562413 src=service-ricxapp-dddterm-rmr.ricxapp:4560 target=172.30.202.103:38000 open=0 succ=0 fail=0 (hard=0 soft=0)
1640562413268 1/RMR [INFO] sends: ts=1640562413 src=service-ricxapp-dddterm-rmr.ricxapp:4560 target=service-ricxapp-dddd-rmr.ricxapp:4560 open=1 succ=215531 fail=0 (hard=0 soft=0)
1640562413268 1/RMR [INFO] sends: ts=1640562413 src=service-ricxapp-dddterm-rmr.ricxapp:4560 target=service-ricxapp-dddd-rmr.ricxapp:4560 open=0 succ=0 fail=0 (hard=0 soft=0)
1640562413268 1/RMR [INFO] sends: ts=1640562413 src=service-ricxapp-dddterm-rmr.ricxapp:4560 target=service-ricplt-submgr-rmr.ricplt:4560 open=0 succ=0 fail=0 (hard=0 soft=0)
1640562413268 1/RMR [INFO] sends: ts=1640562413 src=service-ricxapp-dddterm-rmr.ricxapp:4560 target=service-ricxapp-ddddd-rmr.ricxapp:4560 open=1 succ=6055874824 fail=149 (hard=6 soft=143)
1640562413268 1/RMR [INFO] sends: ts=1640562413 src=service-ricxapp-dddterm-rmr.ricxapp:4560 target=service-ricplt-e2mgr-rmr.ricplt:3801 open=0 succ=0 fail=0 (hard=0 soft=0)
1640562413268 1/RMR [INFO] sends: ts=1640562413 src=service-ricxapp-dddterm-rmr.ricxapp:4560 target=service-ricxapp-ueec-rmr.ricxapp:4560 open=0 succ=0 fail=0 (hard=0 soft=0)
1640562413268 1/RMR [INFO] sends: ts=1640562413 src=service-ricxapp-dddterm-rmr.ricxapp:4560 target=service-ricplt-a1mediator-rmr.ricplt:4562 open=0 succ=0 fail=0 (hard=0 soft=0)
1640562714290 1/RMR [INFO] sends: ts=1640562714 src=service-ricxapp-dddterm-rmr.ricxapp:4560 target=service-ricxapp-dddd-rmr.ricxapp:4560 open=0 succ=0 fail=0 (hard=0 soft=0)
1640562714290 1/RMR [INFO] sends: ts=1640562714 src=service-ricxapp-dddterm-rmr.ricxapp:4560 target=service-ricxapp-ddddd-rmr.ricxapp:4560 open=1 succ=6228540660 fail=0 (hard=0 soft=0)
1640562714290 1/RMR [INFO] sends: ts=1640562714 src=service-ricxapp-dddterm-rmr.ricxapp:4560 target=172.30.202.103:38000 open=0 succ=0 fail=0 (hard=0 soft=0)
1640562714290 1/RMR [INFO] sends: ts=1640562714 src=service-ricxapp-dddterm-rmr.ricxapp:4560 target=service-ricxapp-ddddd-rmr.ricxapp:4560 open=1 succ=215832 fail=0 (hard=0 soft=0)
1640562714290 1/RMR [INFO] sends: ts=1640562714 src=service-ricxapp-dddterm-rmr.ricxapp:4560 target=service-ricxapp-dddddd-rmr.ricxapp:4560 open=0 succ=0 fail=0 (hard=0 soft=0)
1640562714290 1/RMR [INFO] sends: ts=1640562714 src=service-ricxapp-dddterm-rmr.ricxapp:4560 target=service-ricplt-submgr-rmr.ricplt:4560 open=0 succ=0 fail=0 (hard=0 soft=0)
1640562714290 1/RMR [INFO] sends: ts=1640562714 src=service-ricxapp-dddterm-rmr.ricxapp:4560 target=service-ricxapp-dddddd-rmr.ricxapp:4560 open=1 succ=6064529241 fail=149 (hard=6 soft=143)
1640562714290 1/RMR [INFO] sends: ts=1640562714 src=service-ricxapp-dddterm-rmr.ricxapp:4560 target=service-ricplt-e2mgr-rmr.ricplt:3801 open=0 succ=0 fail=0 (hard=0 soft=0)
1640562714290 1/RMR [INFO] sends: ts=1640562714 src=service-ricxapp-dddterm-rmr.ricxapp:4560 target=service-ricxapp-ueec-rmr.ricxapp:4560 open=0 succ=0 fail=0 (hard=0 soft=0)
1640562714290 1/RMR [INFO] sends: ts=1640562714 src=service-ricxapp-dddterm-rmr.ricxapp:4560 target=service-ricplt-a1mediator-rmr.ricplt:4562 open=0 succ=0 fail=0 (hard=0 soft=0)
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x1a948 pc=0x7f9874a5a0be]
runtime stack:
runtime.throw(0x10d7765, 0x2a)
/opt/go/1.16/src/runtime/panic.go:1117 +0x72
runtime.sigpanic()
/opt/go/1.16/src/runtime/signal_unix.go:718 +0x2e5
goroutine 776 [syscall]:
runtime.cgocall(0xd6312b, 0xc006251e18, 0x7f970800ade0)
/opt/go/1.16/src/runtime/cgocall.go:154 +0x5b fp=0xc006251de8 sp=0xc006251db0 pc=0x40703b
gerrit.o-ran-sc.org/r/ric-plt/xapp-frame/pkg/xapp._Cfunc_rmr_send_msg(0x7f9820000d10, 0x7f9784368f90, 0x0)
_cgo_gotypes.go:420 +0x49 fp=0xc006251e18 sp=0xc006251de8 pc=0xaf8d69
gerrit.o-ran-sc.org/r/ric-plt/xapp-frame/pkg/xapp.(*RMRClient).SendBuf.func3(0xc00020ca80, 0x7f9784368f90, 0xc006251ee8)
/ws/cache/gocache/pkg/mod/gerrit.o-ran-sc.org/r/ric-plt/xapp-frame.git@v0.9.8/pkg/xapp/rmr.go:430 +0x86 fp=0xc006251e50 sp=0xc006251e18 pc=0xafe3e6
gerrit.o-ran-sc.org/r/ric-plt/xapp-frame/pkg/xapp.(*RMRClient).SendBuf(0xc00020ca80, 0x7f9784368f90, 0x7f9784368f00, 0x0, 0x0)
/ws/cache/gocache/pkg/mod/gerrit.o-ran-sc.org/r/ric-plt/xapp-frame.git@v0.9.8/pkg/xapp/rmr.go:430 +0x3eb fp=0xc006251ec0 sp=0xc006251e50 pc=0xafb74b
gerrit.o-ran-sc.org/r/ric-plt/xapp-frame/pkg/xapp.(*RMRClient).Send(0xc00020ca80, 0xc004962f80, 0xe4b900, 0xc05c831ac0)
/ws/cache/gocache/pkg/mod/gerrit.o-ran-sc.org/r/ric-plt/xapp-frame.git@v0.9.8/pkg/xapp/rmr.go:410 +0x6f fp=0xc006251ef8 sp=0xc006251ec0 pc=0xafb30f
main.(*dddTerm).ConverterSend(0xc000562460, 0xc004962f80, 0x16)
/ws/cmd/dddterm/dddterm.go:232 +0x3e fp=0xc006251f28 sp=0xc006251ef8 pc=0xd58c3e
main.(*pipelineSendGeo).Run(0xc09ac6caa0)
/ws/cmd/dddterm/conv_geo.go:59 +0x1c2 fp=0xc006251f80 sp=0xc006251f28 pc=0xd531c2
ddddd.com/ric/dddddpkgs/pkg/worker.(*WorkerQueue).Handle(0xc011ace240, 0x0)
/ws/cache/gocache/pkg/mod/bhgerrit.ext.net.nokia.com/ric/nokiapkgs.git@v1.7.0/pkg/worker/workerqueue.go:87 +0xd9 fp=0xc006251fd0 sp=0xc006251f80 pc=0xb7c179
runtime.goexit()
/opt/go/1.16/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc006251fd8 sp=0xc006251fd0 pc=0x470d81
created by ddddddd.com/ric/nokiapkgs/pkg/worker.(*WorkerQueue).ResizeRunners
/ws/cache/gocache/pkg/mod/ddddddd.com/ric/ddddd.git@v1.7.0/pkg/worker/workerqueue.go:109 +0x7f