1. Whenever E2T pod Stops and Starts (Restart), E2T keep sending the E2_TERM_INIT to E2M until E2T receives the E2_TERM_KEEP_ALIVE_REQ from E2M. which is happening as expected.

            _Line 23: {"ts":1634559366146,"crit":"INFO","id":"E2Terminator","mdc":

{"PID":"25","POD_NAME":"e2term-alpha-568687b664-nlb94","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"master","SYSTEM_NAME":"SEP"}

,"msg":"E2_TERM_INIT successfully sent "}_

            _Line 64: {"ts":1634559466240,"crit":"INFO","id":"E2Terminator","mdc":

{"PID":"25","POD_NAME":"e2term-alpha-568687b664-nlb94","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"master","SYSTEM_NAME":"SEP"}

,"msg":"E2_TERM_INIT successfully sent "}_

 

1. The moment E2T receives the E2_TERM_KEEP_ALIVE_REQ from E2M, E2T will response to the E2M with E2_TERM_KEEP_ALIVE_RESP and stops sending the E2_TERM_INIT. This is also working as expected

_

{"ts":1634559565001,"crit":"DEBUG","id":"E2Terminator","mdc":

{"PID":"139686259332864","POD_NAME":"e2term-alpha-568687b664-nlb94","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"master","SYSTEM_NAME":"SEP"}

,"msg":"Got Keep Alive Request send : {\"address\": \"10.97.68.34:38000\",\"fqdn\": \"10.97.68.34\",\"pod_name\": \"e2term\"}"}_

_

{"ts":1634559565001,"crit":"DEBUG","id":"E2Terminator","mdc":

{"PID":"139686259332864","POD_NAME":"e2term-alpha-568687b664-nlb94","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"master","SYSTEM_NAME":"SEP"}

,"msg":"EXIT OK from receiveXappMessages"}_

_

{"ts":1634559565001,"crit":"DEBUG","id":"E2Terminator","mdc":

{"PID":"139686259332864","POD_NAME":"e2term-alpha-568687b664-nlb94","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"master","SYSTEM_NAME":"SEP"}

,"msg":"message handling is 46 seconds 43931057 nanoseconds"}_

_

{"ts":1634559565001,"crit":"DEBUG","id":"E2Terminator","mdc":

{"PID":"139686259332864","POD_NAME":"e2term-alpha-568687b664-nlb94","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"master","SYSTEM_NAME":"SEP"}

,"msg":"Start EPOLL Wait. Timeout = -1"}_

_

{"ts":1634559565032,"crit":"INFO","id":"E2Terminator","mdc":

{"PID":"25","POD_NAME":"e2term-alpha-568687b664-nlb94","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"master","SYSTEM_NAME":"SEP"}

,"msg":"Got a message from some application, stop sending E2_TERM_INIT"}_

 

1. E2M, runs the timer to receive the E2_TERM_KEEP_ALIVE_RESP, and over the time expiry E2M deletes the E2T entries from DBASS.

{"crit":"WARN","ts":1634559565001,"id":"E2Manager","msg":"#E2TKeepAliveWorker.E2TKeepAliveExpired - e2t address: 10.97.68.34:38000 time expired, shutdown e2 instance","mdc":{"time":"2021-10-18 12:19:25.001"}}

{"crit":"INFO","ts":1634559565001,"id":"E2Manager","msg":"#E2TShutdownManager.Shutdown - E2T 10.97.68.34:38000 is Dead, RIP","mdc":{"time":"2021-10-18 12:19:25.001"}}

{"crit":"INFO","ts":1634559565001,"id":"E2Manager","msg":"#RnibDataService.SaveE2TInstance - E2T instance address: 10.97.68.34:38000, podName: e2term, state: TO_BE_DELETED, associated RANs count: 0, keep Alive ts: 1634559085001985608","mdc":{"time":"2021-10-18 12:19:25.001"}}

{"crit":"INFO","ts":1634559565001,"id":"E2Manager","msg":"#E2TAssociationManager.RemoveE2tInstance -  Removing E2T 10.97.68.34:38000 and dessociating its associated RANs.","mdc":{"time":"2021-10-18 12:19:25.001"}}

 

1. Here E2T has send the E2_TERM_KEEP_ALIVE_REQ successfully over RMR but E2M has not received the same over RMR.

 __ 

Problem : It seems E2_TERM_KEEP_ALIVE_REQ  is getting drop somewhere in RMR, so further need to be debugged from RMR side.

 

Additional observation : Sometime I have noticed the below error as well from RMR:

            _

{"ts":1634555605001,"crit":"ERROR","id":"E2Terminator","mdc":

{"PID":" ","POD_NAME":"e2term-alpha-7845d68c4-kzdhl","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"master","SYSTEM_NAME":"SEP"}

,"msg":"Failed to send E2_TERM_KEEP_ALIVE_RESP, on RMR state = 2 ( RMR_ERR_NOENDPT - send//call could not find an endpoint based on msg type)"}_