- Whenever E2T pod Stops and Starts (Restart), E2T keep sending the E2_TERM_INIT to E2M until E2T receives the E2_TERM_KEEP_ALIVE_REQ from E2M. which is happening as expected.
_Line 23: {"ts":1634559366146,"crit":"INFO","id":"E2Terminator","mdc":
{"PID":"25","POD_NAME":"e2term-alpha-568687b664-nlb94","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"master","SYSTEM_NAME":"SEP"},"msg":"E2_TERM_INIT successfully sent "}_
_Line 64: {"ts":1634559466240,"crit":"INFO","id":"E2Terminator","mdc":
{"PID":"25","POD_NAME":"e2term-alpha-568687b664-nlb94","CONTAINER_NAME":"container-ricplt-e2term","SERVICE_NAME":"RIC_E2_TERM","HOST_NAME":"master","SYSTEM_NAME":"SEP"},"msg":"E2_TERM_INIT successfully sent "}_
- The moment E2T receives the E2_TERM_KEEP_ALIVE_REQ from E2M, E2T will response to the E2M with E2_TERM_KEEP_ALIVE_RESP and stops sending the E2_TERM_INIT. This is also working as expected
_
{"ts":1634559565001,"crit":"DEBUG","id":"E2Terminator","mdc":
,"msg":"Got Keep Alive Request send : {\"address\": \"10.97.68.34:38000\",\"fqdn\": \"10.97.68.34\",\"pod_name\": \"e2term\"}"}_
_
{"ts":1634559565001,"crit":"DEBUG","id":"E2Terminator","mdc":
,"msg":"EXIT OK from receiveXappMessages"}_
_
{"ts":1634559565001,"crit":"DEBUG","id":"E2Terminator","mdc":
,"msg":"message handling is 46 seconds 43931057 nanoseconds"}_
_
{"ts":1634559565001,"crit":"DEBUG","id":"E2Terminator","mdc":
,"msg":"Start EPOLL Wait. Timeout = -1"}_
_
{"ts":1634559565032,"crit":"INFO","id":"E2Terminator","mdc":
,"msg":"Got a message from some application, stop sending E2_TERM_INIT"}_
- E2M, runs the timer to receive the E2_TERM_KEEP_ALIVE_RESP, and over the time expiry E2M deletes the E2T entries from DBASS.
{"crit":"WARN","ts":1634559565001,"id":"E2Manager","msg":"#E2TKeepAliveWorker.E2TKeepAliveExpired - e2t address: 10.97.68.34:38000 time expired, shutdown e2 instance","mdc":{"time":"2021-10-18 12:19:25.001"}}
{"crit":"INFO","ts":1634559565001,"id":"E2Manager","msg":"#E2TShutdownManager.Shutdown - E2T 10.97.68.34:38000 is Dead, RIP","mdc":{"time":"2021-10-18 12:19:25.001"}}
{"crit":"INFO","ts":1634559565001,"id":"E2Manager","msg":"#RnibDataService.SaveE2TInstance - E2T instance address: 10.97.68.34:38000, podName: e2term, state: TO_BE_DELETED, associated RANs count: 0, keep Alive ts: 1634559085001985608","mdc":{"time":"2021-10-18 12:19:25.001"}}
{"crit":"INFO","ts":1634559565001,"id":"E2Manager","msg":"#E2TAssociationManager.RemoveE2tInstance - Removing E2T 10.97.68.34:38000 and dessociating its associated RANs.","mdc":{"time":"2021-10-18 12:19:25.001"}}
- Here E2T has send the E2_TERM_KEEP_ALIVE_REQ successfully over RMR but E2M has not received the same over RMR.
__
Problem : It seems E2_TERM_KEEP_ALIVE_REQ is getting drop somewhere in RMR, so further need to be debugged from RMR side.
Additional observation : Sometime I have noticed the below error as well from RMR:
_
{"ts":1634555605001,"crit":"ERROR","id":"E2Terminator","mdc":
,"msg":"Failed to send E2_TERM_KEEP_ALIVE_RESP, on RMR state = 2 ( RMR_ERR_NOENDPT - send//call could not find an endpoint based on msg type)"}_