All the NEs from the Manager for 6k that are not dual managed disappeared from the OneControl NE List tab. The same NEs have the Completed state in the Manager for 6k NE List tab.
All the NEs from the Manager for 6k that are dual managed have the suspect state in the OneControl NE List tab. The same NEs have the Completed state in the Manager for 6k NE List tab.
The EMS servers tab under the Manage EMS Configuration tab shows no Role (--) and no Connectivity state (--) against the line for the co-resident Manager for 6k.
Errors similar to the following one is seen in the /opt/ciena/CIENAone/OneControl/logs/OneControl-<hostname>.log log file:
###<Nov 22, 2018 1:44:03.777 PM EST> <Error> <com.ciena.rm.ne.messaging.NeManagerStateChangeMdb> <onxl4443.ott.ciena.com> <OneControl-onxl4443> <[ACTIVE] ExecuteThread: '15' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <BEA1-371D5E2BFE778E8DCB27> <> <1542912243777> <BEA-000000> <failed to process EMS client state change event 'ObjectMessage[ID:<310746.1542912243774.0>,NeManagerStateChangeEvent{serverId='onxl4443.ott.ciena.com:1099', action=CONN2FAILURE, connDetail=LDAP_GATEWAY_DOWN, grState=NONE, userData=NeManagerEntity{id=0, manager type=OME, ems type=OMEA, server name=onxl4443.ott.ciena.com:1099, comm state=FAILURE, connection detail=CONNECTING, host=onxl4443.ott.ciena.com, ip=10.180.70.70, port=1099, cliCommProtocol=null, snmp credential=CredentialEntity{id=0, userId=rjilsec, authHmac=null, authP=B53E83047B48A7E8749166D9058CE3D7, privacyCipher=null, privP=B53E83047B48A7E8749166D9058CE3D7}, connectivity=CONNECTING, role=MASTER, mode=PRIMARY, preference=true, peer=onxl4405.ott.ciena.com:1099}}]', Error: java.lang.NullPointerException: Not found NeManagerEntity with name: onxl4443.ott.ciena.com:1099 at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:208) at com.ciena.rm.ne.messaging.NeManagerStateChangeMdb.getNeManagerEntityFromDb(NeManagerStateChangeMdb.java:379) at com.ciena.rm.ne.messaging.NeManagerStateChangeMdb.processMsg(NeManagerStateChangeMdb.java:145) at com.ciena.rm.ne.messaging.NeManagerStateChangeMdb.onMessage(NeManagerStateChangeMdb.java:75) at weblogic.ejb.container.internal.MDListener.execute(MDListener.java:585) at weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:488) at weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:385) at weblogic.jms.client.JMSSession.onMessage(JMSSession.java:4659) at weblogic.jms.client.JMSSession.execute(JMSSession.java:4345) at weblogic.jms.client.JMSSession.executeMessage(JMSSession.java:3821) at weblogic.jms.client.JMSSession.access$000(JMSSession.java:115) at weblogic.jms.client.JMSSession$UseForRunnable.run(JMSSession.java:5170) at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:545) at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256) at weblogic.work.ExecuteThread.run(ExecuteThread.java:221) >
OneControl
Manager for 6k
Procedure to fix the Manager for 6k EMS server entry: Note: You can safely delete the Manager for 6k EMS server under the EMS Configuration tab in OneControl GUI. The EMS server deletion does not impact the other OneControl server in the GR/HA pair. No information will be lost if you re-add the EMS server.
STEP 1) Launch OneControl GUI
STEP 2) Select Manage EMS Configuration under Administration menu
STEP 3) Select the EMS Servers tab
STEP 4) Write down the information under the Hostname/IP column for the co-resident Manager for 6k server
STEP 5) Delete the Manager for 6k pair that does not show Connected against the primary nor the backup Manager for 6k servers. Some times you need to highlight the Resilient pair line in the table to delete the Manager for 6k pair and other times you need to select each Manager for 6k server to be able to delete them.
STEP 6) Re-add the Manager for 6k EMS server that is co-resident on this OneControl server using the long hostname of the OneControl server. Wait 2 minutes to see if OneControl is able to connect to the Manager for 6k. Go to next step if it does not connect.
STEP 7) Delete the Manager for 6k pair that you just added.
STEP 8) Re-add the Manager for 6k EMS server that is co-resident on this OneControl server using the short hostnameof the OneControl server. Wait 2 minutes to see if OneControl is able to connect to the Manager for 6k. Go to next step if it does not connect.
STEP 9) Delete the Manager for 6k pair that you just added.
STEP 10) Re-add the Manager for 6k EMS server that is co-resident on this OneControl server using the IP addressof the OneControl server. Wait 2 minutes to see if OneControl is able to connect to the Manager for 6k.
