Certificates and VMware don’t have a great history and there are a lot of posts out there centered around people’s struggles with vCenter, Lookup Service or Web Client Certificate management. I’ve recently had a little fun with a revoked vCenter certificate (thanks Symantec) that required replacement. Without going into the details of the pain I went through to successfully get the certificate updated and working with vCenter and the Web Client, when I did eventually get things in working order with the new publicly signed certificate I logged back into the Web Client and saw that I had no NSX Managers listed in the Web Client.
I’ve blogged before about how to deal with that particular error as it relates to user permissions, but as nothing had changed from a permissions point of view this was surly due to the certificate changes on the vCenter. Logging into the NSX Manager and going to the Manage Tab and NSM Management Service the vCenter Server Status was listed as Disconnected.
I also found corresponding errors in the Manager Logs as shown below.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
2015-11-25 11:38:21.447 GMT INFO ViInventoryConnKeepAliveThread ViInventory$ViInventoryConnKeepAliveThread:6571 - Connection Handler is either null or not connected 2015-11-25 11:38:22.439 GMT INFO systemEventsPool-1 DefaultVcConnection:276 - Disconnect default vc connection 2015-11-25 11:38:22.439 GMT INFO systemEventsPool-1 VSMAgentStateUpdater$VcConnectionLifecycleListener:231 - Detected VC disconnect 2015-11-25 11:38:22.919 GMT INFO ViInventoryThread ViInventory:548 - Inventory cannot connect to VC because:null 2015-11-25 11:38:24.367 GMT ERROR DefaultVcConnectionKeepaliveThread SoapBindingImpl:134 - SOAP fault javax.xml.ws.soap.SOAPFaultException: Invalid credentials at com.sun.xml.internal.ws.fault.SOAP11Fault.getProtocolException(Unknown Source) at com.sun.xml.internal.ws.fault.SOAPFaultBuilder.createException(Unknown Source) at com.sun.xml.internal.ws.client.dispatch.DispatchImpl.doInvoke(Unknown Source) at com.sun.xml.internal.ws.client.dispatch.DispatchImpl.invoke(Unknown Source) at com.vmware.vim.sso.client.impl.SoapBindingImpl.sendMessage(SoapBindingImpl.java:131) at com.vmware.vim.sso.client.impl.SoapBindingImpl.sendMessage(SoapBindingImpl.java:82) at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.sendRequest(SecurityTokenServiceImpl.java:677) at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.executeRoundtrip(SecurityTokenServiceImpl.java:611) at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl.acquireToken(SecurityTokenServiceImpl.java:120) at com.vmware.vshield.vsm.vcserver.VcConnection.getSamlToken(VcConnection.java:389) at com.vmware.vshield.vsm.vcserver.VcConnection.defaultLogin(VcConnection.java:201) at com.vmware.vshield.vsm.vcserver.VcConnection.login(VcConnection.java:186) at com.vmware.vshield.vsm.vcserver.VcConnection.login(VcConnection.java:174) at com.vmware.vshield.vsm.vcserver.DefaultVcConnection.checkConnect(DefaultVcConnection.java:228) at com.vmware.vshield.vsm.vcserver.DefaultVcConnection.getVcConnection(DefaultVcConnection.java:187) at com.vmware.vshield.vsm.vcserver.DefaultVcConnectionKeepaliveThread.getVcConnection(DefaultVcConnectionKeepaliveThread.java:121) at com.vmware.vshield.vsm.vcserver.DefaultVcConnectionKeepaliveThread.run(DefaultVcConnectionKeepaliveThread.java:73) 2015-11-25 11:38:24.368 GMT INFO DefaultVcConnectionKeepaliveThread SecurityTokenServiceImpl$RequestResponseProcessor:742 - Provided credentials are not valid. 2015-11-25 11:38:26.181 GMT ERROR NVPStatusCheck ControllerServiceImpl:1658 - vsm UUID not match for controller 172.17.0.202: 421E9E74-5C5F-27C0-1E93-3394D3AC56A0 2015-11-25 11:38:26.181 GMT ERROR NVPStatusCheck ControllerServiceImpl:1658 - vsm UUID not match for controller 172.17.0.201: 421E9E74-5C5F-27C0-1E93-3394D3AC56A0 2015-11-25 11:38:26.181 GMT ERROR NVPStatusCheck ControllerServiceImpl:1658 - vsm UUID not match for controller 172.17.0.200: 421E9E74-5C5F-27C0-1E93-3394D3AC56A0 2015-11-25 11:38:27.478 GMT INFO DefaultVcConnectionKeepaliveThread DefaultVcConnection:276 - Disconnect default vc connection 2015-11-25 11:38:27.479 GMT INFO DefaultVcConnectionKeepaliveThread VSMAgentStateUpdater$VcConnectionLifecycleListener:231 - Detected VC disconnect 2015-11-25 11:38:27.479 GMT INFO DefaultVcConnectionKeepaliveThread DefaultVcConnectionKeepaliveThread:124 - Could not get VC Connection:com.vmware.vshield.vsm.vcserver.VcConnectionNotAvailableException: core-services:500:vCenter Connection is not available.:com.vmware.vim.binding.vim.fault.InvalidLogin: |
The reason for this happening is the NSX Manager trusted the previous certificate and needs to be reconnected so that the new certificate can be trusted and accepted.
Once that’s been done you should have a green light and the NSX Manager will resync up with the vCenter Inventory and all operations will be back to normal…an easy fix to a logical issue!