All BEA Publisher Portlets Down

This past weekend we had our servers rebooted after a patch maintenance and after that when the services were turned on, we noticed something peculiar. The portal was up and running but none of the Publisher portlets were working (HTTP 404 Errors) and neither was the Publisher diagnostic page opening up for us. Hoping something was wrong with publisher, we tried bouncing the Publisher server and then proceeded to do of all the servers in our portal environment…No go….Nothing…the portal page opened but the Publisher portlets were dead as a dodo :)
We opened up all the logs and ran the diagnostics…et all….but were not able to figure it out for quiet sometime….
The Portal Diagnostics bat file showed the following message:
===================================================
PORTAL_HOME=C:\bea\alui\ptportal\10.3.0Expected PORTAL_HOME=C:\bea\alui\ptportal\10.3.0Running Diagnostic Tool...OpenLog: verbosity level = 2 Diagnostic system initialization OK. Portal Startup begins: 15 total tasks.OpenLog: Registered application name: ConfigInitialization (local machine only)OpenLog: Registered application name: portal.sbymas096.svcwciadmin (local machine only)Base library name : openkernelsearch_4-3j
OpenLog native: Registered application name portal.sbymas096.svcwciadmin (localmachine only)Task #1 (InitPortalObjects) END (OK)Task #2 (CheckDevKit) END (OK)Task #3 (InitializePaths) END (OK)Task #4 (InitializeApplication) END (OK)Task #5 (InitializeAppWarmUpHelper) END (OK)Task #6 (InitializeImageServerHelper) END (OK)Task #7 (InitializeConfigHelper) END (OK)Task #8 (InitializeResourceManager) END (OK)Task #9 (LoadVersionHelper) END (OK) FATAL: Task #10 (InitializeActivitySpaces) step 2 failed to complete. InitializeManagers: Error initializing TagLibManager. LoadActivitySpace problem: managers will not load. RECOMMENDATION: Check that version.xml is located in the correct directory and is not corrupt.com.plumtree.openfoundation.util.XPException: Fatal error loading lib file taglibJoinCommunityPageEditor: Unable to locate Tag Library meta data. at com.plumtree.portaluiinfrastructure.tags.manager.TagLibManager.InternalLoadLibFile(TagLibManager.java:460) at com.plumtree.portaluiinfrastructure.tags.manager.TagLibManager.InternalLoadClasses(TagLibManager.java:400) at com.plumtree.portaluiinfrastructure.tags.manager.TagLibManager.LoadClasses(TagLibManager.java:327) at com.plumtree.portaluiinfrastructure.application.PTAppWarmUpHelper.InitializeManagers(PTAppWarmUpHelper.java:431) at com.plumtree.uiinfrastructure.application.startup.InitializeActivitySpaces.executeStepTwo(InitializeActivitySpaces.java:165) at com.plumtree.uiinfrastructure.application.startup.InitializeActivitySpaces.executeStep(InitializeActivitySpaces.java:108) at com.plumtree.uiinfrastructure.application.AppWarmUp.LoopStartupTasks(AppWarmUp.java:995) at com.plumtree.uiinfrastructure.application.AppWarmUp.RunStartupTasks(AppWarmUp.java:893) at com.plumtree.uiinfrastructure.application.AppWarmUp.Init(AppWarmUp.java:200) at com.plumtree.uiinfrastructure.diagnostic.DiagnosticRunner.executeAllTestsWithoutPause(DiagnosticRunner.java:127) at com.plumtree.uiinfrastructure.diagnostic.DiagnosticRunner.executeAllTests(DiagnosticRunner.java:138) at com.plumtree.diagtool.diagnostic_net.main(diagnostic_net.java:40)Caused by: com.plumtree.openfoundation.util.XPException: Fatal error loading lib file taglibJoinCommunityPageEditor: Unable to locate Tag Library meta data. at com.plumtree.portaluiinfrastructure.tags.manager.TagLibManager.InternalLoadLibFile(TagLibManager.java:434) ... 11 more
Task 'InitializeActivitySpaces' Failed on Startup Task #10 (InitializeActivitySpaces) Step #2--------------------------------------------------------------------------------
error: LoadActivitySpace problem: managers will not load.platform: Windows (Java)failure: FATAL: LoadActivitySpace problem: managers will not load.recommendation: RECOMMENDATION: Check that version.xml is located in the correct directory and is not corrupt.exception: XPException Fatal error loading lib file taglibJoinCommunityPageEditor: Unable to locate Tag Library meta data.
message: Fatal error loading lib file taglibJoinCommunityPageEditor: Unable to locate Tag Library meta data.secondary exception: at com.plumtree.portaluiinfrastructure.tags.manager.TagLibManager.InternalLoadLibFile(TagLibManager.java:460)--------------------------------------------------------------------------------
FAILED TO INITIALIZE SYSTEM. YOU CANNOT PROGRESS BEYOND THIS POINT. ======= Diagnostic Run Complete ======= Startup completion percentage: 62% (31 steps out of 50 complete) Diagnostic Test execution complete.Press key to exit.
==================================================================

Now if that was not confusing, the Publisher Container logs were even more confusing. However there is always light at the end of a tunnel :) Lets check what the container logs had in them
==================================================================

2011-09-26 02:05:38,032 83937 ERROR [WebModule] (main:jboss.web.deployment:war=ROOT.war,id=-89327743 WebModule) Starting failed jboss.web.deployment:war=ROOT.war,id=-89327743
org.jboss.deployment.DeploymentException: Error during deploy; - nested throwable: (javax.naming.NameNotFoundException: comp not bound)
at org.jboss.web.AbstractWebDeployer.start(AbstractWebDeployer.java:329)
at org.jboss.web.WebModule.startModule(WebModule.java:62)
at org.jboss.web.WebModule.startService(WebModule.java:40)
at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:271)
at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:221)
……………………….
2011-09-26 02:05:04,220 50125 WARN [ServiceController] (main:) Problem starting service jboss:service=Naming javax.management.MBeanException: java.rmi.server.ExportException: Port already in use: 1098; nested exception is: java.net.BindException: Address already in use: JVM_Bind
at org.jboss.mx.server.ReflectedDispatcher.handleInvocationExceptions(ReflectedDispatcher.java:91)
at org.jboss.mx.server.ReflectedDispatcher.dispatch(ReflectedDispatcher.java:64)
==================================================================

The clincher was here. The port 1098 was already in use. Although at the time of first seeing this, I had overlooked this as it came under a warning message. But that’s how BEA Publisher works…lolz…the most important information comes in a warning, while the error message shows something which is too difficult to comprehend…. :)
Anyways…we then found out that BEA Publisher uses the concept of ephemeral ports. The port 1098 is an ephemeral port which BEA Publisher needs for some time when the service is restarted. However BEA Publisher does not reserve this port 1098 exclusively for itself. As a result it might happen that port 1098 is gobbled up by some other application running on your machine……

Solution:
We used NETSTAT command to find out if port 1098 is free or not. As it turned out, port 1098 was indeed not free and being used by some other application running on the server. Once we found it was in use, we found out the PID associated with the process and using the Task Manger killed the process corresponding to the PID. Thereafter a restart of BEA ALI Publisher service resulted in Publisher starting up correctly.