All BEA Publisher Portlets Down

This past weekend we had our servers rebooted after a patch maintenance and after that when the services were turned on, we noticed something peculiar. The portal was up and running but none of the Publisher portlets were working (HTTP 404 Errors) and neither was the Publisher diagnostic page opening up for us. Hoping something was wrong with publisher, we tried bouncing the Publisher server and then proceeded to do of all the servers in our portal environment…No go….Nothing…the portal page opened but the Publisher portlets were dead as a dodo :)
We opened up all the logs and ran the diagnostics…et all….but were not able to figure it out for quiet sometime….
The Portal Diagnostics bat file showed the following message:
===================================================
PORTAL_HOME=C:\bea\alui\ptportal\10.3.0Expected PORTAL_HOME=C:\bea\alui\ptportal\10.3.0Running Diagnostic Tool...OpenLog: verbosity level = 2 Diagnostic system initialization OK. Portal Startup begins: 15 total tasks.OpenLog: Registered application name: ConfigInitialization (local machine only)OpenLog: Registered application name: portal.sbymas096.svcwciadmin (local machine only)Base library name : openkernelsearch_4-3j
OpenLog native: Registered application name portal.sbymas096.svcwciadmin (localmachine only)Task #1 (InitPortalObjects) END (OK)Task #2 (CheckDevKit) END (OK)Task #3 (InitializePaths) END (OK)Task #4 (InitializeApplication) END (OK)Task #5 (InitializeAppWarmUpHelper) END (OK)Task #6 (InitializeImageServerHelper) END (OK)Task #7 (InitializeConfigHelper) END (OK)Task #8 (InitializeResourceManager) END (OK)Task #9 (LoadVersionHelper) END (OK) FATAL: Task #10 (InitializeActivitySpaces) step 2 failed to complete. InitializeManagers: Error initializing TagLibManager. LoadActivitySpace problem: managers will not load. RECOMMENDATION: Check that version.xml is located in the correct directory and is not corrupt.com.plumtree.openfoundation.util.XPException: Fatal error loading lib file taglibJoinCommunityPageEditor: Unable to locate Tag Library meta data. at com.plumtree.portaluiinfrastructure.tags.manager.TagLibManager.InternalLoadLibFile(TagLibManager.java:460) at com.plumtree.portaluiinfrastructure.tags.manager.TagLibManager.InternalLoadClasses(TagLibManager.java:400) at com.plumtree.portaluiinfrastructure.tags.manager.TagLibManager.LoadClasses(TagLibManager.java:327) at com.plumtree.portaluiinfrastructure.application.PTAppWarmUpHelper.InitializeManagers(PTAppWarmUpHelper.java:431) at com.plumtree.uiinfrastructure.application.startup.InitializeActivitySpaces.executeStepTwo(InitializeActivitySpaces.java:165) at com.plumtree.uiinfrastructure.application.startup.InitializeActivitySpaces.executeStep(InitializeActivitySpaces.java:108) at com.plumtree.uiinfrastructure.application.AppWarmUp.LoopStartupTasks(AppWarmUp.java:995) at com.plumtree.uiinfrastructure.application.AppWarmUp.RunStartupTasks(AppWarmUp.java:893) at com.plumtree.uiinfrastructure.application.AppWarmUp.Init(AppWarmUp.java:200) at com.plumtree.uiinfrastructure.diagnostic.DiagnosticRunner.executeAllTestsWithoutPause(DiagnosticRunner.java:127) at com.plumtree.uiinfrastructure.diagnostic.DiagnosticRunner.executeAllTests(DiagnosticRunner.java:138) at com.plumtree.diagtool.diagnostic_net.main(diagnostic_net.java:40)Caused by: com.plumtree.openfoundation.util.XPException: Fatal error loading lib file taglibJoinCommunityPageEditor: Unable to locate Tag Library meta data. at com.plumtree.portaluiinfrastructure.tags.manager.TagLibManager.InternalLoadLibFile(TagLibManager.java:434) ... 11 more
Task 'InitializeActivitySpaces' Failed on Startup Task #10 (InitializeActivitySpaces) Step #2--------------------------------------------------------------------------------
error: LoadActivitySpace problem: managers will not load.platform: Windows (Java)failure: FATAL: LoadActivitySpace problem: managers will not load.recommendation: RECOMMENDATION: Check that version.xml is located in the correct directory and is not corrupt.exception: XPException Fatal error loading lib file taglibJoinCommunityPageEditor: Unable to locate Tag Library meta data.
message: Fatal error loading lib file taglibJoinCommunityPageEditor: Unable to locate Tag Library meta data.secondary exception: at com.plumtree.portaluiinfrastructure.tags.manager.TagLibManager.InternalLoadLibFile(TagLibManager.java:460)--------------------------------------------------------------------------------
FAILED TO INITIALIZE SYSTEM. YOU CANNOT PROGRESS BEYOND THIS POINT. ======= Diagnostic Run Complete ======= Startup completion percentage: 62% (31 steps out of 50 complete) Diagnostic Test execution complete.Press key to exit.
==================================================================

Now if that was not confusing, the Publisher Container logs were even more confusing. However there is always light at the end of a tunnel :) Lets check what the container logs had in them
==================================================================

2011-09-26 02:05:38,032 83937 ERROR [WebModule] (main:jboss.web.deployment:war=ROOT.war,id=-89327743 WebModule) Starting failed jboss.web.deployment:war=ROOT.war,id=-89327743
org.jboss.deployment.DeploymentException: Error during deploy; - nested throwable: (javax.naming.NameNotFoundException: comp not bound)
at org.jboss.web.AbstractWebDeployer.start(AbstractWebDeployer.java:329)
at org.jboss.web.WebModule.startModule(WebModule.java:62)
at org.jboss.web.WebModule.startService(WebModule.java:40)
at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:271)
at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:221)
……………………….
2011-09-26 02:05:04,220 50125 WARN [ServiceController] (main:) Problem starting service jboss:service=Naming javax.management.MBeanException: java.rmi.server.ExportException: Port already in use: 1098; nested exception is: java.net.BindException: Address already in use: JVM_Bind
at org.jboss.mx.server.ReflectedDispatcher.handleInvocationExceptions(ReflectedDispatcher.java:91)
at org.jboss.mx.server.ReflectedDispatcher.dispatch(ReflectedDispatcher.java:64)
==================================================================

The clincher was here. The port 1098 was already in use. Although at the time of first seeing this, I had overlooked this as it came under a warning message. But that’s how BEA Publisher works…lolz…the most important information comes in a warning, while the error message shows something which is too difficult to comprehend…. :)
Anyways…we then found out that BEA Publisher uses the concept of ephemeral ports. The port 1098 is an ephemeral port which BEA Publisher needs for some time when the service is restarted. However BEA Publisher does not reserve this port 1098 exclusively for itself. As a result it might happen that port 1098 is gobbled up by some other application running on your machine……

Solution:
We used NETSTAT command to find out if port 1098 is free or not. As it turned out, port 1098 was indeed not free and being used by some other application running on the server. Once we found it was in use, we found out the PID associated with the process and using the Task Manger killed the process corresponding to the PID. Thereafter a restart of BEA ALI Publisher service resulted in Publisher starting up correctly.

Stubborn Publisher Cache

So you have been working on BEA Publisher and you face that issue, which I think everyone faces with Publisher……the dreaded Caching of content. It is evil my friends. Evil because, Publisher has the habit of caching content and displaying this content. While this is good on some occasions because it helps in faster page load, however, when you are working on some portlet and want to immediately see the effects of your change, this same boon becomes your bane.
Recently I was working on this and found out some tips which can provide relief from the caching content of Publisher.

This is a 2 step process:

Step1: The easiest way is turn off caching in the Publisher WebService.

In the HTTP Configuration page of your Publisher Web Service, look for the Gateway Caching section present at the top of the page. Make both the Minimum Cache Time and Maximum Cache Time as ZERO seconds. Don’t forget to un-check the check box for Suppress errors where possible, so that it is blank.
Click on Finish and your Publisher portlets will no longer show cached content. However do take note, that you should keep a record of you Min and Max Cache Times and restore them once your work is completed, otherwise you are gonna hear a lot of complaining that portlets are taking a long long time to load in the page.

Step 2:
While the Step 1 above works every time, there is one other thing which you can do to make absolutely certain that no cached content remains anywhere. This step needs access to the server on which BEA Publisher 6.5 is installed.
The steps to perform are as follows:
~ Log into the server where BEA Publisher is installed
~ Stop the Publisher service running on that system
~ Search for the folder named localhost in the ptcs installation directory. Normally it will be found like D://BEA/alui/ptcs/6.5/container/work/jboss.web
~ Delete the whole local host folder
~ Restart the Publisher web service once again (The localhost folder will be re-created once again)