IBM WebSphere Application Server Fixes

Check what ports your server is using show up under the virtual host that the app you are looking for is mapped to.
For ISIM check Servers -> All Servers -> isimServer1 - expand "ports"
Then make sure
Virtual Hosts > default_host > Host Aliases
shows that port. If not add and restart the server. IF it does, check that ISIM app is mapped to default_host

Snoop application is inaccessible


Make sure it is mapped to you servers (by default it's mapped to server1 which may have been removed). Also make sure that everyone can access it, otherwise login using your WAS credentials first.

IHS WebServer Plugin is not updated

  1. In the WAS admin console expand servers, server types.
  2. Click on Web servers, then webserver1
  3. Make sure the “Web server installation location is correct”
  4. Verify the plugin location corresponds to the one at the end of ...\HTTPServer\conf\httpd.conf e.g. ...\HTTPServer\Plugins\config\webserver1\plugin-cfg.xml
  5. If it is broken modify ...\AppServer\profiles\AppSrv01\config\cells\HOSTNode01Cell\nodes\HOSTNode01\servers\webserver1\server.xml and delete from the same folder plugin-cfg.xml
  6. Use WAS admin console to regenerate and propagate the plugin. Make sure to copy (update) the kdb files (key stores) as well using the "copy to Webserver" button on the plug-in properties page.
  7. Restart IHS if needed

Kerberos/SPNEGO integration is not working

Very few understand inner workings of Kerberos, so troubleshooting it is very tricky. The first step is to make sure your WAS -> AD link is properly set-up.

  • Enable tracing on com.ibm.ws.security.spnego.* and check your ..\profiles\AppSrv01\logs\server1\trace.log when WAS starts-up. When the first SPNEGO request is made you should see a successful init of the SPNEGO module.
  • Once that is squared away make sure the AD user you use for the integration is set-up properly. It should have userPrinicipalName starting with your host name and a servicePrincipalName identical to it. If you have an alias add servicePrinicpalName value to the AD spnego user to say the alias server name too, so that it has both, the alias and the real name, both in the required format: HTTP/servername.domain.dom
  • If the user is correct and you are dealing with aliases try joining the kerberos keytabs into a single keytab for both names. First, create the second keytab with the following command (probably the same you used for the first keytab):

ktpass -out secondSPNEGO.keytab -princ HTTP/alias.domain.dom@DOMAIN.DOM -mapuser spnegouser@domain.dom -pass * -ptype KRB5_NT_PRINCIPAL

Then use ktutil from the krb5-user package on Ubuntu. Run it, type rkt first and then rkt second keytab, then wkt to write to a combined keytab. Use that keytab in WAS configuration.

  • If that does not work make sure that the alias and the real host name are both A records - run

dig aliasname

If it shows up as a CNAME try to change the DNS to an A record. You might have to add another network interface for that to happen and configure it to be used for certain traffic. Alternatively you could put a front-end reverse proxy or a load balancer that would resolve to that A name.

Analyzing IBM WebSphere Java heap dumps

Depending on the mode of failure, jvm hard exception, like a seg fault/a hardware exception or a jvm soft exception like a out of memory issue, you could get either a "core.*.dmp" (system dump) file or a portable heap dump "*.phd"
The best tool to use for the heap dump analysis is Eclipse's Memory Analyzer. Together with the IBM Diagnostic Tool Framework for Java it can open most of the possible dumps, including the Sun's HPROF dump. To open the system dump you first have to pre-pack it by running jextract that comes with IBM's JVM (for example in "F:\Program Files\IBM\WebSphere\AppServer\java\jre\bin\jextract.exe"),
jextract -J-Xcompressedrefs ''<name of .dmp>''

jextract basically collects all the loaded libraries, generates required header information and zips it all up. When running eclipse give it enough memory to be able to consume the dump:
eclipse -vmargs -Xmx2g -XX:-UseGCOverheadLimit

As an alternative portable heap dumps can be analyzed in IBM's Heap Analyzer. However it does not seem to support system dumps (core*dmp). You got to give it enough memory too:
java -Xmx3000m -jar ha408.jar ''<phd file name>''

You will also get a javacore.*.txt file with a list of threads that you can analyze with the Java Core Analyzer. Note though that looking at the txt file directly shows a bit more useful information, like the name of the thread that was active when the failure occurred.

The Snap.*.trc files contain the tracepoint data held in the trace buffers (calls as they are made through time). You have to convert it to the text format by running IBM's trace formatter:
java com.ibm.jvm.format.TraceFormat ''<input file>''

Note that the time might be in a different time zone. Assume the end of the tracepoint data corresponds to the timestamp on the snap file

WAS does not start on windows with WSLoginFailedException

com.ibm.websphere.security.auth.WSLoginFailedException: Authentication failed for user itimservice with the following error message Logon failure: unknown user name or bad password.
If the password for the account was changed after WAS was installed then change it in the properties/soap.client.props file and in config/cells/[nodename]/security.xml

  • Double-check that the WAS windows service is using the correct password for your windows user that the service is configured to run as.
  • Next make sure that the user is a local admin on the box or at least has rights to start a service on this box.
  • Check that the account is not locked or expired.

WAS can not start

Disable WAS Security to test:

wsadmin -conttype NONE
securityoff


@Troubleshooting @WebSphere