IBM Security Identity Manager Fixes

ISIM 7 appliance hungs with "press any key to continue"

Enable COM port in your virtual machine configuration

Where are trace logs on the appliance?

/var/ibm/tivoli/common/CTGIM/logs/

Validating the ISIM install

Examine

/opt/IBM/isim/install_logs/itim_install_activity.log
/opt/IBM/isim/install_logs/runConfigFirstTime.stdout
/opt/IBM/isim/install_logs/dbConfig.stdout
/opt/IBM/isim/install_logs/ldapConfig.stdout

Second ISIM node show encryption errors

Make sure all files in /data/keytabs are the same. On both servers:
md5sum /opt/IBM/isim/data/keytabs/*
If not you need to synchronize ISIM cryptography:

1. Stop second node
2. Make a backup of files in /opt/IBM/isim/data/keytabs/ and copy them from node1
3. Also copy
4. enRole.properties. Edit it to correct enrole.appServer.name
5. enRoleLDAPConnecton.properties
6. enRoleDatabase.properties

SSO from SAM is not working

You can verify that the Basic headers are coming through by checking the snoop app: https://samserver/isimdev/snoop. Look for BA under the Request headers.
If snoop is not coming up make sure you have it mapped to the target server (in WAS console). You might also want to allow everybody to access it, not just logged in WAS users.

Middleware configuration tool does not start in a graphical mode

Make sure you install all the GUI prerequisite libraries in a 32 bit format
yum install libXmu.i686 libXp.i686 libXft.i686 libXtst.i686
If it starts but fonts are blank/corrupted when you are forwarding X then you need to update X server fonts on the X server you are forwarding it to.

Service fails to connect and locks future reconciliations

Examine the resource providers table.

1. On the database server
2. Connect to the ITIMDB database
3. Run the following command to verify that all values in the resource_status column are either 0 or Null

1. If some are 1 run the following command

SSL Handshake errors in SystemOut log

If you see CWPKI0022E SSL HANDSHAKE FAILURE: A signer with SubjectDN "...." was sent from target host:port . The signer may need to be added to local trust store "/opt/IBM/WebSphere/AppServer/java/jre/lib/security/cacerts" do the following on the first node

WAS_HOME=/opt/IBM/WebSphere/AppServer
CACERT_FILE=$WAS_HOME/java/jre/lib/security/cacerts
$WAS_HOME/java/jre/bin/keytool -export -keystore $WAS_HOME/profiles/Dmgr01/config/cells/isimCell01/nodes/isim1CellManager01/root-key.p12 -storepass WebAS -storetype pkcs12 -alias root -file /tmp/WasDMRoot.cert
$WAS_HOME/java/jre/bin/keytool -export -keystore $WAS_HOME/profiles/Dmgr01/config/cells/isimCell01/nodes/isim2Node01/root-key.p12 -storepass WebAS -storetype pkcs12 -alias root -file /tmp/WasNodeRoot.cert
$WAS_HOME/java/jre/bin/keytool -import -noprompt -trustcacerts -alias wasdmroot -file /tmp/WasDMRoot.cert -keystore $CACERT_FILE -storepass changeit -storetype jks
$WAS_HOME/java/jre/bin/keytool -import -noprompt -trustcacerts -alias wasnoderoot -file /tmp/WasNodeRoot.cert -keystore $CACERT_FILE -storepass changeit -storetype jks

On the second node copy /tmp/wasroot.cert and run import

scp user@$ISIM1_HOST:/tmp/WasDMRoot.cert /tmp
$WAS_HOME/java/jre/bin/keytool -export -keystore $WAS_HOME/profiles/AppSrv01/config/cells/isimCell01/nodes/isim2Node01/root-key.p12 -storepass WebAS -storetype pkcs12 -alias root -file /tmp/WasNodeRoot.cert
$WAS_HOME/java/jre/bin/keytool -import -noprompt -trustcacerts -alias wasdmroot -file /tmp/WasDMRoot.cert -keystore $CACERT_FILE -storepass changeit -storetype jks
$WAS_HOME/java/jre/bin/keytool -import -noprompt -trustcacerts -alias wasnoderoot -file /tmp/WasNodeRoot.cert -keystore $CACERT_FILE -storepass changeit -storetype jks

Restart the cluster

To check
$WAS_HOME/java/jre/bin/keytool -list -keystore $CACERT_FILE -storepass changeit -storetype jks | grep wasroot

Full TAM Combo adapter rgy reconciliation fails with SizeLimitExceeded

if the value for 'ldap.bind-dn=', is 'cn=root', change "ibm-slapdSizeLimit" value to 0, so it's unlimited, and restart LDAP, else create an ibm-searchLimits group in TAM's LDAP, to allow a regular user to do a big search, and add the user to that group.

How to clear ITIM message queues

• Stop Websphere service
• Delete \\SIMs server\c$\Program Files\ibm\WebSphere\AppServer\profiles\AppSrv01\tranlog\xxxNode01Cell\xxxNode01\server1\transaction
• Execute the following SQL commands on the enrole DB:

delete from ITIML000.SIB000
delete from ITIML000.SIB001

• Start Websphere service

How to check for locked services in ITIM enrole DB

Display services with their lock status
select s.erservicename, rp.recon_status, rp.lock_service, rp.resource_status from enrole.resource_providers rp, enrole.service s where s.dn = rp.resource_dn order by s.erservicename
Display services with hung resource_status
select s.erservicename, rp.recon_status, rp.lock_service, rp.resource_status from enrole.resource_providers rp, enrole.service s where s.dn = rp.resource_dn and resource_status > 0 order by s.erservicename
Display services with hung recon_status
select s.erservicename, rp.recon_status, rp.lock_service, rp.resource_status from enrole.resource_providers rp, enrole.service s where s.dn = rp.resource_dn and recon_status > 0 order by s.erservicename

Check for any type of lock in the DB:

select count(*) from enrole.resource_providers where recon_status > 0;
select count(*) from enrole.resource_providers where resource_status > 0;
select count(*) from enrole.resource_providers where lock_service > 0;

Resetting the locks

update enrole.resource_providers set recon_status=0 where recon_status > 0
update enrole.resource_providers set resource_status=0 where resource_status > 0
update enrole.resource_providers set lock_service=0 where lock_service > 0

How to resolve ITIM hunging ops due to an adapter problem

Here are some queries you could try in the db:

select * from enrole.resource_providers
update enrole.resource_providers set resource_status=0 where resource_status=2
select count (*) from enrole.remote_services_requests
select distinct service_dn from enrole.remote_services_requests
update enrole.resource_providers set resource_status=1 where resource_dn like 'erglobalid=xxxx%'
update enrole.resource_providers set resource_status=1 where resource_status=2
select * from enrole.remote_services_requests where request_id='xxxxxxxxxxx'
select request_id from enrole.resource_providers where resource_status=2
select request_id from enrole.remote_services_requests where service_dn like 'erglobalid=xxxxx%'
update enrole.resource_providers set request_id='xxxxxxxxxxxxxxxx' where resource_status=2
update enrole.resource_providers set resource_status=1 where resource_status=2
select type from enrole.remote_services_requests
select request_id from enrole.remote_services_requests where type=2

1. The following query provided a list of "good" request id's, which we used in one of the above sql queries to try to set the request_id value to an ID that the system could use. This might not solve the problem
2. Rdited Program Files\ibm\itim\data\enRoleLogging.properties and set

logger.trace.com.ibm.itim.remoteservices.level=DEBUG_MAX

1. Find the service id's that might be involved

erglobalid=xx,ou=services,erglobalid=00000000000000000000,ou=xxx,dc=itim,dc=dom
And do do
select request_id from enrole.resource_providers where resource_status=2

Aborting pending reports synchronization AKA unlocking ITIM Data synchronization

Also known as How to kill a Data Sync request in ITIM

 SELECT STATUS, STATUS_DETAIL, STARTED_TIME FROM enrole.SYNCHRONIZATION_HISTORY WHERE REQ_TYPE ='DS'
 select count(status) FROM enrole.SYNCHRONIZATION_HISTORY WHERE status ='Started' and req_type='DS'
 update enrole.SYNCHRONIZATION_HISTORY set status = 'Aborted' where status = 'Started'

Audit DB2 locks and indoubt transaction keeps ITIM from starting up

clear them first. first try to roll them back. If fails try to cancel them. then clear the locks as needed (most if not all locks will be cleared by rolling back the indoubt transaction)

DB2 Tables for reporting

The ENTITY_COLUMN table is the one which contains the report status for the various report tables. It is rare that this does become out of sync but I have seen it happen. When it does it is not always that easy to determine which tables need to be adjusted in the AVAILABLE_FOR_REPORTING column and it would be best to open a pmr if you ever run into this type of situation.

GCE failst to start

copy JRE 1.4.2 from IBM into the root folder of GCE under /jre

DB2 has locks prohibiting changes

db2 get snapshot for locks on database

DB2 improperly shutdown with transaction in flight

Shutdown everything. Rollback any indoubt db2 to clear possible locks (if any are present when nothing is connecting to the db2) db2 snapshot on locks ... to see if any are presetn start half way up (not jmsserver) recreate message queues according to this page

Solution In the commands shown below - typical values for parameters are: For server the typical value is "server1" For nodename the typical value is the server host name
Examples of similar commands being used during the ITIM installation process can be seen in the $WAS_HOME\logs\createMQ.nodename_server.log file.
To Delete the Queue Manager and underlying content: Stop ITIM via the "stopServer" command in $WAS_HOME\bin
stopServer <server>
Delete the existing queue manager via the deletemq command in $WAS_HOME\bin
deletemq WAS <nodename> <server>
To create the Queue Manager: Create a new queue manager via the createmq command in $WAS_HOME\bin
createmq $WAS_HOME WAS <nodename> <server> $MQ_HOME $MQ_HOME\WEMPS
To start the Queue Manager: Start the queue manager via the "strmqm" command in $MQ_HOME\bin
strmqm WAS_<nodename>_<server>
To create the local queues within the queue manager: Start ITIM via the "startServer" command in $WAS_HOME\bin
startServer <server>
To verify that the local queues have been created: Utilize "runmqsc" found in the $MQ_HOME\bin directory against the queue manager (note: after the runmqsc command, you will enter the display qlocal command at a blank prompt.):
runmqsc WAS_<nodename>_<server>
display qlocal(*)
end
Look for the following local queues:
WQ_itim_adhocSync
WQ_itim_import_export
WQ_itim_ms
WQ_itim_policy
WQ_itim_policy_simulation
WQ_itim_ps
WQ_itim_rs
WQ_itim_rs_pending
WQ_itim_wf
WQ_itim_wf_shared

Note: If the local queues are not created after starting up ITIM, the WebSphere transaction logs might have to be deleted and ITIM restarted. To clear the WebSphere transaction logs, go to the $WAS_HOME\tranlog\<server>\transaction and delete the two directories and the two files in each of them.
start jmsserver

ITIM Generated PDFs are not opening correctly

There is an APAR at IBM (search the support site) showing that you need to remove pragma-cache and another caching variable But there is another case too: We have found two registry keys that seem to be the cause of our problems. The keys are listed below:

HKEY_USERS\ ?Users SSID? \Software\Microsoft\Active Setup\Installed Components\{89820200-ECBD-11cf-8B85-00AA005B4383}
HKEY_USERS\ ?Users SSID? \Software\Microsoft\Windows\CurrentVersion\Internet Settings

When we delete the two keys listed above, it fixes it (the keys are recreated after user logoff/logon)

ITIM does not work with Enterprise MQ

In addition, I was informed that the install MUST use the embedded MQ installation, as TIM does not operate correctly with the Enterprise MQ.

ITIM requests stuck in Pending Requests queue and aborts dont respond.

Problem Requests remain Pending queue with a 'Running' status and requests submitted after that remain in 'Not Started' state. Attempts to abort the Requests don't execute. The itim.log file shows no activity, browsing the application via the UI still works fine.

Cause: It is possible that a request could still be processing in the ITIM server, specifically in JMS messaging.
Solution This solution applies to Websphere Application Server 5.0.x with MQ Series 5.3 embedded. Check the MQ queues for existing messages, specifically the CURDEPTH attribute. This can be done via a command line prompt on either Windows or Unix server installs:

1. For single server environments, type "runmqsc WAS_hostname_server instance name" at the command line prompt to access MQ's Script Command functionality. For example, enter 'runmqsc WAS_alpha_server1' to connect to the 'server1' instance with the Script Command utility on the "alpha" server.
2. For cluster environments, type "runmqsc WAS_hostname_jmsserver" at the command line prompt to access MQ's Script Command functionality. For example, enter 'runmqsc WAS_alpha_jmsserver' to connect to the Script Command utility on the "alpha" server.

1. Enter "dis ql('WQ_itim_*') CURDEPTH" to display the CURDEPTH value for all of the ITIM queues. This should display a value of zero for all queues (see sample output below) if there are no messages currently pending, processing:

dis ql('WQ_itim_*') CURDEPTH
dis ql('WQ_itim_*') CURDEPTH
AMQ8409: Display Queue details.
QUEUE(WQ_itim_adhocSync) CURDEPTH(0)
AMQ8409: Display Queue details.
QUEUE(WQ_itim_ms) CURDEPTH(0)
AMQ8409: Display Queue details.
QUEUE(WQ_itim_rs) CURDEPTH(0)
AMQ8409: Display Queue details.
QUEUE(WQ_itim_rs_pending) CURDEPTH(0)
AMQ8409: Display Queue details.
QUEUE(WQ_itim_wf) CURDEPTH(99)
AMQ8409: Display Queue details.
QUEUE(WQ_itim_wf_abort) CURDEPTH(0)
AMQ8409: Display Queue details.
QUEUE(WQ_itim_wf_pending) CURDEPTH(0)

1. If any of these returns a CURDEPTH value greater than zero then there are still messages being processed. This would most likely occur if a large request or large volume of requests were submitted in ITIM. If there is a CURDEPTH value/s greater than zero, continue running the "dis ql('WQ_itim_*') CURDEPTH" command and watch to see if the numbers/values are changing. The numbers should change (at least every few minutes) and eventually return to zero.

1. If it is necessary to purge a queue/s of current messages you must stop WAS first. If you don't you will likely see a response like "AMQ8148: WebSphere MQ object in use." when you attempt clear the queue/s.

When all queues have a CURDEPTH value of zero restart ITIM/WAS and confirm the previously hung process is again processing. This should allow ITIM to continue processing normally again and allow all Pending requests to process through.

It is possible however that this will not resolve the problem and allow Pending requests to continue through to completion. The requests may need to be aborted and resubmitted as a new request. The restart of ITIM/WAS should restart the MQ Queue Manager and allow it to receive and process requests/messages, but there could be something more to the problem that would prevent this from being sufficient action. If the problem persists at this point, that is following these steps has not resolved the problem, contact support for further assistance.

ITIM server wont process any new requests, all remain in the Pending Requests queue and a Request abort doesnt seem to respond

Connect to the mq you can run dspmq to see the list of queue servers to use with runmqsc. The server must have Running next to it
runmqsc WAS_<hostname>_<server instance name> (i.e runmqsc WAS_alpha_server1)
you will get NO prompt. Dont worry - just start inputing the lines. run to see if tim mq is not empty
dis ql('WQ_itim_*') CURDEPTH
clear qlocal('WQ_itim_ms'). If it says
"AMQ8143 MQSeries queue not empty"

First try to restart the jmsservers. No need to exit runmqsc, just stopserver jmsserver then startserver jmsserver. You might first have to start the channel by running start channel(WAS.JMS.SVRCONN) (same for all itim installs). If you are trying to clear up WQ_itim_wf and it is not clearing up try restarting server1, dmgs,nodeagent and jmsserver. Try to clean it again. Also try
alter ql('WQ_itim_wf') FORCE

Try starting ITIM and check again (?) More information

Logging

C:\Program Files\IBM\itim\data\enRoleLogging.properties
Set
logger.trace.level=DEBUG_MIN, MID or MAX

Provisioning policy enforcement fails with Object Class violation

CTGIMO017E The following directory server schema violation occurred. Error: [LDAP: error code 65 - Object Class Violation] Another symptom is it happens when empty values are being enforced to non-empty ones. May be only e-mail related. Fix: Restart the ITIM server. If not possible or a small number of error - try to localize the field(s) that is getting enforced and is causing the object class violation. Now open the LDAP browser for the ITIM LDAP, search for the failed entry and change the field(s) causing problems from null to anything else. Then re-run the policy enfrocement. TIM FP21 fixes (or was supposed to) the error. 2

Purging Tivoli Identity Manager database content

For more info about ITIM DB2 usage see the problem determination guide. 3

Cause The data in the ITIM database tables is not maintained automatically by ITIM, and therefore it must be managed manually. Solution The database tables used to store processes information are the following (with a brief explanation taken from the ITIM Problem Determination Guide): PROCESS: stores all the pending, running, and historical requests submitted to the Tivoli Identity Manager workflow. Each request is represented as a process. PROCESSLOG: maintains a record of audit events associated with a process. PROCESSDATA: stores the runtime process data of a process. After the process is completed, the record is removed. ACTIVITY: contains records of each workflow process's execution flow. WORKITEM: maintains a record of workitems associated with manual workflow activies for running processes. The records associated with the process are removed after the process is completed. PASSWORD_TRANSACTION: is used during secure password delivery to store information. After the password is retrieved, the record is deleted from the table. If the password is never picked up, this record is deleted upon password pickup expiration. PENDING: stores all the provisioning requests that are being processed, but not completed yet.

The audit information regarding completed/terminated processes can be purged by following these steps below, regardless the kind of DBMS being used (important note: once this archived data is removed there will no longer be archive/audit data for ITIM to reference; for this reason it is highly recommended that a backup of the database be done before the purge is executed): Start a SQL command processor and connect to the ITIM database (authenticating as the "enrole" user). Execute the following statements:

delete from ENROLE.PROCESSLOG where PROCESS_ID in (select ID from ENROLE.PROCESS where COMPLETED IS NOT NULL)
delete from ENROLE.PROCESSDATA where PROCESS_ID in (select ID from ENROLE.PROCESS where COMPLETED IS NOT NULL)
delete from ENROLE.WORKITEM where PROCESS_ID in (select ID from ENROLE.PROCESS where COMPLETED IS NOT NULL)
delete from ENROLE.PENDING where PROCESS_ID in (select ID from ENROLE.PROCESS where COMPLETED IS NOT NULL)
delete from ENROLE.PASSWORD_TRANSACTION where PROCESS_ID in (select ID from ENROLE.PROCESS where COMPLETED IS NOT NULL)
delete from ENROLE.ACTIVITY where PROCESS_ID in (select ID from ENROLE.PROCESS where COMPLETED IS NOT NULL)
delete from ENROLE.PROCESS where COMPLETED IS NOT NULL.

Close the connection to ITIM database and exit from the SQL command processor. The statements execution order is relevant because of the presence of foreign key constraints.

Reconciliation does not pull the values of attributes from the IDI search AL

Reconciliation does not pull the values of attributes from the IDI search assembly line. Check all attributes to match the ITIM config. especially if the could or could not be multivalued. In my example office code was multivalued in the ldap and it made none of the rest of the ldap attributes come accross back to ITIM. No errors were reported on the IDI or ITIM gui side (did not check the log).

SPNEGO error message

"Browser Configuration Error The application you are trying to access requires a change to your security settings." To resolve the problem yourself, click "Tools", "Internet Options", "Security", and select {zone you are in}. Click on "Custom Level", scroll to the bottom under "User Authentication", "Logon". Select "Anonymous logon", click "Ok" to close the window, and "Ok" to close the previous window. Shut down your browser and attempt to logon again.

Some attributes can not be created or deleted

Attributes and class names can not contain underscores for IDS. This is an incompatibility known to IBM. It seems to be due to the DB2 running under IDS

Workflows are not getting completed in a clustered TIM environment

Make sure the other half of the cluster is up. Stop the servers, delete WAS/translog/transactions folder and clear the mq queues. Start them back up. If still stuck check the DB2 indoubt transactions and the locks

Agent fails to start

If you have multiple agents check the ADK version required by an agent (use agentCfg) sometimes it is different with different agents and they dont work with the one installed

Fixing hung data synchronization for reports

To check if the data sync lock is held issue the following DB2 commands: 1) Start a DB command line as the ITIM DB2 instance owner. 2) CONNECT TO the ITIM DB. 3) Execute the following from the DB command line

EXPORT TO "C:\Scheduled_Message.wsf" OF WSF MODIFIED BY 1 MESSAGES "C:\Scheduled_Message.txt" SELECT * FROM ENROLE.SCHEDULED_MESSAGE;
EXPORT TO "C:\Synchronization_History.wsf" OF WSF MODIFIED BY 1 MESSAGES "C:\Synchronization_History.txt" SELECT * FROM ENROLE.SYNCHRONIZATION_HISTORY;

Then send the .wsf files. This are binary files. Or text files can be generated from the commands:

SELECT * FROM ENROLE.SCHEDULED_MESSAGE;
SELECT * FROM ENROLE.SYNCHRONIZATION_HISTORY;

To Clear out the "In Progress" state: Delete the row in the SYNCHRONIZATION_HISTORY table. Using the following command.

DELETE  FROM ENROLE.SYNCHRONIZATION_HISTORY WHERE STATUS='Started';  COMMIT;

You can determine that the lock is held in the SYNCHRONIZATION_HISTORY table then removing it from the table will clear the lock. If you continue to have problems the following entries in the ITIM enroleLogging.properties file will provide more information:

logger.trace.com.ibm.itim.report.level=DEBUG_MAX
logger.trace.com.ibm.itim.dataservices.model.level=DEBUG_MAX
logger.trace.com.ibm.itim.adhocreport.level=DEBUG_MAX
logger.trace.com.ibm.itim.apps.ejb.adhocreport.level=DEBUG_MAX

Providing correct timezone offset from TAM to TIM

Configuring the Time Zone Offset: After you logon to the Tivoli Identity Manager console, the Effective Date field displays the server time in GMT (Greenwich Mean Time) instead of the local browser time. The administrator can provide a workaround to have the desired time displayed.

• Embed the time zone offset value in the URL provided (publish) to users in the following format:

• Alternatively, you can publish the URL of a Web page on the Web server used by WebSEAL, or a Web page that is part of your corporate portal. The Web page presents the link to the WebSEAL SSO. The page should contain a JavaScript function that calculates the time zone offset between the client browser and GMT. Once clicked, the link should submit an HTTP request (with the auto-calculated time zone offset value passed as an HTTP parameter) to the Tivoli Identity Manager Server. When using this approach, the offset value is automatically calculated.

Tivoli Identity Manager Performance

• Complicated provisioning policies can result in complicated directory and database queries with poor performance. Policies with small numbers of roles and services will perform best.
• Provisioning policies without account approval workflows perform better than those with account approval workflows due to optimizations for the former case. Provisioning policies created by the system when a service is created use a Default Account Request Workflow. This account workflow should be removed from the provisioning policy if it is not needed to improve provisioning policy performance.
• Dynamic roles affect people in a given scope, either one-level or subtree. When a person object within that scope is modified or added, that role must be re-evaluated. This is true for every dynamic role in the system. For instance, if there are three dynamic roles with subtree scope and a person object within that scope is updated, all three dynamic roles will be re-evaluated. For this reason, it is recommended that you limit the number of dynamic roles, either by number or by scope, that affect people that are modified frequently. It does not matter if the dynamic role ends up enrolling the person or not, the evaluation itself is the performance-impacting overhead.

Workflow Design