Troubleshooting Tips for Unified IC 9.0

From DocWiki

(Difference between revisions)
Jump to: navigation, search
m (1 revision)
 

Latest revision as of 14:20, 23 August 2012

Back: Troubleshooting Unified Intelligence Center

Add tip: Create Troubleshooting Tips

Print PDF: Print Troubleshooting Tips for Unified IC 9.0


The following tips were added by Cisco Documentation:

Contents

Installation

Migration

System

Reporting

Accessibility

Charts and Gauges

Dashboards

Data Sources

Report Definitions

Reports

Schedules

Security Center


Value Lists and Collections

Administration (OAMP)

Admin User Management

Command Line Interface

Control Center

Device Configuration

License Management

Log and Trace Settings

RTMT

SNMP

Unified CCE User Integration

CUIC Log file Growth - How to delete unwanted logs


The following commands can be run from either /var/log/active or /var/log/inactive

1. To print the total number of log files under current directory:

find . -type f -print | egrep -i "*\.out$|*\.zip$|*\.csv$|*\.txt$|*\.log$|*\.gz$|*_activity\.log_*" | wc -l

2. To print the total size of logs in bytes. (It may come as an exponential number, like 1.96367e+10. You can read it as 1.96367 * 10^10, ie, 19636700000 bytes, equalling around 19.6 Gb.)

find . -type f -print | egrep -i "*\.out$|*\.zip$|*\.csv$|*\.txt$|*\.log$|*\.gz$|*_activity\.log_*" | xargs ls -all | awk '{total += $5} END {print total}'

3. To delete all the log files under the current directory (/var/log/active or /var/log/inactive)

find . -type f -print | egrep -i "*\.out$|*\.zip$|*\.csv$|*\.txt$|*\.log$|*\.gz$|*_activity\.log_*" | xargs /bin/rm -f


After deleting the logs, you can run the command #1 and #2 above to check the count and size and make sure that they are deleted. Some files may still remain, but that is expected.



Log locations and Troubleshooting tips

 
Component/Area Common Issues Reasons for Failure and What needs to be done Log Level to be enabled Log files Comments
CUIC - Report • Execution fails with dataset error

• Importing a zip file fails
• Importing a report file fails
• Import.outOfSync
• Mismatch in the data reported by CUIC

Report execution failed due to some issues in the production of a dataset


Import could not be completed: Unexpected error. This has been logged in the server logs. Please contact Cisco support
Ex : 619441835, This error Import.outOfSync comes when the fields in the report definition of the imported XML and the fields in the existing report definition do not match.


Take the SQL query build by CUIC and execute the same directly against MSSQL. Compare the no: of rows returned.

Report Failures :CUIC1 -> Cisco Unified Intelligence Center -> Log level = Detailed


Subsystem Trace Definitions -> CUIC_SUBSYS_SETUP, CUIC_SUBSYS_Object

/ var/log/active/cuic/logs/cuic/Error-cuic*.log

/ var/log/active/cuic/logs/cuic/CCBU-cuic*.log

If there are issues with parsing the XML, CUIC logs will clearly mark it in the logs.


Ex : [EXCEPTION=org.xml.sax.SAXParseException: The element type "valueListQuery" must be terminated by the matching end-tag "</valueListQuery>".]: There was an exception that occurred.


CUIC - Report Definition Creating query parameters fail • Create the parameters or correct the query syntax and recreate the parameters. An expression of non-boolean type specified in a context where a condition is expected, near ';'.

Note : End of SQL statement ';' should not be mentoned in the report definition query used in CUIC
• Run the query directly against MSSQL and find if the query validation is successful.
• If the query is successfully validated in MSSQL, it may be an issue with CUIC SQL parser.


/var/log/active/cuic/logs/cuic/Error-cuic*. Query syntax errors are clearly defined in the logs


CUIC - Datasource Exception message something like:

Cannot open database "<..>" requested by the login. The login failed

One of the following could be wrong:


- database name
- port number
- charset
- user credentials

NA /var/log/active/cuic/logs/cuic/Error-cuic*.startup.log


/var/log/active/cuic/logs/cuic/CCBU-cuic*.startup.log

Exceptionmessage something like:

Cannot open database "<..>" requested by the login. The login failed.



CUIC -- Dashboard, Valuelists


/var/log/active/cuic/logs/cuic/Error-cuic*.startup.log


/var/log/active/cuic/logs/cuic/CCBU-cuic*.startup.log

 ??


CUIC - Scheduler Schedule successful, but Email not sent


Email body contains : Scheduled report execution failed - data processing error

Check the following

• SMTP server is reachable • SFTP location is reachable
Check the following • Datasource connectivity is up and running

Data processing center failure :

CUIC1 -> CUIC DATA PROCESSING -> Log level = Detailed Subsystem Trace Definitions -> DCP,DCP_WORKER

/var/log/active/cuic/logs/cuic/Error-cuic*.


/var/log/active/cuic/logs/cuic/CCBU-cuic*.


OAMP - Licensing
License file is invalid
Check the following
• License file is not tampered
• License file is generated with the correct licensing MAC.
• To get the licensing MAC of CUIC, run the CLI 'show status'
OAMP -> Infrastructure -> Log level = DetailedSubsystem Trace Definitions -> TRACE_LICENSING
/var/log/active/cuic/logs/cuic/CCBU-cuic*.log, Error-cuic*.log


OAMP- CUIC status - unknown

• Run the foll URL : https://<IP>:8443/cuic-dp/rest/ReportingService/csvc/GetStatus
to get the CUIC service status

• /var/log/active/cuic/logs/diag-portal/CCBU-diag-portal.*.startup.log
• /var/log/active/cuic/logs/diag-portal /Error-diag-portal.*.startup.log


DBReplication
• Replication totally not setup
• Replication is fine, but data is not seen across the cluster [OS Cache]

• Check the following:
• Network connectivity.
• Firewall blocking port 1500
• RTTgreater than 80 secs (WAN)
• A Cisco DB Replicator svc should be up and running
• A Cisco DB service service should be up and running
• Cisco Database Layer Monitor service should be up and running
• All nodes in the cluster should be installed with same CUIC version
• All nodes in the cluster should have the same timestamp
Look for the following messages in logs: (Check "Log Files" column for the list of log files )



  • ERROR: Error executing

How to fix the Replication setup issue:
Run the following CLIs in sequence
• utils dbreplication stop all
• utils dbreplication reset all
If the above steps does not solve,
then, execute the following in sequence:
• utils dbreplication stop all
• utils dbreplication clusterreset
• utils dbreplication reset all
Still Replication issue not resolved:
Then, we need to suspect either shadow tables or syscdr database partially created.
Follow the below steps to fix:
• utils dbreplication stop all
• utils dbreplication dropadmindbforce (run this command on all nodes in cluster)
• utils dbreplication reset all
How to fix if there is any descrepancy in data in nodes:
Run the following CLI command:
• utils dbreplication repair all
How to syncup data of publisher onto Subscriber:
Run the following CLI command:
• utils dbreplication forcedatasyncsub


• /var/log/active/cm/trace/dbl/sdi/replication_scripts_output.log
• /var/log/active/cm/trace/dbl/sdi/startrpc.log
• /var/log/active/cm/trace/dbl/ *.dbl_repl_cdr_define.log
• /var/log/active/cm/trace/dbl/*_dbl_repl_cdr_Broadcast.log
• /var/log/active/cm/trace/dbl/*_dbl_repl_output_Broadcast.log
• /var/log/active/cm/trace/dbl/sdi/dblrpc.err


CUIC-Oscache
Entities are not getting replicated across cluster although DB Replication is UP.

 - Execute the command: netstat -an | grep -i 45566. \\
- sockets status should be in LISTEN/ESTABLISHED.\\
- Output of the command should have entries same as number of nodes in cluster

  • CUIC service " Intelligence Center Reporting Service" in other nodes may be down, bring it up
  • port 45566 may not be LISTEN/ESTABLISHED state

  • /var/log/active/cuic/logs/cuic/Error-cuic*.startup.log
  • /var/log/active/cuic/logs/cuic/CCBU-cuic*.startup.log

DRF (Backup and Restore)
  • Failed to take a successful backup
  • Restore failed on a node
 check the following services on publisher:
- Cisco DRF Master
- Cisco DRF Local
- A Cisco DB
check the following service on all subscribers:
- Cisco DRF Local


  • /var/log/active/platform/drf/log/*log

CLI –
show cuic component-status
show cuic license-info
show cuic properties
show cuic tech
show cuic trace

Command executed unsuccessfully


  • /var/log/active/platform/log/cli*log

CLI - platform



  • /var/log/active/platform/log/cli*log

Install
  • Install failed due to DB issue
  • Install failed due to corrupt media
Refer to CUIC Install Guide Sec 'What do I do if the installation fails?'

  • /common/log/install/install*log
  • /var/log/inactive/cm/trace/dbl/sdi/installdb_in.log.err

Upgrade

Search for the following messages in the log files:
  • Error in line
  • Exception during execution of
  • ISAM error

  •  /common/log/install/install*log
  •  /common/log/install/mmca-install.log
  •  /opt/cisco/cuic/log/cuic_l2_upgrade.log
  •  /partB/opt/cisco/cuic/log/cuic_l2_upgrade.log
Search for the following messages in the log files:
  • Error in line
  • Exception during execution of
  • ISAM error
AFG (Answer File Generator)

check the following:
  • Should not use IP of other nodes' in the same cluster as new IP



  • /var/log/active/cuic/logs/cuic_ip_change.log 
  • /var/log/active/platform/log/cli*log

Failed to Start “A Cisco DB” service



  • /var/log/active/cm/trace/dbl/sdi/start.log

Failed to start “Intelligence Center Reporting Service” service



  •  /var/log/active/cuic/logs/cuicsrvr/catalina.out
  •  /var/log/active/cuic/logs/cuicsrvr /localhost.*.log
  •  /var/log/active/cuic/logs/cuic/CCBU-cuic.*.startup.log
  •  /var/log/active/cuic/logs/cuic/Error-cuic.*.startup.log

Failed to start “Cisco Tomcat” service



  • /var/log/active/cuic/logs/oamp/


User Authentication (LDAP)
Invalid username or password. Please try again.

Usually occurs when CUIC is not able to authenticate the user entered.

check the following:

Check the Active Directory Configuration OAMP -> Cluster Configuration -> Reporting Configuration.

  • If LDAP server authentication is successful, then verify if the User Search Base is valid
CUIC1 -> CUIC Security -> Log level = Detailed Subsystem Trace Settings ->PERMISSION_OBJECT, PERMISSION_OPERATION
  • /var/log/active/cuic/logs/cuic/CCBU-cuic.*.startup.log 
  • /var/log/active/cuic/logs/cuic/Error-cuic.*.startup.log
This will give detailed level of logging including the VALID domains for login, logged-in user’s dn etc.
UCCE User Integration Sync fails with errors
Previous sync has not deleted the temp tabls created for bulk insert of UCCE entities.
CUIC1 -> CUIC Security -> Log level = Detailed Subsystem Trace Settings ->PERMISSION_OBJECT, PERMISSION_OPERATION
  • / var/log/active/cuic/logs/cuic/Error-cuic*.log
SQLException: Temp table (tanya_perm_toadd_vlteams) already exists in session.: There was an exception that occurred. 0000000415: 10.78.90.232: Jun 26 2012 16:53:14.470


Audit Trail Report Schema

 
Column Name Data Type Maximum Allowed Length Description
ID Char 32 UUID to uniquely indentify a row in a table
USERID Char 32 User ID of the user who performed a particular operation
RUNASUSERID Char 32 User ID of the RunAs user who performed a particular operation
CATEGORY nvarchar 32 Entity type, for example, REPORT, DATASOURCE, VALUELIST, REPORTDEFINITION
OPERATION nvarchar 32 Operation performed by a user, for example, CREATE, SAVE, UPDATE, IMPORT,EXPORT
LOGGEDTIME datetime year to fraction(3) Date and Time of the operation
DESCRIPTION lvarchar 1000 Detailed description of the performed operation
SERVERIP nvarchar 100 IP address of the Unified Intelligence Center server
SERVERNAME nvarchar 100 Hostname of the Unified Intelligence Center server
STATUS nvarchar 32 Status of the operation SUCCESS or FAILURE


Raw Schema:

CREATE TABLE informix.CUICAUDITLOG(
ID char(32),
USERID char(32),
RUNASUSERID char(32),
CATEGORY nvarchar(32),
OPERATION nvarchar(32),
LOGGEDTIME datetime year to fraction(3) NOT NULL,
DESCRIPTION lvarchar(1000),
SERVERIP nvarchar(100),
SERVERNAME nvarchar(100),
STATUS nvarchar(32)
)

Rating: 2.0/5 (2 votes cast)

Personal tools