From DocWiki

Jump to: navigation, search

Back: Troubleshooting Unified Intelligence Center

Add tip: Create Troubleshooting Tips

Print PDF: Print Troubleshooting Tips for Unified IC 9.0

The following tips were added by Cisco Documentation:

Installation

Installation: Locating Installation Log Files

Migration

System

Reporting

Accessibility

Charts and Gauges

Reports

Schedules

Security Center

Security Center: Updating User Roles

Value Lists and Collections

Administration (OAMP)

Admin User Management

Command Line Interface

Control Center

Device Configuration

License Management

Log and Trace Settings

RTMT

SNMP

Unified CCE User Integration

Unable to Delete UCCE system collections

CUIC Log file Growth - How to delete unwanted logs

The following commands can be run from either /var/log/active or /var/log/inactive

1. To print the total number of log files under current directory:

find . -type f -print | egrep -i "*\.out$|*\.zip$|*\.csv$|*\.txt$|*\.log$|*\.gz$|*_activity\.log_*" | wc -l

2. To print the total size of logs in bytes. (It may come as an exponential number, like 1.96367e+10. You can read it as 1.96367 * 10^10, ie, 19636700000 bytes, equalling around 19.6 Gb.)

find . -type f -print | egrep -i "*\.out$|*\.zip$|*\.csv$|*\.txt$|*\.log$|*\.gz$|*_activity\.log_*" | xargs ls -all | awk '{total += $5} END {print total}'

3. To delete all the log files under the current directory (/var/log/active or /var/log/inactive)

find . -type f -print | egrep -i "*\.out$|*\.zip$|*\.csv$|*\.txt$|*\.log$|*\.gz$|*_activity\.log_*" | xargs /bin/rm -f

After deleting the logs, you can run the command #1 and #2 above to check the count and size and make sure that they are deleted. Some files may still remain, but that is expected.

Log locations and Troubleshooting tips


Component/Area	Common Issues	Reasons for Failure and What needs to be done	Log Level to be enabled	Log files	Comments
CUIC - Report	• Execution fails with dataset error • Importing a zip file fails • Importing a report file fails • Import.outOfSync • Mismatch in the data reported by CUIC	Report execution failed due to some issues in the production of a dataset Import could not be completed: Unexpected error. This has been logged in the server logs. Please contact Cisco support Ex : 619441835, This error Import.outOfSync comes when the fields in the report definition of the imported XML and the fields in the existing report definition do not match. Take the SQL query build by CUIC and execute the same directly against MSSQL. Compare the no: of rows returned.	Report Failures :CUIC1 -> Cisco Unified Intelligence Center -> Log level = Detailed Subsystem Trace Definitions -> CUIC_SUBSYS_SETUP, CUIC_SUBSYS_Object	/ var/log/active/cuic/logs/cuic/Error-cuic.log / var/log/active/cuic/logs/cuic/CCBU-cuic.log	If there are issues with parsing the XML, CUIC logs will clearly mark it in the logs. Ex : [EXCEPTION=org.xml.sax.SAXParseException: The element type "valueListQuery" must be terminated by the matching end-tag "</valueListQuery>".]: There was an exception that occurred.
CUIC - Report Definition	Creating query parameters fail	• Create the parameters or correct the query syntax and recreate the parameters. An expression of non-boolean type specified in a context where a condition is expected, near ';'. • Note : End of SQL statement ';' should not be mentoned in the report definition query used in CUIC • Run the query directly against MSSQL and find if the query validation is successful. • If the query is successfully validated in MSSQL, it may be an issue with CUIC SQL parser.		/var/log/active/cuic/logs/cuic/Error-cuic*.	Query syntax errors are clearly defined in the logs
CUIC - Datasource	Exception message something like: Cannot open database "<..>" requested by the login. The login failed	One of the following could be wrong: - database name - port number - charset - user credentials	NA	/var/log/active/cuic/logs/cuic/Error-cuic.startup.log /var/log/active/cuic/logs/cuic/CCBU-cuic.startup.log	Exceptionmessage something like: Cannot open database "<..>" requested by the login. The login failed.
CUIC -- Dashboard, Valuelists				/var/log/active/cuic/logs/cuic/Error-cuic.startup.log /var/log/active/cuic/logs/cuic/CCBU-cuic.startup.log	??
CUIC - Scheduler	Schedule successful, but Email not sent Email body contains : Scheduled report execution failed - data processing error	Check the following • SMTP server is reachable • SFTP location is reachable Check the following • Datasource connectivity is up and running	Data processing center failure : CUIC1 -> CUIC DATA PROCESSING -> Log level = Detailed Subsystem Trace Definitions -> DCP,DCP_WORKER	/var/log/active/cuic/logs/cuic/Error-cuic. /var/log/active/cuic/logs/cuic/CCBU-cuic.
OAMP - Licensing	License file is invalid	Check the following • License file is not tampered • License file is generated with the correct licensing MAC. • To get the licensing MAC of CUIC, run the CLI 'show status'	OAMP -> Infrastructure -> Log level = DetailedSubsystem Trace Definitions -> TRACE_LICENSING	/var/log/active/cuic/logs/cuic/CCBU-cuic.log, Error-cuic.log
OAMP- CUIC status - unknown		• Run the foll URL : https://<IP>:8443/cuic-dp/rest/ReportingService/csvc/GetStatus to get the CUIC service status		• /var/log/active/cuic/logs/diag-portal/CCBU-diag-portal..startup.log • /var/log/active/cuic/logs/diag-portal /Error-diag-portal..startup.log
DBReplication	• Replication totally not setup • Replication is fine, but data is not seen across the cluster [OS Cache]	• Check the following: • Network connectivity. • Firewall blocking port 1500 • RTTgreater than 80 secs (WAN) • A Cisco DB Replicator svc should be up and running • A Cisco DB service service should be up and running • Cisco Database Layer Monitor service should be up and running • All nodes in the cluster should be installed with same CUIC version • All nodes in the cluster should have the same timestamp Look for the following messages in logs: (Check "Log Files" column for the list of log files ) ERROR: Error executing How to fix the Replication setup issue: Run the following CLIs in sequence • utils dbreplication stop all • utils dbreplication reset all If the above steps does not solve, then, execute the following in sequence: • utils dbreplication stop all • utils dbreplication clusterreset • utils dbreplication reset all Still Replication issue not resolved: Then, we need to suspect either shadow tables or syscdr database partially created. Follow the below steps to fix: • utils dbreplication stop all • utils dbreplication dropadmindbforce (run this command on all nodes in cluster) • utils dbreplication reset all How to fix if there is any descrepancy in data in nodes: Run the following CLI command: • utils dbreplication repair all How to syncup data of publisher onto Subscriber: Run the following CLI command: • utils dbreplication forcedatasyncsub		• /var/log/active/cm/trace/dbl/sdi/replication_scripts_output.log • /var/log/active/cm/trace/dbl/sdi/startrpc.log • /var/log/active/cm/trace/dbl/ .dbl_repl_cdr_define.log • /var/log/active/cm/trace/dbl/_dbl_repl_cdr_Broadcast.log • /var/log/active/cm/trace/dbl/*_dbl_repl_output_Broadcast.log • /var/log/active/cm/trace/dbl/sdi/dblrpc.err
CUIC-Oscache	Entities are not getting replicated across cluster although DB Replication is UP.	- Execute the command: netstat -an \| grep -i 45566. \\ - sockets status should be in LISTEN/ESTABLISHED.\\ - Output of the command should have entries same as number of nodes in cluster CUIC service " Intelligence Center Reporting Service" in other nodes may be down, bring it up port 45566 may not be LISTEN/ESTABLISHED state		/var/log/active/cuic/logs/cuic/Error-cuic.startup.log /var/log/active/cuic/logs/cuic/CCBU-cuic.startup.log
DRF (Backup and Restore)	Failed to take a successful backup Restore failed on a node	check the following services on publisher: - Cisco DRF Master - Cisco DRF Local - A Cisco DB check the following service on all subscribers: - Cisco DRF Local		/var/log/active/platform/drf/log/*log
CLI – show cuic component-status show cuic license-info show cuic properties show cuic tech show cuic trace	Command executed unsuccessfully			/var/log/active/platform/log/cli*log
CLI - platform				/var/log/active/platform/log/cli*log
Install	Install failed due to DB issue Install failed due to corrupt media	Refer to CUIC Install Guide Sec 'What do I do if the installation fails?'		/common/log/install/install*log /var/log/inactive/cm/trace/dbl/sdi/installdb_in.log.err
Upgrade		Search for the following messages in the log files: Error in line Exception during execution of ISAM error		/common/log/install/install*log /common/log/install/mmca-install.log /opt/cisco/cuic/log/cuic_l2_upgrade.log /partB/opt/cisco/cuic/log/cuic_l2_upgrade.log	Search for the following messages in the log files: Error in line Exception during execution of ISAM error
AFG (Answer File Generator)		check the following: Should not use IP of other nodes' in the same cluster as new IP		/var/log/active/cuic/logs/cuic_ip_change.log /var/log/active/platform/log/cli*log
Failed to Start “A Cisco DB” service				/var/log/active/cm/trace/dbl/sdi/start.log
Failed to start “Intelligence Center Reporting Service” service				/var/log/active/cuic/logs/cuicsrvr/catalina.out /var/log/active/cuic/logs/cuicsrvr /localhost..log /var/log/active/cuic/logs/cuic/CCBU-cuic..startup.log /var/log/active/cuic/logs/cuic/Error-cuic.*.startup.log
Failed to start “Cisco Tomcat” service				/var/log/active/cuic/logs/oamp/
User Authentication (LDAP)	Invalid username or password. Please try again.	Usually occurs when CUIC is not able to authenticate the user entered. check the following: Check the Active Directory Configuration OAMP -> Cluster Configuration -> Reporting Configuration. If LDAP server authentication is successful, then verify if the User Search Base is valid	CUIC1 -> CUIC Security -> Log level = Detailed Subsystem Trace Settings ->PERMISSION_OBJECT, PERMISSION_OPERATION	/var/log/active/cuic/logs/cuic/CCBU-cuic..startup.log /var/log/active/cuic/logs/cuic/Error-cuic..startup.log	This will give detailed level of logging including the VALID domains for login, logged-in user’s dn etc.
UCCE User Integration	Sync fails with errors	Previous sync has not deleted the temp tabls created for bulk insert of UCCE entities.	CUIC1 -> CUIC Security -> Log level = Detailed Subsystem Trace Settings ->PERMISSION_OBJECT, PERMISSION_OPERATION	/ var/log/active/cuic/logs/cuic/Error-cuic*.log	SQLException: Temp table (tanya_perm_toadd_vlteams) already exists in session.: There was an exception that occurred. 0000000415: 10.78.90.232: Jun 26 2012 16:53:14.470

Audit Trail Report Schema


Column Name	Data Type	Maximum Allowed Length	Description
ID	Char	32	UUID to uniquely indentify a row in a table
USERID	Char	32	User ID of the user who performed a particular operation
RUNASUSERID	Char	32	User ID of the RunAs user who performed a particular operation
CATEGORY	nvarchar	32	Entity type, for example, REPORT, DATASOURCE, VALUELIST, REPORTDEFINITION
OPERATION	nvarchar	32	Operation performed by a user, for example, CREATE, SAVE, UPDATE, IMPORT,EXPORT
LOGGEDTIME	datetime year to fraction(3)		Date and Time of the operation
DESCRIPTION	lvarchar	1000	Detailed description of the performed operation
SERVERIP	nvarchar	100	IP address of the Unified Intelligence Center server
SERVERNAME	nvarchar	100	Hostname of the Unified Intelligence Center server
STATUS	nvarchar	32	Status of the operation SUCCESS or FAILURE

Raw Schema:

CREATE TABLE informix.CUICAUDITLOG(
ID char(32),
USERID char(32),
RUNASUSERID char(32),
CATEGORY nvarchar(32),
OPERATION nvarchar(32),
LOGGEDTIME datetime year to fraction(3) NOT NULL,
DESCRIPTION lvarchar(1000),
SERVERIP nvarchar(100),
SERVERNAME nvarchar(100),
STATUS nvarchar(32)
)

Currently 0.00/5
1
2
3
4
5

Rating: 0.0/5 (0 votes cast)

Troubleshooting Tips for Unified IC 9.0