Web Plugin Framework stopped working

From DocWiki

Jump to: navigation, search

Web Plugin Framework (web admin UI) stopped working on a Remote Expert Mobile HA Cluster

Problem Summary Remote Expert Mobile Server is installed in HA mode, i.e. Master node and at least one Slave node is installed.

While accessing web plugin framework using following URL:


User fails to use web admin UI with error message being displayed based on web page being loaded.

Server log on master node would contain ERROR messages related to "PerformanceLogCache". Below is an example:

2015-04-20 23:03:06,325 ERROR [org.infinispan.loaders.file.FileCacheStore] (MSC service thread 1-10) | ISPN000062: Error while reading from file: /opt/cisco/ java.io.EOFException: The stream ended unexpectedly. Please check whether the source of the stream encountered any issues generating the stream.
at org.infinispan.marshall.VersionAwareMarshaller.objectFromObjectStream(VersionAwareMarshaller.java:191) [infinispan-core-5.1.6.FINAL.jar:5.1.6.FINAL]
at org.infinispan.marshall.AbstractDelegatingMarshaller.objectFromObjectStream(AbstractDelegatingMarshaller.java:79) [infinispan-core-5.1.6.FINAL.jar:5.1.6.FINAL]
at org.infinispan.loaders.file.FileCacheStore.objectFromInputStreamInReentrantMode(FileCacheStore.java:433) [infinispan-core-5.1.6.FINAL.jar:5.1.6.FINAL]
at org.infinispan.loaders.file.FileCacheStore.loadBucket(FileCacheStore.java:306) [infinispan-core-5.1.6.FINAL.jar:5.1.6.FINAL]
at org.infinispan.loaders.file.FileCacheStore.loopOverBuckets(FileCacheStore.java:101) [infinispan-core-5.1.6.FINAL.jar:5.1.6.FINAL]
at org.infinispan.loaders.bucket.BucketBasedCacheStore.loadAllLockSafe(BucketBasedCacheStore.java:185) [infinispan-core-5.1.6.FINAL.jar:5.1.6.FINAL]
at org.infinispan.loaders.LockSupportCacheStore.loadAll(LockSupportCacheStore.java:152) [infinispan-core-5.1.6.FINAL.jar:5.1.6.FINAL]
at org.infinispan.loaders.CacheLoaderManagerImpl.loadState(CacheLoaderManagerImpl.java:217) [infinispan-core-5.1.6.FINAL.jar:5.1.6.FINAL]
at org.infinispan.loaders.CacheLoaderManagerImpl.preload(CacheLoaderManagerImpl.java:186) [infinispan-core-5.1.6.FINAL.jar:5.1.6.FINAL]

Error Message Web plugin page will receive 500 error or some other error based on page being requested,
Possible Cause

Issue is caused by time not being in sync on between Master and Slave(s) node of the HA Cluster.

This result in infinispan cache being corrupted hence cluster fails to respond correctly.

Recommended Action Resolution of this issue require 2 steps:

Step 1 Ensure time and date are in sync between all REAS cluster elements:

It is recommended to use NTP service for this purpose. Please check with your network administrator if you are unsure about your NTP server.

Below are some example commands to help you with NTP server config:

  • NTP Server configuration: NTP server can be configured in "/etc/ntp.conf". The conf file "/etc/ntp.conf" should contain following line:

server <your ntp server address>

  • NTP Service : The ntpd service needs to be started to synchronise date and time with configured NTP server. Use following command to check status of ntpd service:

# service ntpd status
Note:Please restart ntpd service after changing ntp.conf, "service ntpd restart"

  • Test NTP Config: To verify that time has been successfully synchronised use ntpstat command. Allow couple of minutes for service to get synchronised. Below is an example of successful run of ntpstat command:

# ntpstat

synchronised to <ntp server name> at stratum 3
time correct to within 463 ms
polling server every 64 s

Step 2 Delete corrupted cache files:

Following steps can be taken to delete corrupted cache files:

  • Stop all services on master as well as slave node(s)

service reas stop

  • Delete cache files from master & slave(s). below are example commands:

cd <reas server install directory>/domain/servers/appserver-<node address>/data/infinispan/fcsdk/PerformanceStatistics/

rm *.

Note: If the files names start with a - e.g. -12345 you need to run rm -- -12345 to remove it.

  • Once cache files are removed start REAS service on all nodes.

service reas start

Release Remote Expert Mobile
Associated CDETS # None

Rating: 0.0/5 (0 votes cast)

Personal tools