Support #8692
MNDeployment #3213: University of Illinois, Chicago member node
UIC problem with replica node status
100%
Description
Issue: The replication queue sent from the Coordinating Node to our Member Node stays in the requested status forever.
Discoveries / Actions:
1. Discovered that the logs are in /var/log/messages and not the tomcat logs
2. The root logger in /var/lib/tomcat/webapps/metacat/WEB-INF was changed from WARN to DEBUG
Log4j.rootlogger=DEBUG, stdout
3. Discovered that the above issue is caused by SELinux preventing java from accessing the UIC node certificate.
Aug 26 04:32:59 dataone-membernode setroubleshoot: SELinux is preventing /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.151-2.6.11.1.el7_4.x86_64/jre-abrt/bin/java from getattr access on the file /var/metacat/cert/urn_node_UIC.pem
Deliverables:
1. Ade will ensure that the SELinux issue is fixed
2. Ade will contact Chris to setup another meeting, for the queue to be cleared and the replication request sent again to our node.
Related issues
History
#1 Updated by Amy Forrester over 5 years ago
- Blocks Task #8607: ESSDIVE: Register in Production added
#2 Updated by Amy Forrester over 5 years ago
- % Done changed from 0 to 100
- Status changed from New to Closed
9/5/185- per Chris
After looking at the log messages from tomcat, Ade figured out that the SELinux security rules were denying create/read/write permissions for tomcat in the /var/metacat directory (which is outside of its default allowed policies). Ade updated the SELinux policies to include this directory, and replication has begun to succeed again.
UIC currently has around .5TB of space, so Ade will monitor the usage as it fulfills replica requests over the next few days. ESS-DIVE requests will likely total around 30-50GB, but there are multiple other MNs making requests too. Ade may be able to provision more disk space.
#3 Updated by Amy Forrester over 5 years ago
9/13/18: update from Ade
As discussed, attached is the disk usage on our DataONE membernode monitored over a few days. At the moment there is a total available space of 356GB on our member node server and the status has remained same since 7th of September as indicated in the document.
per Chris: It looks like the UIC disk space is doing fine. The ESS_DIVE replicas on UIC are basically complete, with a few that have issues on the CN so won’t replicate over yet.
