FreeIPA pki-tomcatd Service: STOPPEDUpdated at by ospi
After an update in March a small bug swam under my radar until the moment I had to get rid of a host in FreeIPA and Web UI just gave an internal server error 500 on delete command.
# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING pki-tomcatd Service: STOPPED ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful
Systemd service was green but the app failed on every request
systemctl status email@example.com server: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@4487f3fa background process server: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable server: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
/var/log/pki/pki-tomcat/ca/debug could provide more insight:
[09/Mar/2023:02:54:17][authorityMonitor]: Can't create master connection in LdapBoundConnFactory::getConn! Could not connect to LDAP server host free-as-in-ipa.local port 636 Error netscape.ldap.LDAPException: Unable to create socket: java.net.ConnectException: Connection refused (Connection refused) (-1) [09/Mar/2023:02:54:17][authorityMonitor]: authorityMonitor: failed to get LDAPConnection. Retrying in 1 second. [09/Mar/2023:02:54:17][ContainerBackgroundProcessor[StandardEngine[Catalina]]]: Can't create master connection in LdapBoundConnFactory::getConn! Could not connect to LDAP server host free-as-in-ipa.local port 636 Error netscape.ldap.LDAPException: Unable to create socket: java.net.ConnectException: Connection refused (Connection refused) (-1)
Floblanc's post Troubleshooting FreeIPA: pki-tomcatd fails to start points to it being an issue of tomcat not being able to read it's credentials or the credentials are out-of-sync with dirsrv.
When listing the mentioned NSS database content with
certutil I noticed that all the keys had "NSS Certificate DB:" -prefix. And
certutil would happily return the correct key or certificate with and without prefix in
-n nickname parameter.
Keys had the prefix
certutil -d dbm:/etc/pki/pki-tomcat/alias -K -f /etc/pki/pki-tomcat/alias/pwdfile.txt
Certificates without prefix
certutil -d dbm:/etc/pki/pki-tomcat/alias -L -f /etc/pki/pki-tomcat/alias/pwdfile.txt
Tomcat-pki's credentials for the authentication in
/etc/pki/pki-tomcat/ca/CS.cfg didn't have the prefix:
clientCertNickname with the prefixed version.
authz.instance.DirAclAuthz.ldap.ldapauth.clientCertNickname=NSS Certificate DB:subsystemCert cert-pki-ca
ipactl restart all services were running normally.