FreeIPA pki-tomcatd Service: STOPPED

After an update in March a small bug swam under my radar until the moment I had to get rid of a host in FreeIPA and Web UI just gave an internal server error 500 on delete command.

# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
pki-tomcatd Service: STOPPED
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful

Systemd service was green but the app failed on every request

systemctl status pki-tomcatd@pki-tomcat.service

server[22516]: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@4487f3fa background process
server[22516]: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
server[22516]: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)

Luckily /var/log/pki/pki-tomcat/ca/debug could provide more insight:

[09/Mar/2023:02:54:17][authorityMonitor]: Can't create master connection in LdapBoundConnFactory::getConn! Could not connect to LDAP server host free-as-in-ipa.local port 636 Error netscape.ldap.LDAPException: Unable to create socket: java.net.ConnectException: Connection refused (Connection refused) (-1)
[09/Mar/2023:02:54:17][authorityMonitor]: authorityMonitor: failed to get LDAPConnection. Retrying in 1 second.
[09/Mar/2023:02:54:17][ContainerBackgroundProcessor[StandardEngine[Catalina]]]: Can't create master connection in LdapBoundConnFactory::getConn! Could not connect to LDAP server host free-as-in-ipa.local port 636 Error netscape.ldap.LDAPException: Unable to create socket: java.net.ConnectException: Connection refused (Connection refused) (-1)

Floblanc's post Troubleshooting FreeIPA: pki-tomcatd fails to start points to it being an issue of tomcat not being able to read it's credentials or the credentials are out-of-sync with dirsrv.

When listing the mentioned NSS database content with certutil I noticed that all the keys had "NSS Certificate DB:" -prefix. And certutil would happily return the correct key or certificate with and without prefix in -n nickname parameter.

Keys had the prefix

certutil -d dbm:/etc/pki/pki-tomcat/alias -K -f /etc/pki/pki-tomcat/alias/pwdfile.txt

Certificates without prefix

certutil -d dbm:/etc/pki/pki-tomcat/alias -L -f /etc/pki/pki-tomcat/alias/pwdfile.txt

Tomcat-pki's credentials for the authentication in /etc/pki/pki-tomcat/ca/CS.cfg didn't have the prefix:

authz.instance.DirAclAuthz.ldap.ldapauth.clientCertNickname=subsystemCert cert-pki-ca

After replacing clientCertNickname with the prefixed version.

authz.instance.DirAclAuthz.ldap.ldapauth.clientCertNickname=NSS Certificate DB:subsystemCert cert-pki-ca

and ipactl restart all services were running normally.