Произошла неизвестная ошибка zimbra

Если при попытке залогиниться в Zimbra webmail начало ругаться «network error», то скорее всего кто-то еще с вашего IP адреса (актуально для офисов) несколько раз подряд залогинился неправильно, либо, что менее вероятно, кто-то или что-то сделало 30 запросов на вебсервер zimbra за секунду.

Проверить догадку можно в логе /opt/zimbra/log/mailbox.log:

INFO  [qtp509886383-44962:http://10.117.231.45:80/service/soap/AuthRequest] [name=sales@example.com;ip=10.117.231.45;ua=zclient/8.6.0_GA_1153;] SoapEngine - handler exception
INFO  [qtp509886383-44962:http://10.117.231.45:80/service/soap/AuthRequest] [name=sales@example.com;ip=10.117.231.45;ua=zclient/8.6.0_GA_1153;] soap - AuthRequest elapsed=3
INFO  [qtp509886383-44966:http://127.0.0.1:80/service/soap/AuthRequest] [name=sales@example.com;oip=1.2.3.4;ua=zclient/8.6.0_GA_1153;] SoapEngine - handler exception
INFO  [qtp509886383-44966:http://127.0.0.1:80/service/soap/AuthRequest] [name=sales@example.com;oip=1.2.3.4;ua=zclient/8.6.0_GA_1153;] soap - AuthRequest elapsed=3
WARN  [qtp509886383-44965:https://10.117.231.45:443/?loginOp=relogin&username=sales@example.com] [] webclient - auth credentials have expired
INFO  [qtp509886383-44966:http://127.0.0.1:80/service/soap/AuthRequest] [name=sales@example.com;oip=1.2.3.4;ua=zclient/8.6.0_GA_1153;] SoapEngine - handler exception: authentication failed for [sales@example.com], invalid password
INFO  [qtp509886383-44966:http://127.0.0.1:80/service/soap/AuthRequest] [name=sales@example.com;oip=1.2.3.4;ua=zclient/8.6.0_GA_1153;] soap - AuthRequest elapsed=4
INFO  [qtp509886383-44962:http://127.0.0.1:80/service/soap/AuthRequest] [] misc - Access to IP 1.2.3.4suspended, for repeated failed login.
INFO  [qtp509886383-44965:http://127.0.0.1:80/service/soap/AuthRequest] [] misc - Access to IP 1.2.3.4suspended, for repeated failed login.

Чтобы решить эту проблему достаточно добавить офисный IP (например 1.2.3.4) в whitelist и перезагрузить mailboxd:

# su - zimbra
$ zmprov mcf +zimbraHttpThrottleSafeIPs 1.2.3.4
$ zmmailboxdctl restart

Более подробно описано тут: https://wiki.zimbra.com/wiki/DoSFilter.

Problem

• Not able to login to admin and webclient after changing value of any attribute which exceeding the max allowed size (i.e 1000MB).
• Such as zimbraFileUploadMaxSize, zimbraMailContentMaxSize, zimbraMtaMaxMessageSize etc

While investigating, the admin may notice the following log entries which are constantly appearing in /opt/zimbra/log/mailbox.log

022-04-27 20:53:19,281 WARN  [qtp195615004-782:https://xxx.xxx.xxx.xxx/] [] webclient - system failure: ,: Service Unavailable
com.zimbra.common.service.ServiceException: system failure: error while proxying request to target server: Service Unavailable
ExceptionId:qtp195615004-782:https://xxx.xxx.xxx.xxx/:1651072999281:2a0061c1a4fb9afe
Code:service.FAILURE at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:288)

And, following entries from /opt/zimbra/log/nginx.log

2022/04/27 21:37:58 [error] 211623#0: *5 zm lookup: lookup handler  xxx.xxx.xxx.xxx:7072 sent error response: 503 Service Unavailable while SSL handshaking to lookup handler, client: aaa.aaa.aaa.aaa, server: mail.example.tld, request: "POST /service/soap/NoOpRequest HTTP/1.1", host: " xxx.xxx.xxx.xxx", referrer: "https:// xxx.xxx.xxx.xxx/#8"
2022/04/27 21:37:58 [warn] 211623#0: *5 zmauth: an error occurs during zm lookup: , fall back to IPHASH to get the upstream route while SSL handshaking to lookup handler, client: aaa.aaa.aaa.aaa, server: mail.exampl.tld, request: "POST /service/soap/NoOpRequest HTTP/1.1", host: " xxx.xxx.xxx.xxx", referrer: "https:// xxx.xxx.xxx.xxx/#8"

Analysis

• If the issue is unknown, start thinking of all the changes done on the server recently and revert all the changes one-by-one and check which change was affecting the server.(if the changes were done from the CLI, checking bash history will help quickly.)
• If various changes are done from the admin panel but not sure which were they, execute the below command and check all Size related attributes and check whether the size of any attribute is showing exceptionally high.

su - zimbra
zmprov -l gacf | grep -i size
zmprov -l gs `zmhostname` | grep -i size

• Also, search for the below kind of error in the /opt/zimbra/log/mailbox.log, there will be only one entry of this hence it is possible it may get overlooked while analysing the log file. (instead of zimbraFileUploadMaxSize there could be any attribute hence don’t search using the keyword zimbraFileUploadMaxSize)

com.zimbra.cs.account.AccountServiceException: zimbraFileUploadMaxSize must be a valid long:
at com.zimbra.cs.account.AccountServiceException.INVALID_ATTR_VALUE(AccountServiceException.java:212)

Once find the root cause do the following

Solution

• Verify the existing value on zimbraFileUploadMaxSize.

zmprov -l gcf zimbraFileUploadMaxSize 

• Now change the value to the allowed limit as follows (Example: let’s set it to 21MB) and restart mailbox services.

zmprov -l mcf zimbraFileUploadMaxSize 2147483647
zmmailboxdctl restart

Next

• If the above doesn’t resolve the issue, NE customers can raise a case with Zimbra support and provide the following details while submitting the case.
• Log files from the below location (if having a multi-server environment, provide the logs from the respective servers)

-/opt/zimbra/log/mailbox.log

-/opt/zimbra/log/ngnx.log

• An output of the following commands.

su - zimbra
zmprov -l gacf
zmprov -l gs `zmhostname`
zmlocalconfig 
zmcontrol -v

Above are quite basic details, that are required to start an investigation on the initial level at least, further, the support engineer may ask for more details If needed.

Submitted by: Amol Mistry

Zimbra на одном из дружественных серверов внезапно перестала выходить на связь. Сообщая при авторизации «Ошибка сети»

Перезапуск сервера ничего не дал. Запуск зимбры вручную выдавал ошибку ldap:

$ zmcontrol start
Host mail.yourdomain.com
Unable to determine enabled services from ldap.
Unable to determine enabled services. Cache is out of date or doesn’t exist.

Попытки проверить права доступа, постучать в бубен, выйти и войти ничего не дали. Единственой зацепкой стал тот факт, что устонавливали зимбру примерно год назад.

Гугление принесло интересный результат — сдох сертификат SSL. Чисто от старости. Благо, что все равно самоподписанный.
Единственная найденная осмысленная дока давала следующие советы:

Первые шаги надо делать от рута.
Генерим Certificate Authority (CA).

# /opt/zimbra/bin/zmcertmgr createca -new

** Creating /opt/zimbra/ssl/zimbra/ca/zmssl.cnf…done
** Creating CA private key /opt/zimbra/ssl/zimbra/ca/ca.key…done.
** Creating CA cert /opt/zimbra/ssl/zimbra/ca/ca.pem…done.
Теперь генерим сертификат, подписанный CA еще на 365 дней.

# /opt/zimbra/bin/zmcertmgr createcrt -new -days 365

Validation days: 365
** Creating /opt/zimbra/conf/zmssl.cnf…done
** Backup /opt/zimbra/ssl/zimbra to /opt/zimbra/ssl/zimbra.20101009200401
** Generating a server csr for download self -new -keysize 1024
** Creating /opt/zimbra/conf/zmssl.cnf…done
** Backup /opt/zimbra/ssl/zimbra to /opt/zimbra/ssl/zimbra.20101009200401
** Creating server cert request /opt/zimbra/ssl/zimbra/server/server.csr…done.
** Saving server config key zimbraSSLPrivateKey…failed.
** Signing cert request /opt/zimbra/ssl/zimbra/server/server.csr…done.

Теперь развертываем сертификат.

# /opt/zimbra/bin/zmcertmgr deploycrt self

** Saving server config key zimbraSSLCertificate…done.
** Saving server config key zimbraSSLPrivateKey…done.
** Installing mta certificate and key…done.
** Installing slapd certificate and key…done.
** Installing proxy certificate and key…done.
** Creating pkcs12 file /opt/zimbra/ssl/zimbra/jetty.pkcs12…done.
** Creating keystore file /opt/zimbra/mailboxd/etc/keystore…done.
** Installing CA to /opt/zimbra/conf/ca…done.

Теперь развертываем CA

# /opt/zimbra/bin/zmcertmgr deployca

** Importing CA /opt/zimbra/ssl/zimbra/ca/ca.pem into CACERTS…done.
** Saving global config key zimbraCertAuthorityCertSelfSigned…done.
** Saving global config key zimbraCertAuthorityKeySelfSigned…done.
** Copying CA to /opt/zimbra/conf/ca…done.

И, наконец, смотрим что у нас получилось:

# /opt/zimbra/bin/zmcertmgr viewdeployedcrt

::service mta::
notBefore=Oct 9 13:04:03 2010 GMT
notAfter=Oct 9 13:04:03 2011 GMT
subject= /C=US/ST=N/A/O=Zimbra Collaboration Suite/OU=Zimbra Collaboration

Suite/CN=mail.yourdomain.com
issuer= /C=US/ST=N/A/L=N/A/O=Zimbra Collaboration Suite/OU=Zimbra Collaboration

Suite/CN=mail.yourdomain.com
SubjectAltName=
::service proxy::
notBefore=Oct 9 13:04:03 2010 GMT
notAfter=Oct 9 13:04:03 2011 GMT
subject= /C=US/ST=N/A/O=Zimbra Collaboration Suite/OU=Zimbra Collaboration

Suite/CN=mail.yourdomain.com
issuer= /C=US/ST=N/A/L=N/A/O=Zimbra Collaboration Suite/OU=Zimbra Collaboration

Suite/CN=mail.yourdomain.com
SubjectAltName=
::service mailboxd::
notBefore=Oct 9 13:04:03 2010 GMT
notAfter=Oct 9 13:04:03 2011 GMT
subject= /C=US/ST=N/A/O=Zimbra Collaboration Suite/OU=Zimbra Collaboration

Suite/CN=mail.yourdomain.com
issuer= /C=US/ST=N/A/L=N/A/O=Zimbra Collaboration Suite/OU=Zimbra Collaboration

Suite/CN=mail.yourdomain.com
SubjectAltName=
::service ldap::
notBefore=Oct 9 13:04:03 2010 GMT
notAfter=Oct 9 13:04:03 2011 GMT
subject= /C=US/ST=N/A/O=Zimbra Collaboration Suite/OU=Zimbra Collaboration

Suite/CN=mail.yourdomain.com
issuer= /C=US/ST=N/A/L=N/A/O=Zimbra Collaboration Suite/OU=Zimbra Collaboration

Suite/CN=mail.yourdomain.com
SubjectAltName=
#

Все!

Теперь переключаемся в пользователя zimbra и пробуем запустить:

~$ zmcontrol start
Host mail.yourdomain.com
Starting ldap…Done.
Starting logger…Done.
Starting convertd…Done.
Starting mailbox…Done.
Starting antispam…Done.
Starting antivirus…Done.
Starting snmp…Done.
Starting spell…Done.
Starting mta…Done.
Starting stats…Done.
$

Варианты сообщений об ошибке:

Unable to determine enabled services from ldap
Unable to determine enabled services Cache is out of date or doesnt exist
Saving global config key zimbraCertAuthorityCertSelfSigned failed
Saving server config key zimbraSSLPrivateKey failed
Unable to determine enabled services from ldap Unable to determine enabled services Cache is out of date or doesnt exist

0

3

Debug message: auth credentials have expired
An exception:
com.zimbra.common.soap.SoapFaultException: auth credentials have expired
ExceptionId:qtp509886383-139:https://192.168.0.4:8443/service/soap/CreateWaitSetRequest:1494401874887:990b7f4e44716f41:SoapEngine266
Code:service.AUTH_EXPIRED

Since zmprov and zmmailbox (with the -z/--zadmin option) have access to authenticate as admin, they should (at least have the option to) automatically re-authenticate when the admin auth token lifetime has passed.