Esxi лог ошибок

Обновлено 18.02.2019

vmware logo

Добрый день! Уважаемые читатели и гости IT блога Pyatilistnik.org. Любой системный администратор просто обязан уметь производить диагностику и мониторинг своих служб и сервисов. Он должен знать где хранятся события служб и уметь их анализировать. Не так давно я вам подробно расписывал, как посмотреть логи Windows платформ, сегодня же мы разберем, где и как посмотреть логи ESXI гипервизора. Хотя сама система монолитная и стабильная, но и там приходится отлавливать различные нестандартные события, о которых вам могут подробно рассказать только логи.

Методы просмотра логов ESXI

Предположим у вас в инфраструктуре есть несколько установленных VMware ESXI 6.5 хостов и нет vCenter для управления ими, где централизовано можно мониторить логи.Случается какой-то инцидент и вам необходимо провести расследование, чтобы выяснить причину. Вот для таких вот ситуаций вы должны знать, где располагаются логи ESXI. Методов их посмотреть уйма, я вам покажу самые простые из них.

  • Посмотреть логи VMware ESXI хоста можно из shell оболочки
  • Получить доступ и выгрузить логи вы можете и через ssh клиента
  • С помощью веб интерфейса, удобно так как метод является кросплатформенным
  • Через оболочку и комадлеты PowerCLI

Месторасположение журналов в VMware ESXI

Перед тем, как я вам покажу все методы, я вам приведу сводную информацию, о том, где собственно лежат все файлы с логами. Напоминаю, что гипервизор построен на базе операционной системы Red Hat Enterprise Linux (RHEL), а из этого следует, что все логи располагаются в каталоге /var/logs/:

  • auth.log — один из важнейших журналов безопасности в ESXI. Сюда попадают данные, о всех подключениях по SSH к хосту. В файле можно обнаружить вот такие записи:

2019-02-15T14:57:53Z sshd[104228]: Connection from 192.168.59.1 port 60509
2019-02-15T14:57:53Z sshd[104228]: Accepted keyboard-interactive/pam for root from 192.168.59.1 port 60509 ssh2
2019-02-15T14:57:53Z sshd[104228]: pam_unix(sshd:session): session opened for user root by (uid=0)
2019-02-15T14:57:53Z sshd[104231]: Session opened for ‘root’ on /dev/char/pty/t0

auth.log в ESXI 6.5

Тут будет информация, о том, кто подключился и с какого ip-адреса, это полезно для аудита безопасности.

  • Файл boot.gz — данный архив содержит в себе текстовый файл, в который заносится информация, о всех загружаемых компонентах, драйверах. устройствах ESXI хоста. Разархивировать и посмотреть содержимое файла, можно с помощью обычного RAR архиватора.
  • clomd.log — Когда вы запускаете хост ESXi или повторно сканируете адаптер, хост обнаруживает все физические пути к устройствам хранения, доступным хосту. На основе набора правил утверждений хост определяет, какой подключаемый модуль многопутевого (multipathing) распространения (MPP) должен запрашивать пути к конкретному устройству и становится ответственным за управление поддержкой многопутевого распространения для устройства. Информация, о сканировании будет в файле clomd.log
  • configRP.log — сюда записываются изменения пула ресурсов
  • dhclient.log — Журнал DHCP-клиента, включая обнаружение, запросы на аренду и продление адресов.

dhclient.log

  • esxcli-software.log — содержит события, проверки подгружаемых VIB пакетов
  • esxupdate.log — лог по патчам исправления и обновлениям
  • hostd.log — Журналы службы управления хостом включают в себя задачи и события виртуальной машины и хоста ESXi, связь с агентом vSphere Client и VMware vCenter Server vpxa, а также соединения SDK.
  • hostdCgiServer.log — Автономная служба CGI, которая обрабатывает все HTTP-запросы для URL/cgi-bin. Она была перемещена из процесса hostd в собственный демон, чтобы у нас был способ собрать пакет vm-support, если hostd заблокирован.
  • hostd-probe.log — Проверяет доступность службы управления хостом

hostd-probe.log

  • hostprofiletrace.log — Журналы профиля хоста
  • iofilter-init.log — логи iofilterd_spm модуля
  • iofiltervpd.log — VAIO это платформа, которая позволяет третьим сторонам (партнерам) разрабатывать фильтры, которые работают в ESXi и могут перехватывать любые запросы ввода-вывода из гостевой операционной системы на виртуальный диск. IO не будет выпущен или зафиксирован на диске без обработки IO Filters, созданной третьими лицами. Фильтры выполняются внутри ESXi, а не в виртуальной машине. Кроме того, VAIO Framework обеспечивает централизованную установку и обновление. Логи, о данных событиях будут в данном журнале.

iofiltervpd.log

  • jumpstart-stdout.log — содержит информацию, о стартуемых плагинах
  • lacp.log — Журналы протокола управления агрегацией каналов (LACP)
  • nfcd.log — Журнал стека TCP/IP
  • rabbitmqproxy.log — rabbitmqproxy, прокси, работающий на хосте ESXi, который позволяет приложениям, работающим на виртуальных машинах, обмениваться данными с брокерами AMQP, работающими в сетевом домене vCenter.
  • rhttpproxy.log — HTTP-соединения проксируемые от имени других веб-сервисов хоста ESXi

rhttpproxy.log

  • sdrsinjector.log — Журнал инжектора устройства vSphere Storage DRS (Подробнее https://kb.vmware.com/s/article/2149938)
  • shell.log — Журналы использования ESXi Shell, включая включение/отключение и каждую введенную команду. В моем примере видно, что я вводил в консоли shell команду uname -a, для просмотра ядра.

shell.log

  • storagerm.log — Storage I/O Control логи
  • swapobjd.log — Swap VVol Object Daemon логи
  • sysboot.log — Журнал загрузки системы. Записывает события ESXi Power-ON, Power-OFF и события перезапуска
  • syslog.log — Инициализация службы управления, дисковые устройства, запланированные задачи и использование DCUI.
  • usb.log — Информация об устройствах USB, такая как обнаружение и передача, проброс на виртуальные машины.
  • vitd.log — vSAN iSCSI target Daemon. Содержит события целевой службы vSAN iSCSI.
  • vmauthd.log — информация об аутентификации
  • vmkdevmgr.log — Журнал диспетчера устройств VMware. Содержит информацию об устройствах, подключенных к хосту ESXi, таких как NIC, HBA
  • vmkernel.log — Базовые журналы VMkernel, в том числе данные об обнаружении устройств, хранении и сетевых устройствах, а также информация о драйвере, а также информация о запуске виртуальной машины и событиях драйвера.
  • vmkeventd.log — Это утилита для захвата событий VMkernel
  • vmkwarning.log — Сводка сообщений журнала предупреждений и предупреждений, взятых из журналов VMkernel.
  • vmksummary.log — Сводная информация о запуске и завершении работы хоста ESXi, а также о ежечасном heartbeat с указанием времени безотказной работы, количества запущенных виртуальных машин и потребления ресурсов службы.
  • vmware-vmsvc.log — лог, о версии Vmware Tools
  • vobd.log — Наблюдение за событиями VMkernel.
  • vprobe.log — Гибкая система динамического инструментария, предназначенная для обеспечения глубокого наблюдения за ESXi и виртуальными машинами, работающими на нем.
  • vpxa.log — Журналы агента vCenter Server vpxa, включая связь с vCenter Server и агентом hostd Host Management.
  • vvold.log — Журнал виртуальных томов
  • vsanmgmt.log — vSAN Management Service.
  • vsansystem.log — vSAN System Managed Service.
  • vsanvpd.log — vSAN Storage Provider Daemon

Просмотр логов ESXI через shell оболочку

Теперь когда мы с вами знаем, о месторасположении и назначении каждого из журналов ESXI хоста, мы можем рассмотреть методы их просмотра. Первым будет самый классический, редко используемый, через Shell оболочку. Для выполнения данного метода необходимо либо физическое присутствие перед монитором сервера, либо же обладать портом портом управления, который позволяет, через Java или HTML консоль видеть изображение с сервера, например ILO, IPMI, IMM или IDRAC.

Логинимся на сервер через DCUI интерфейс и находим пункт меню «View System Logs». У вас будут шесть пунктов:

  1. Syslog
  2. VmKernel
  3. Config
  4. Management Agent (hostd)
  5. VirtualCenter Agent (vpxa)
  6. Vmware ESXI Observation log (vobd)

логи ESXI DCUI

Для примера я выберу цифру 1 и посмотрю системные логи. Перед вами будет окно с логами ESXI, для перемещения вы можете использовать стрелки вниз или вверх, если нужно быстрее, то клавиши Page Up или Page Down.

логи ESXI shell

Полный список команд можно получить нажав клавишу «H». Выход в меню DCUI осуществляется через кнопку Q. Кстати если вы нажмете сочетание клавиш Alt+F12, то вы перейдете в лог vmkernel. Выйти из него Alt+F2. Alt+F1 даст вам доступ в shell консоли, где вы легко можете перемещаться по каталога гипервизора.

vmware-esxi-logs

Просмотр логов ESXI через браузер

Данный метод более универсальный и кросплатформенный, для его реализации нам понадобится всего лишь браузер, желательно свежий. В адресной строке введите путь:

https://dns-имя сервера или ip/host

У вас появится форма авторизации, указываем логин и пароль.

логи ESXI через браузер

На выходе вы получите полный список логов гипервизора. Кликнув по нужному он скачается у вас в браузере, после чего вы можете открывать его любым текстовым редактором.

логи ESXI через браузер

Просмотр логов ESXI через ssh

Данный метод так же очень простой и потребует ssh клиента, и предварительное включение ssh на ESXI. Тут нужно отметить два варианта доступа. Первый это классическая ssh консоль, где вы можете водить команды для просмотра нужного лога:

cat /var/log/hostd.log

Если нужна фильтрация, то воспользуемся grep

cat /var/log/hostd.log | grep error (у меня будут выведены все строки содержащие слово error)

логи ESXI через ssh

Либо вы может использовать графическое подключение через ssh, например, через WinSCP. У вас будет обычный файловый менеджер, где вы легко найдет нужный вам лог.

логи ESXI WinSCp

Просмотр логов ESXI через web-интерфейс

Начиная с версии ESXI 6, у гипервизора появился свой веб-интерфейс, для этого просто откройте в браузере вот такой путь:

https://ip-адрес или dns-имя сервера/ui

В левом меню выберите пункт «Monitor», далее вкладку «Logs», тут в верхней части будет список всех файлов, если выбрать любой из них, то вы увидите его содержимое.

логи ESXI через веб-интерфейс

Для удобства просмотра вы можете открыть его в новом окне

логи ESXI через веб-интерфейс

Либо сгенерировать пакет со всеми логами, так называемый «Bundle». Нажимаем «Generate support bundle». У вас появится задание по генерации.

логи ESXI через веб-интерфейс, создание boudle архива

Загружаем его, пакет в среднем весит около 100 мб.

Скачивание boundle пакета с логами esxi

Просмотр логов ESXI через Power-CLI

Есть еще один быстрый способ выгрузить boundle пакет, используя для этого командлеты Power-CLI. Открываем оснастку и соединяемся с сервером:

Connect-VIServer -Server ip-адрес или имя сервера

логи ESXI через Power-CLI, подключение к серверу

У вас выскочит форма авторизации, введите логин и пароль, после чего у вас должен появиться статус.

Для генерации пакета с логами ESXI введите команду:

Get-Log -Bundle -DestinationPath C:logs

логи ESXI через Power-CLI, генерация логов

В результате чего у вас в C:logs появится пакет, который будет так же некоторое время генерироваться.

логи ESXI-13

Еще можно централизованно собирать логи на сервер коллектор, но это уже другая история. С вами был Иван Семин, автор и создатель IT блога Pyatilistnik.org.

Log files are an important component of troubleshooting attacks and obtaining information about breaches. Logging to a secure, centralized log server can help prevent log tampering. Remote logging also provides a long-term audit record.

To increase the security of the host, take the following measures.

  • Configure persistent logging to a datastore. By default, the logs on ESXi hosts are stored in the in-memory file system. Therefore, they are lost when you reboot the host, and only 24 hours of log data is stored. When you enable persistent logging, you have a dedicated activity record for the host.
  • Remote logging to a central host allows you to gather log files on a central host. From that host, you can monitor all hosts with a single tool, do aggregate analysis, and search log data. This approach facilitates monitoring and reveals information about coordinated attacks on multiple hosts.
  • Configure the remote secure syslog on ESXi hosts by using ESXCLI or PowerCLI, or by using an API client.
  • Query the syslog configuration to make sure that the syslog server and port are valid.

See the vSphere Monitoring and Performance documentation for information about syslog setup, and for additional information on ESXi log files.

Configure Syslog on ESXi Hosts

You can use the vSphere Client, the VMware Host Client, or the esxcli system syslog command to configure the syslog service.

For information about using the esxcli system syslog command and other ESXCLI commands, see Getting Started with ESXCLI. For details how to open the ESXi firewall for the port specified in each remote host specification, see Configuring the ESXi Firewall .

Procedure

  1. Browse to the host in the vSphere Client inventory.
  2. Click Configure.
  3. Under System, click Advanced System Settings.
  4. Click Edit.
  5. Filter for syslog.
  6. To set up logging globally and configure various advanced settings, see ESXi Syslog Options.
  7. (Optional) To overwrite the default log size and log rotation for any of the logs:
    1. Click the name of the log that you want to customize.
    2. Enter the number of rotations and the log size you want.
  8. Click OK.

Results

Changes to the syslog options take effect.

Note: Syslog parameter settings that you define by using the vSphere Client or VMware Host Client are effective immediately. However, most settings you define by using ESXCLI require an additional command to take effect. For more details, see
ESXi Syslog Options.

ESXi Syslog Options

You can define the behavior of ESXi syslog files and transmissions by using a set of syslog options.

Apart from the base settings, such as Syslog.global.logHost, starting from ESXi 7.0 Update 1, a list of advanced options is available for customizations, and NIAP compliance.

Note: All audit record settings, beginning with
Syslog.global.auditRecord, take effect immediately. However, for other settings that you define by using ESXCLI, make sure to run the
esxcli system syslog reload command to enable the changes.

Table 1.
Legacy Syslog Options

Option ESXCLI command Description
Syslog.global.logHost

esxcli system syslog config set —loghost=<str>

Defines a comma-delimited list of remote hosts and specifications for message transmissions. If the loghost=<str> field is blank, no logs are forwarded. While no hard limit to the number of remote hosts to receive syslog messages exists, good practice is to keep the number of remote hosts to five or less. The format of a remote host specification is: protocol://hostname|ipv4|'['ipv6']'[:port]. The protocol must be one of TCP, UDP, or SSL. The value of a port can be any decimal number from 1 through 65535. If a port is not provided, SSL and TCP use 1514. UDP uses 514. For example: ssl://hostName1:1514.
Syslog.global.defaultRotate esxcli system syslog config set —default-rotate=<long> Maximum number of old log files to keep. You can set this number globally and for individual subloggers (see Syslog.global.defaultSize).
Syslog.global.defaultSize esxcli system syslog config set —default-size=<long> Default size of log files, in KiB. After a file reaches the default size, the syslog service creates a new file. You can set this number globally and for individual subloggers.
Syslog.global.logDir esxcli system syslog config set —logdir=<str> Directory where logs reside. The directory can be on mounted NFS or VMFS volumes. Only the /scratch directory on the local file system is persistent across reboots. Specify the directory as [datastorename] path_to_file, where the path is relative to the root of the volume backing the datastore. For example, the path [storage1] /systemlogs maps to the path /vmfs/volumes/storage1/systemlogs.
Syslog.global.logDirUnique esxcli system syslog config set —logdir-unique=<bool> Specifies the ESXi host name to be concatenated to the value of Syslog.global.logDir. It is critical that you enable this setting when multiple ESXi hosts log to a shared file system. Selecting this option creates a subdirectory with the name of the ESXi host under the directory specified by Syslog.global.LogDir. A unique directory is useful if the same NFS directory is used by multiple ESXi hosts.
Syslog.global.certificate.checkSSLCerts esxcli system syslog config set —check-ssl-certs=<bool> Enforces checking of SSL certificates when transmitting messages to remote hosts.

Table 2.
Syslog Options Available Starting from ESXi 7.0 Update 1

Option ESXCLI command Description
Syslog.global.auditRecord.storageCapacity esxcli system auditrecords local set —size=<long> Specifies the capacity of the audit record storage directory located on the ESXi host, in MiB. You cannot decrease the capacity of the audit record storage. You can increase the capacity before or after the audit record storage is enabled (see Syslog.global.auditRecord.storageEnable).
Syslog.global.auditRecord.remoteEnable esxcli system auditrecords remote enable Enables sending audit records to remote hosts. Remote hosts are specified by using the Syslog.global.logHost parameter.
Syslog.global.auditRecord.storageDirectory esxcli system auditrecords local set —directory=<dir> Specifies the location of the audit record storage directory. You cannot change the audit record storage directory while audit record storage is enabled (see Syslog.global.auditRecord.storageEnable).
Syslog.global.auditRecord.storageEnable esxcli system auditrecords local enable Enables the storage of audit records on an ESXi host. If the audit record storage directory does not exist, it is created with the capacity specified by Syslog.global.auditRecord.storageCapacity.
Syslog.global.certificate.checkCRL esxcli system syslog config set —crl-check=<bool> Enables checking the revocation status of all the certificates in an SSL certificate chain.

Enables verification of X.509 CRLs, which are not checked by default in compliance with industry conventions. A NIAP-validated configuration requires CRL checks. Due to implementation limitations, if CRL checks are enabled, then all certificates in a certificate chain must provide a CRL link.

Do not enable the crl-check option for installations not related to certification, because of the difficulty in properly configuring an environment that uses CRL checks.

Syslog.global.certificate.strictX509Compliance esxcli system syslog config set —x509-strict=<bool> Enables strict compliance with X.509. Performs additional validity checks on CA root certificates during verification. These checks are generally not performed, as CA roots are inherently trusted, and might cause incompatibilities with existing, misconfigured CA roots. A NIAP-validated configuration requires even CA roots to pass validations.

Do not enable the x509-strict option for installations not related to certification, because of the difficulty in properly configuring an environment that uses CRL checks.

Syslog.global.droppedMsgs.fileRotate esxcli system syslog config set —drop-log-rotate=<long> Specifies the number of old dropped message log files to keep.
Syslog.global.droppedMsgs.fileSize esxcli system syslog config set —drop-log-size=<long> Specifies the size of each dropped message log file before switching to a new one, in KiB.
Syslog.global.logCheckSSLCerts esxcli system syslog config set —check-ssl-certs=<bool> Enforces checking of SSL certificates when transmitting messages to remote hosts.

Note: Deprecated. Use
Syslog.global.certificate.checkSSLCerts in ESXi 7.0 Update 1 and later.

Syslog.global.logFilters esxcli system syslog logfile [add | remove | set] … Specifies one or more log filtering specifications. Each log filter must be separated by a double vertical bar «||». The format of a log filter is: numLogs | ident | logRegexp. numLogs sets the maximum number of log entries for the specified log messages. After reaching this number, the specified log messages are filtered and ignored. ident specifies one or more system components to apply the filter to the log messages that these components generate. logRegexp specifies a case-sensitive phrase with Python regular expression syntax to filter the log messages by their content.
Syslog.global.logFiltersEnable Enables the use of log filters.
Syslog.global.logLevel esxcli system syslog config set —log-level=<str> Specifies the log filtering level. You must change this parameter only when troubleshooting an issue with the syslog daemon. You can use the values debug for the most detailed level, info for the default detail level, warning for only warnings or errors, or error, only for errors.
Syslog.global.msgQueueDropMark esxcli system syslog config —queue-drop-mark=<long>) Specifies the percent of the message queue capacity at which messages are dropped.
Syslog.global.remoteHost.connectRetryDelay esxcli system syslog config set —default-timeout=<long> Specifies the delay before retrying to connect to a remote host after a connection attempt fails, in seconds.
Syslog.global.remoteHost.maxMsgLen esxcli system syslog config set —remote-host-max-msg-len=<long> For the TCP and SSL protocols, this parameter specifies the maximum length of a syslog transmission before truncation occurs, in bytes. The default maximum length for remote host messages is 1 KiB. You can increase the maximum message length to up to 16 KiB. However, raising this value above 1 KiB does not ensure that long transmissions arrive untruncated to a syslog collector. For example, when the syslog infrastructure that issues a message is external to ESXi.

RFC 5426 sets the maximum message transmission length for the UDP protocol to 480 bytes for IPV4 and 1180 bytes for IPV6.

Syslog.global.vsanBacking esxcli system syslog config set —vsan-backing=<bool> Allows log files and the audit record storage directory to be placed on a vSAN cluster. However, enabling this parameter might cause the ESXi host to become unresponsive.

ESXi Log File Locations

ESXi records host activity in log files, using a syslog facility.

Table 3.
ESXi Log File Locations

Component Location Purpose
Authentication /var/log/auth.log Contains all events related to authentication for the local system.
ESXi host agent log /var/log/hostd.log Contains information about the agent that manages and configures the ESXi host and its virtual machines.
Shell log /var/log/shell.log Contains a record of all commands typed into the ESXi Shell and shell events (for example, when the shell was enabled).
System messages /var/log/syslog.log Contains all general log messages and can be used for troubleshooting. This information was formerly located in the messages log file.
vCenter Server agent log /var/log/vpxa.log Contains information about the agent that communicates with vCenter Server (if the host is managed by vCenter Server).
Virtual machines The same directory as the affected virtual machine’s configuration files, named vmware.log and vmware*.log. For example, /vmfs/volumes/datastore/virtual machine/vwmare.log Contains virtual machine power events, system failure information, tools status and activity, time sync, virtual hardware changes, vMotion migrations, machine clones, and so on.
VMkernel /var/log/vmkernel.log Records activities related to virtual machines and ESXi.
VMkernel summary /var/log/vmksummary.log Used to determine uptime and availability statistics for ESXi (comma separated).
VMkernel warnings /var/log/vmkwarning.log Records activities related to virtual machines.
Quick Boot /var/log/loadESX.log Contains all events related to restarting an ESXi host through Quick Boot.
Trusted infrastructure agent /var/run/log/kmxa.log Records activities related to the Client Service on the ESXi Trusted Host.
Key Provider Service /var/run/log/kmxd.log Records activities related to the vSphere Trust Authority Key Provider Service.
Attestation Service /var/run/log/attestd.log Records activities related to the vSphere Trust Authority Attestation Service.
ESX Token Service /var/run/log/esxtokend.log Records activities related to the vSphere Trust Authority ESX Token Service.
ESX API Forwarder /var/run/log/esxapiadapter.log Records activities related to the vSphere Trust Authority API forwarder.


check-circle-line


exclamation-circle-line








close-line


Scroll to top icon

A VMware ESXi instance will generate a ton of events throughout its lifetime. These events, depending on type, will be written to one or more log files, the bulk of which are found under /var/log in the ESXi filesystem. As with any log file, the idea here is to help you and others troubleshoot issues and keep an eye on things by perhaps forwarding important events to a syslog server and such. Log files also make for an excellent audit trail so you can determine, for instance, who or what is accessing your hosts, VMs, etc.

As per VMware’s documentation, I’ve listed eight of the most frequently used log files on ESXi.

Some of the log files present on an ESXi host

Some of the log files present on an ESXi host

Note: If you are experiencing problems with ESXi you may consider patching to ensure you have the latest bugs fixed.


In addition to the above, every Virtual Machine hosted on ESXi will have corresponding VM log files called vmware.log recording events related to machine activity, system failures, hardware changes, migrations, status and so on. To view the log file, SSH to the ESXi host and navigate to the datastore on which the VM folder resides.

The path to the log file should be similar to /vmfs/volumes/<datastore>/<virtual machine>/vwmare.log as shown in the next screenshot.

Displaying the contents of a VM's log file on ESXi

Displaying the contents of a VM’s log file on ESXi

There are some adjustments you can make to the log rotation and logging options for a VM. I’ve summarized these as follows:

  • Disable or enable logging: Set the logging value in the VM’s VMX file to false or true respectively. You can do this by editing the VM’s settings (VM Options -> Advanced -> Edit Configuration) in vSphere Web client as shown.

Modifying the log file options for a VM

Modifying the log file options for a VM

Alternatively, logging can be enabled or disabled via the Enable Logging option. It is also possible to include diagnostic information via the Debugging and statistics drop down box.

Enabling logging for a VM in vSphere Web client

Enabling logging for a VM in vSphere Web client

  • Logfile size: Similarly, you can control the log file size by adding the log.rotateSize to VM’s VMX file. Ex: log.rotateSize=1024000 sets the log file maximum size to 1 MB.
  • Logfile retention: To set the level of log file rotation, add the log.keepOld value to the VM’s VMX file. Ex: log.keepOld=5 sets a maximum of 5 log files at any one instance, with the oldest being overwritten when a new log file is created. You’ll find these listed as vmware.log, vmware-1.logvmware-n.log.
  • Logfile name: To change the log file filename and/or location, add the log.fileName to the VM’s VMX file. Ex: log.fileName=vmLog.txt or  log.fileName=/vmfs/volume/VMLogFiles/vm1Log.txt.

Note: The above should work for most pre-ESXi 6.5 versions as per this KB. I’ve also tested this on ESXi 6.5 and the only parameter that seems to be unsupported is log.fileName in that the vmware.log filename is retained regardless of the value set.

Viewing ESXi log file contents


From Shell

An easy way to view logfiles on VMware ESXi is to SSH to the host and use old fashioned Linux commands such as cat, more, less, tail and head with a little bit of grep thrown in for filtering. Here are a couple of examples.

In this first one, I’m displaying the last 15 lines from the vmkwarning.log using the command tail -n 15 <filename>. This is marked (1) in the next screen screenshot. The text labeled (2), tells us that the host is failing to connect to host 192.168.20.20. As it turns out, 192.168.20.20 happens to be the IP address of a replication server I had set up on a second vCenter instance, which was powered off at the time.

Tailing and inspecting the content of a VMware ESXi log file

Tailing and inspecting the content of a VMware ESXi log file

Here’s another example, where I use the auth.log log file to determine if connections are being established from subnet 192.168.11.0 and by whom. To do this, cat auth.log and pipe it into grep filtering by the string 192.168.11 as shown. The output shows a number of successfully established SSH connections via the root account from the 192.168.11.45.

Using the auth.log file on ESXi to determine who accessed the host

Using the auth.log file on ESXi to determine who accessed the host

Using the ESXi host client

The ESXi host client makes it even easier to view the contents of a select number of log files. Navigate to Monitor -> Logs to view the list of logfiles available for viewing. Highlighting a log file, displays the contents in the underlying pane which can be copied to the clipboard and exported to file.

Viewing ESXi log files using the ESXi host client

Viewing ESXi log files using the ESXi host client

You can also generate a log bundle (see next section) using the Generate support bundle button labeled (4) in the above screenshot. When the task finishes, you are prompted to download the bundle to a folder on your computer.

Downloading a generated bundle file using the ESXi host client

Downloading a generated bundle file using the ESXi host client

Generating an ESXi Log Bundle


There may be times where you simply cannot solve an issue. Calling VMware support is one way to go about it and if you do, generating a log or host support file bundle is one thing you’ll be asked to do. This is then uploaded to VMware for further troubleshooting and diagnostics.

We’ve already seen how the bundle is generated via the ESXi host client. There are a couple more methods you can use.

The first is to run /usr/bin/vm-support from within an SSH session while logged as root. Once the bundle file is generated, you can copy it using scp or similar.

Generating an ESXi log bundle via script

Generating an ESXi log bundle via script

The second method is easier. Just point a browser to http://<ESXi IP address/cgi-bin/vm-support.cgi. You are promoted for the host’s credentials after which, the vm-support script is executed on ESXi. The generated bundle is then downloaded as a compressed tar file (TGZ). The process may take a while depending on the size of the logs, host utilization, uptime, etc.

Generating and downloading an ESXi log bundle via a CGI script

Generating and downloading an ESXi log bundle via a CGI script

You can then upload the bundle to VMware’s site using the vSphere Web client. Just navigate to Administration -> Support -> Upload File to Service Request and click on the Upload File to Service Request button. Finish off by selecting the bundle file with the Choose File button and press OK.

upload bundle

This post should have given you a basic introduction to some of the log files generated by VMware ESXi and VMs and how they are used to troubleshoot and diagnose problems. We’ve also had a look at ways by which a support bundle is generated whenever VMware support is called for.

For further details, have a look at the System Log Files section on the vSphere 6.5 documentation website.

[the_ad id=”4738″][the_ad id=”4796″]

In this blog, we will discuss the procedure to get the VMware ESXi/Hosts diagnostic logs. VMware technical support might request several files to help resolve technical issues. During internal troubleshooting, you might need to check the basic logs for yourself to identify the issue even before going to raise any case with VMware.

There are many components/locations available where you can see the various logs by logging to ESXi via putty itself. Let’s understand the VMware ESXi Log File Locations.

VMkernel

VMkernel contains records activities related to virtual machines and ESXi.

File Location/directory:

/var/log/vmkernel.log

VMkernel Warnings

VMkernel warnings contains records activities related to virtual machines.

File Location/directory:

/var/log/vmkwarning.log

VMkernel Summary

It is used to determine uptime and availability statistics for ESXi.

File Location/directory:

/var/log/vmksummary.log

ESXi Host Agent Log

ESXi Host Agent Log contains information about the agent that manages and configures the ESXi host and its virtual machines.

File Location/directory:

/var/log/hostd.log

vCenter Agent Log

It contains information about the agent that communicates with vCenter Server (if the host is managed by vCenter Server).

File Location/directory:

/var/log/vpxa.log

Shell Log

Shell log contains a record of all commands typed into the ESXi Shell and related events.

File Location/directory:

/var/log/shell.log

Authentication Logs

It contains all events related to authentication for the local system.

File Location/directory:

/var/log/auth.log

System Messages

It contains all general log messages and can be used for troubleshooting.

File Location/directory:

/var/log/syslog.log

Virtual Machines

It contains virtual machine power events, system failure information, tools status and activity, time sync, virtual hardware changes, vMotion migrations, machine clones, and so on. You can the same virtual machines logs hosted on affected ESXi if required to know more on it.

File Location/directory:

/vmfs/volumes/datastore/virtual machine/vwmare.log

How To Export System Log Files Of An ESXi

Your host must be connected to vCenter server to export the system logs file of an ESXi. You should have at least read-only permission for the same. Please follow the below steps to get the system logs:

  1. Navigate to Host and Cluster from Home Menu.
  2. Right Click on Target Host and select Export System Logs.
export system logs of an esxi
export system logs of an esxi

3. Once you click on Export System Logs, It will give you another window where you can start exporting the logs.

export system logs of an esxi
export system logs of an esxi

4. After clicking on Export Logs, It will take few minutes to gather the logs and gives you an option to download it.

Export the ESXi Support Bunddle Logs If host is not connected to vCenter Server

This is one of the best option to get the esxi system logs if it is not connected to vCenter server. You need to follow below steps:

  1. Login to web client of ESXi by using below URL method and with root credentials:
https://esx01.com/ui

2. Once you logged in, click on Action button. Then click on Generate support bundle.

esxi support bundle

esxi support bundle

3. It will take sometime to collect the esxi logs, can be monitored in recents tasks tab.

4. Once completes the task, you will be given an option to download it.

Download esxi support bundle
download esxi support bundle

How to diagnostic information for VMware ESXi using ESXi Shell

This is also a method to get the support bundle logs for an ESXi. You need to follow below steps:

  1. SSH to the ESXi with root
  2. Run the command vm-support from the command line.
  3. It will start downloading the support bundle that later can be used with the VMware support team for diagnostic purposes.

Another method is DCUI(Direct Console User Interface). Follow the steps below:

  • On iLO/IMM/iDRAC page, take the console of ESXi.
  • Click F2 to see view system logs
  • You will be able to see below options:
    • <1>Syslog
    • <2>VMkernal
    • <3>Config
    • <4>Management Agent(hostd)
    • <5>vCenter Agent(vpxa)
    • <6> VMware ESXi Observation log(vobd)
  • You can press the corresponding key to view the log at the console itself.

Hope this would help you to get the esxi system diagnostic logs easily.

Also read,

  • Collecting diagnostic information for VMware ESX/ESXi
  • How To Verify That The Memory Balloon Driver Is Enabled For VMware Virtual Machines
  • VMware ESXi-Arm Fling
  • 500 Internal Server Error – How To Troubleshoot And Fix

T-Blog Editor/ September 27, 2020/ Datacenter Infrastructure News, Engage the experts, Expert Advise and Opinion, Industry News and Expert Advice, VMware products

Reading Time-approximately: 3 minutes

With access to logs, you can troubleshoot your ESXi host and track issues as they crop up. You can access your logs via the console or a web browser with only a few simple steps.

Vladan Seget

By Vladan Seget

You can troubleshoot an ESXi host in a few different ways. The exact method depends on which version of ESXi you’re trying to troubleshoot and if the host is part of the vCenter server or accessible remotely via Secure Shell. VMware doesn’t recommend that you allow access via Secure Shell, so this guide will concentrate on using VMware ESXi logs with access via console or web browser.

Therefore, this article won’t include situations in which you completely lose connectivity to your ESXi host. In these cases, a hardware problem, such as a physical switch, network interface card or cable, could be the cause.

Access VMware ESXi logs via console

It’s easy to troubleshoot ESXi via logs if you know where to look; this saves you the effort of looking at everything to locate the root cause of your problems.

In order to access VMware ESXi logs from the ESXi Direct Connect User Interface, you have to be in the server room, at the console itself. You might be able to access the server remotely via the Intelligent Platform Management Interface (IPMI), Dell Remote Access Card, Hewlett Packard Enterprise Integrated Lights Out or another hardware remote access system; if so, you can use any of these technologies to fire up the remote ESXi host’s console.

At the console prompt, hit the F2 key, enter your root login and password, and click View System Logs. As its name suggests, this will pull up a list of available system logs, highlighted in Figure A. Choose the log you wish to view.

Figure A. You can use technologies like IPMI to access servers remotely and view system logs.

The log window that results isn’t user-friendly. Press the Q and ESC keys to quit the log window and the current login session.

There are six different types of VMware ESXI logs: Syslog captures message from the VMkernel and other components. Vmkernel shows the uptime and availability statistics. You use Config when a host hangs or crashes or an authentication issue occurs. Management Agent (hostd) records which logs are specific to the host services that connect your vSphere client to your ESXi host. VirtualCenter Agent (vpxa) records additional logs that appear when vCenter connects to and manages your ESXi. Finally, VMware ESXi Observation log (vobd) tracks all changes to the configuration of your ESXi host and results.

Access VMware ESXi logs via web browser

You can access your host directly from a web browser, which is especially useful for older versions of ESXi that don’t have the Host Client yet.

Put your host’s IP address in the address section of the following URL to access the host: https://address/host. When you receive a prompt to log in, use the root username, and enter the root password. This will allow you to access a window from which you can download or view logs directly via your browser.

Newer releases of ESXi have the ESXi Host Client built in, which gives you access to the logs and makes browsing and troubleshooting more convenient.

In your web browser, go to the following URL, replacing “address” with the IP address of your ESXi host: https://address/ui.

Select and expand your host on the left-hand side of the screen, and then choose the Logs tab. The logs should appear in the lower pane. You can choose a log by selecting a specific line; in this case, we’re looking for syslog.log.

You also search for specific logs and log lines with the search mechanism highlighted in Figure B. Once you locate a log, right-click the line to copy and paste. You can also generate a support bundle for the log via the Actions button.

Figure B. Search for logs with the search mechanism in the VMware Host Client.

Depending on what you’re looking for, it can be complex to troubleshoot ESXi. If your environment has central management via vCenter, there are built-in tools that can help. There are also other VMware products for monitoring and troubleshooting, usually available within the vSphere vCloud suite.

I suggest vRealize Operations Manager or vRealize Log Insight, which can ingest many gigabits of logs per day and provide synthetic graphics based on keywords or particular repetitive issues within your infrastructure.

Vladan Seget asks:

Which tools do you use to track and troubleshoot?

Join the Discussion

VMware log files are important because they allow you to find the root cause of a problem or point out that something is about to break. If something strange is happening within your infrastructure, the first thing to do is gather and check the logs.

Contents

  1. vCenter Server log files
  2. VMware ESXi host log files
  3. ESXi logs and locations
  • Author
  • Recent Posts

Vladan Seget is an independent consultant, professional blogger, vExpert 2009-2021, VCAP-DCA/DCD and MCSA. He has been working for over 20 years as a system engineer.

You can then send these log files to VMware support after opening a support request to investigate what’s going on and resolve problems you might have.

But different VMware products have different log locations and different ways to get these logs. How do you keep track of it all? When working in an enterprise environment, you’ll perhaps want to get software that will not only be able to centralize and gather all of these logs but also «ingest» the logs and tell you if there are more errors than usual.

A proactive option is monitoring software that can store logs in one location and analyze them to provide you with patterns identified as warnings or failures.

One such application is VMware Log Insight, but in this post, we will not install and configure this software. Today we only want to know which logs are the most important, how to get them, and how to send them to VMware.

vCenter Server log files

For environments using VMware vCenter Server, this operation of gathering support logs is now greatly simplified. To generate a support bundle, connect to your vCenter Server via the vSphere HTML 5 Web Client and then go to Home > Hosts and Clusters. Select the vCenter Server you want to generate support logs from and go to Actions > Export System Logs.

Exporting system logs from VMware vCenter Server

Exporting system logs from VMware vCenter Server

You can choose to export ESXi host log bundles and vSphere web client log files within the same bundle. With this information, VMware has everything they need to identify the problem. Within these subdirectories, vCenter Server logs are grouped by component and purpose. However, most of the time, you won’t need to know all of this.

All vCenter Server log files (from the VMware knowledge base article)

All vCenter Server log files (from the VMware knowledge base article)

If you’re using a vCenter Server Appliance (VCSA) as VMware recommends, you can also get the logs from a Secure Shell (SSH) session or a direct console session in case you have problems and using the vSphere Web Client isn’t a possibility. You’ll find the main vCenter Server log at vpxd/vpxd.log

VMware ESXi host log files

What if you don’t have vCenter and want to check the logs for a VMware ESXi host? For individual ESXi hosts vCenter Server doesn’t manage, you can get support logs via this procedure:

In your web browser, go to https://ESXI-IP-ADRESS/UI. Then go to Monitor > Logs > Generate log bundle.

You can do two things here:

  1. You can check different logs by selecting a log file and verifying the content in the lower pane.
  2. You can generate a log bundle.

Generate an ESXi log bundle

Generate an ESXi log bundle

Creating logs takes a few minutes, maybe five. Afterward, a small pop-up window appears telling you this:

Download an ESXi log bundle

Download an ESXi log bundle

The logs are quite voluminous; in my case they were like 155 MB in size. It’s pretty convenient and simple to have these logs on your C: drive, but then you’ll have to «ship» them to VMware. But again, it’s simple.

You simply have to connect to your space at the myVMware portal and go to the Get Support section. There you will have to Select an issue and upload a log file.

Getting support at VMware and uploading log files

Getting support at VMware and uploading log files

ESXi logs and locations

You can find different ESXi log locations and the meaning of each log in this table from a VMware knowledge base article.

VMware ESXi logs and their locations

VMware ESXi logs and their locations

As you can see, quite a few logs are stored on ESXi hosts. ESXi records host activity in log files by using syslog, a standard for message logging. It is not proprietary to VMware and ESXi.

Within your environment, you can configure a syslog server. Instead of jumping from one ESXi host to another to check the logs, it’s way better to install and configure a syslog server to pull the data from many ESXi hosts. Without a syslog server, you’d have to check through each ESXi host individually.

The syslog protocol supports a wide range of devices, and you can use it to log different types of events. For example, a router might send messages about users logging on to console sessions, while a web server might log access-denied events.

Syslog is a great way to consolidate logs from multiple sources into a single location, so you can configure each ESXi host to send its logs to a syslog server—to a single, central location.

You can download and test many free syslog servers. One of them for example is Kiwi Syslog Server from SolarWinds, but it is outside the scope of this post.

Whether you administer a small company’s network or an enterprise-grade network infrastructure, in the long term, you should definitely consider employing a syslog monitoring tool.

Subscribe to 4sysops newsletter!

Such a software tool would help you not only see what is happening on your network but also put yourself in a position to be proactive and able to react on events before a failure occurs. After all, it’s all about uptime.

Материал из Xgu.ru

Перейти к: навигация, поиск

В данной статье описывается местонахождение различных основных log-файлов в гипервизорах VMware ESX и VMware ESXi
версий с 3.5 по 4.x по материалам документации с сайта VMware[1]

Доступ к логам гипервизора можно получить следующими способами:

  • Непосредственно с консоли сервера (DCUI) через меню View System Logs

Vmware-logs0.png

  • Через клиента vSphere, подключенного непосредственно к гипервизору (не к vCenter!) через меню Administration — System Logs

Vmware-logs1.png

  • Используя шел PowerCLI[2]
 Get-VMHost HostNameOrIP | Get-Log -Bundle -DestinationPath c:StorageLocation
  • Через браузер, зайдя по адресу https://<ИмяХостаИлиIP>/host и введя логин root и нужный пароль, вы уведите ряд файлов, среди которых есть файлы с логами

Vmware-logs2.png

  • Выполнить экспорт логов через vSphere Client.

При подключении к ESX-хосту :

Vmware-logs3.png

При подключении к vCenter:

Vmware-logs4.png

  • Можно подключаться через SSH к хосту и прочитать (либо скачать) файлы:

В ESX серверах:

 /var/log/vmkernel – Core VMkernel logs, including device discovery, storage and networking device and driver events, and virtual machine startup.
 /var/log/vmkwarning – Summary of Warning and Alert log messages excerpted from the VMkernel logs.
 /var/log/vmksummary – Summary of ESX host startup and shutdown, and an hourly heartbeat with uptime, number of virtual machines running, and service console resource consumption.
 /var/log/messages – Messages from the Service Console Linux kernel, including service startup and shutdown.
 /var/log/vmware/hostd.log – Host management service logs, including virtual machine and host Tasks and Events, communication with the vSphere Client and vCenter Server vpxa agent, and SDK connections.
 /var/log/boot-logs/sysboot.log – Early VMkernel startup, module loading, and host initialization.
 /var/log/vmkiscsid.log – Software iSCSI Client logs.

В ESXi серверах:

 /var/log/vmware/hostd.log – ESXi Service Log
 /var/log/vmware/vpx/vpxa.log – vCenter Agent Logs
 /var/log/vmware/aam- Automatic Availability Manager (AAM) log
 /var/log/messages – VMkernel, hostd, vmkwarning
 /var/log/sysboot.log – System boot log 

[править] См. также

  • Cравнение ESX и ESXi

[править] Примечания

  1. ↑ Location of ESXi 3.5-4.1 log files http://kb.vmware.com/kb/1021801
  2. ↑ Collecting diagnostic information for VMware vCenter Server and ESX/ESXi using the vSphere PowerCLI http://kb.vmware.com/kb/1027932

VMwareLogo.jpg

vCenter Server Log files

  • vCenter Server 5.x on Windows Server 2003: %ALLUSERSPROFILE%Application DataVMwareVMware VirtualCenterLogs
  • vCenter Server 5.x on Windows Server 2008: %ALLUSERSPROFILE%VMwareVMware VirtualCenterLogs
  • vCenter Server 5.x Linux Virtual Appliance: /var/log/vmware/vpx/

With a Windows Server implementation of vCenter, browse to the log file location and open the log in your favorite text editor.

ESXi Log Files

  • /var/log/auth.log: ESXi Shell authentication success and failure attempts.
  • /var/log/dhclient.log: DHCP client log.
  • /var/log/esxupdate.log: ESXi patch and update installation logs.
  • /var/log/hostd.log: Host management service logs, including virtual machine and host Task and Events, communication with the vSphere Client and vCenter Server vpxa agent, and SDK connections.
  • /var/log/shell.log: ESXi Shell usage logs, including enable/disable and every command entered.
  • /var/log/boot.gz: A compressed file that contains boot log information and can be read using zcat /var/log/boot.gz|more.
  • /var/log/syslog.log: Management service initialization, watchdogs, scheduled tasks and DCUI use.
  • /var/log/usb.log: USB device arbitration events, such as discovery and pass-through to virtual machines.
  • /var/log/vob.log: VMkernel Observation events, similar to vob.component.event.
  • /var/log/vmkernel.log: Core VMkernel logs, including device discovery, storage and networking device and driver events, and virtual machine startup.
  • /var/log/vmkwarning.log: A summary of Warning and Alert log messages excerpted from the VMkernel logs.
  • /var/log/vmksummary.log: A summary of ESXi host startup and shutdown, and an hourly heartbeat with uptime, number of virtual machines running, and service resource consumption.

Ways to View vSphere Log Files

There are a number of ways in which you can view log files, depending on whether they are for vCenter or for an ESXi host. I’ll start by looking at ways in which you can view host log files. The first place is simply from the DCUI on the host. You can move down to ‘View System Logs’, then choose the log file that you would like to view:

dcui_log_files

The second way is to use the vSphere client. By making a connection directly to a host, rather than vSphere, you can view the hosts log files:

log_files

log_files2

If you are connected to vCenter rather than a host, you can browse to the same place, but instead of host logs, you can view the vCenter logs files.

Another way to view a host’s log files is to use a web browser. This is a method I always forget is available, but it definitely has its uses. Using a url like this one: https://192.168.0.235/host, will (after you have authenticated) present you with a web page from which you can access host log files:

log_files3

My preferred way, when possible, is to SSH to the host and view the log files from the SSH console. Once connected you can view logs in a variety of ways such as:

  • cat hostd.log | more
  • less hostd.log
  • tail hostd.log
  • tail -f hostd.log
  • zcat /var/log/boot.gz

You can also search logs using grep:

  • cat hostd.log | grep keyword

The last method of viewing host log files that I want to mention is by using PowerCLI. Having established a connection to my vCenter server, running Get-LogType displays the vCenter logs that are available to view:

log_files4

By running this against a host, rather than vcenter, we see the following:

log_files5

We can use the Get-Log cmdlet to view the log files:

(Get-Log -VMHost (Get-VMHost 192.168.0.235) vmkernel).Entries

log_files6

This will show everything in the vmkernel log, however if we are looking for something in particular we can filter the log using something like:

(Get-Log -VMHost (Get-VMHost 192.168.0.235) vmkernel).Entries | Where {$_ -like “*cpu*“}

Понравилась статья? Поделить с друзьями:
  • Evco контроллер ошибка pr2
  • Est 37 коды ошибок
  • Evaporative system ошибка
  • Essentials ошибка null
  • Ev3l21n7 ошибка p1