You should always configure your log files properly. Log files are the best way to get invaluable information in the detection of a problem and in troubleshooting issues. In the upcoming sections, you will find a comprehensive reference about your VMware vCenter and vSphere host infrastructure components.
When you troubleshoot different issues within the virtual environment, vSphere log files become the most important troubleshooting tool one can have. Not only can you use the log files to find and fix the problems, but you can also use them to avoid these problems from occurring in the future.
Logs for vSphere host 5.1 or later are grouped according to the source component (for more detailed information, please visit the VMware Knowledgebase):
When vSphere host 5.1/5.5/6.0 is managed by vCenter Server 5.1, 5.5, and 6.0, two components are installed, each with their own logs, as described in the following table:
|
vsphere related vcenter Log Files |
Description |
|---|---|
|
|
vCenter Server vpxa agent logs, including communication with vCenter Server and the Host Management |
|
|
vSphere host high availability logs, produced by the |
The vCenter Server logs are placed in a different directory on disk depending on the vCenter Server version and the deployed platform. Sometimes, logs are pointed to store in a drive other than the system drive. As each log grows, it is rotated over a series of numbered component-nnn.log files. On some platforms, the rotated logs are compressed.
The vCenter inventory service logs are placed in a different directory on a disk depending on the vCenter Server version and the deployed platform. For vCenter Server 5.x and earlier versions on Windows XP, 2000, 2003:
%ALLUSERSPROFILE%\Application Data\VMware\Infrastructure\Inventory Service\Logs
The default location for vCenter Server 5.x Linux Virtual Appliance is /var/log/vmware/vpx/inventoryservice.
Note
If the vCenter Server inventory service is running under a specific user, the logs may be located in the profile directory of that user instead of %ALLUSERSPROFILE%.
As each log grows, it is rotated over a series of numbered component-nnn.log files. On some platforms, the rotated logs are compressed.
To collect the vSphere 5.1 vCenter Server inventory service logs, navigate to Start | All Programs | VMware | Generate Inventory Service log bundle.
Following are the log file locations for vCenter Server 5.x and earlier versions on Windows Vista, 7, 2008:
|
vcenter Log Files |
Description |
|---|---|
|
|
The main vCenter Inventory Service logs, consisting of all vCenter Server and Single Sign-On connections, internal tasks and events, and information about the xDB |
|
|
Information about the installation of Inventory Service including computer name, operating system revision, the date of installation, and the number of revisions that have been installed or upgraded on the system |
|
|
Information about the status of the Java runtime environment |
The vSphere Profile-Driven Storage logs are placed in a different directory on disk depending on the vCenter Server version and the deployed platform. vCenter Server 5.x and earlier versions on Windows XP, 2000, 2003:
%ALLUSERSPROFILE%\Application Data\VMware\Infrastructure\Profile-Driven Storage\Logs
The default location for vCenter Server 5.x Linux Virtual Appliance is /var/log/vmware/vpx/sps.
Note
If the service is running under a specific user, the logs may be located in the profile directory of that user instead of %ALLUSERSPROFILE%.
vSphere Profile-Driven Storage logs are grouped by component and purpose. As each log grows, it is rotated over a series of numbered component-nnn.log files. On some platforms, the rotated logs are compressed.
You cannot view vSphere Profile-Driven Storage logs using vSphere Client or vSphere Web Client. To export these logs, see Collecting Diagnostic Information in the upcoming part of the chapter.
vCenter Server 5.x and earlier versions on Windows Vista, 7, 2008:
|
vcenter Log Files |
Description |
|---|---|
|
|
The main Profile-Driven Storage logs, consisting of all vCenter Server and Management Webservices connections, internal tasks and events, and information about the storage profile integrity |
|
|
Information about the installation of Profile-Driven Storage including computer name, operating system revision, the date of installation, and the number of revisions that have been installed or upgraded on the system |
|
|
Information about the state of the Java runtime environment |
There are different ways to collect logs from vCenter Server and vSphere hosts. You should configure logs using CLI or vMA, or you can use Host Profiles to configure syslog functionality for a cluster or for similar hosts. (We are not covering Setting up Host Profiles to enable logging). You can use the following ways to gather all the diagnostic information from your VMware infrastructure:
vSphere Client
vSphere Web Client
PowerCLI
vm-support
vm-support from vMA
vSphere host 5.x diagnostic information can be gathered using the vSphere Client connected to the VSphere host or to vCenter Server. To gather diagnostic data using the VMware vSphere Client, follow these steps:
Open the vSphere Client and connect to vCenter Server or directly to an vSphere host 5.x.
Log in using an account with administrative privileges or with the
Global.Diagnosticspermission.Select a vSphere host, cluster, or datacenter in the inventory.
Navigate to File | Export | Export System Logs.
If a group of vSphere hosts is available in the selected context, select the host or group of hosts from the source list.
Click Next.
In the System Logs pane, select the components for which the diagnostic information must be obtained. To collect diagnostic information for all the components, click Select All.
If required, select the Gather performance data option and specify a duration and interval.
Specify the download location in the next screen to download the logs, and then click Next, and on the last screen click Finish to finish the wizard. On the last screen, you will be presented with a summary of logs you are going to download.
Once the wizard completes collecting logs, it will store them in the location you have specified. The log bundle is named with the current date and time, for example,
[email protected].
You can gather the diagnostic information of vSphere host 5.0 and higher version by using the vSphere Web Client. The following steps need to be performed to collect diagnostic data:
Log in to the vSphere Web Client using your credentials.
Click on the Hosts and Cluster.
Select a vSphere host, cluster, or datacenter you want to collect the diagnostic information.
Click on Actions.
Choose All vCenter Actions and then click on the Export System Logs.
Click Next.
You can select the different components of diagnostic logs to be exported in the System Logs pane. You can choose Select All if you want to export all the logs.
You can also select the Gather performance data option and specify a duration and interval.
Then click on Generate Log Bundle and then Download Log Bundle.
VMware has provided a useful tool in order to collect diagnostics information and troubleshooting problems. It is a script-based tool that can also collect the state information of the virtual machines. You can run the vm-support in vSphere hosts or from the vMA appliance. Some of the logs that can be collected using vm-support are vmkernel, host, CIM, virtual machines, security, vpxa, cronjobs, dmesg, update logs, configuration information of the NICs, switches, storage adapters, NAS mounts, multi-path setup.
Tip
Follow the link for more information http://www.vmware.com/files/pdf/techpaper/VMware-Customizing-vm-support-vSphere50.pdf.
Let's use vm-support tool to generate log bundle in VSphere hosts:
Log in to the vSphere host using a Secure Shell Session (SSH).
Type the following command and hit Enter:
vm-supportThe
vm-supportcommand will generate a compressed bundle of logs and will save it in a file with a.tgzextension in one of the following locations:/var/tmp//var/log/The current working directory
You can also use vm-support to output logs on standard output stream. For example, you can get all the logs over an SSH connection without saving anything locally on the host.
From a Linux machine, log in to the vSphere host and type the following command:
ssh root@vSphereHostname Or IPAddress vm-support -s > logbundle.tgzThe
–sflag tells vm-support utility to stream the logs onSTDOUT, following which you can specify the path and the file name you would like to save your logs in. The vm-support utility will generate the logs and store everything into the specified file.Let's do that to generate log bundle and store into a data store:
ssh root@vSphereHostnameOrIPAddress 'vm-support -s > /vmfs/volumes/datastore001/logbundle.tgz'
VMware has provided a nicely written BASH script to collect vSphere host logs from the vMA appliance. You can download the script from the following link: http://goo.gl/e1R4Zu.
The script uses vi-logger that is now deprecated to collect the logs from vSphere hosts.
Follow these steps to collect the logs from vMA 4.0 or vMA 4.1:
Once
1035911_vMA-vilogger-gatherer.txtis downloaded to your vMA appliance, rename it aslog_collector.shby typing the following command:mv 1035911_vMA-vilogger-gatherer.sh log_collector.shGive it executable permission:
chmod +x log_collector.shRun this command to execute the script:
/vMA-vilogger-gatherer.shAn archive of log files is created within the current folder.
We can also collect logs from vMA without using the preceding script:
For the 4.x version, please visit http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1024122to configure vMA to collect logs.
PowerCLI is a very powerful and easy-to-use command-line tool to manage vSphere infrastructure. It lets you have complete control by automating, monitoring, and managing your vSphere infrastructure.
We will now use PowerCLI to collect diagnostic log bundles from vCenter Server. It is very simple and a one-line command to execute. Run the following in your PowerCLI shell:
Get-Log -Bundle -DestinationPath D:\vCenter_logs\logs
The output appears similar to the following:
Data ---- D:\vCenter_logs\logs\vc-support-nnnnnnnn-nnnn-nnnn-nnnn-nnnnnnnnnnnn.tgz
Let's download a vm-support diagnostic log bundle from a vSphere host managed by vCenter Server. Enter the following command:
Get-VMHost HostNameOrIP | Get-Log -Bundle -DestinationPath D:\vCenter_logs\logs
The output appears similar to the following:
Data ---- D:\vCenter_logs\logs\vm-support-nnnnnnnn-nnnn-nnnn-nnnn-nnnnnnnnnnnn.tgz
You can access the log browser directly from the VMware vSphere 5.1 web. The client.Log browser is a plugin that works directly with vSphere 5.1. The log browser can be very beneficial and easy-to-use tool when you are troubleshooting problems. Following are some of the benefits of log browser:
Log comparison
Searching logs
Finding and highlighting keywords
The log browser offers a friendly GUI that you can use to take a snapshot of specified host/vCenter logs. Once you retrieve the logs, you can search, compare, highlight, and categorize the logs based on some specific keywords. You can refresh the snapshots of retrieved and stored logs. The search term then can be highlighted for fast navigation. The admin gets a fast UI with the possibility to have the searched word to appear in color. Once you are done, you can export the logs to a file or as a VMware log bundle.
In the troubleshooting lifecycle of VMware infrastructure, log browser is a very handy and useful tool that you can directly access from the vSphere Client. It installs when you install vSphere, and you do not need to look for any other methods to collect the logs. The only problem is that the log browser is only available in vSphere 5.1.
Let's have a quick walkthrough of the log browser:
Log in to your vSphere 5.1 web client using administrator credentials.
On the left pane, click on the Log Browser option and in the View pane on your right, click on the Select Object option. This will open up a new window from where you can either choose vSphere hosts or any vCenter in your environment.
Once the host is selected, you can choose the log file you would like to view from the Type dropdown menu.
From here, you can browse different log files from different objects.
You can export log files using the log browser by performing the following steps:
Navigate to the log browser and select an object to browse.
Select Action and then click on Export.
Select the type of file that you would like to export.
Once it is done, click on Export. When a new browser window appears, browse a location to save the log bundle.