Book Image

Troubleshooting vSphere Storage

By : Mike Preston
Book Image

Troubleshooting vSphere Storage

By: Mike Preston

Overview of this book

Virtualization has created a new role within IT departments everywhere; the vSphere administrator. vSphere administrators have long been managing more than just the hypervisor, they have quickly had to adapt to become a ‘jack of all trades' in organizations. More and more tier 1 workloads are being virtualized, making the infrastructure underneath them all that more important. Due to this, along with the holistic nature of vSphere, administrators are forced to have the know-how on what to do when problems occur.This practical, easy-to-understand guide will give the vSphere administrator the knowledge and skill set they need in order to identify, troubleshoot, and solve issues that relate to storage visibility, storage performance, and storage capacity in a vSphere environment.This book will first give you the fundamental background knowledge of storage and virtualization. From there, you will explore the tools and techniques that you can use to troubleshoot common storage issues in today's data centers. You will learn the steps to take when storage seems slow, or there is limited availability of storage. The book will go over the most common storage transport such as Fibre Channel, iSCSI, and NFS, and explain what to do when you can't see your storage, where to look when your storage is experiencing performance issues, and how to react when you reach capacity. You will also learn about the tools that ESXi contains to help you with this, and how to identify key issues within the many vSphere logfiles.
Table of Contents (16 chapters)
Troubleshooting vSphere Storage
Credits
About the Author
Acknowledgment
About the Reviewers
www.PacktPub.com
Preface
Index

You cannot connect to your iSCSI storage


The following are some questions and tasks that you can partake in to troubleshoot iSCSI connection failures.

Have all the network requirements for your iSCSI connection been met?

Tip

You can refer Chapter 3, Troubleshooting Storage Visibility, for more information on this topic.

You need to consider the following points when checking network requirements:

  • The ESXi host needs to have the proper networking configuration if using the software iSCSI initiator. A VMkernel port must be set up on the same subnet as your iSCSI storage array with the proper IP addressing and subnet mask.

  • Ensure all DNS and routing information is accurate using the following command:

    esxcfg-route -l
    
  • If you are using VLANs, be sure that proper VLAN configuration has been implemented.

  • Has the iSCSI storage array's IP address been entered in either the dynamic or static discovery tab?

  • Is it possible to ping your storage array?

    vmkping <IP OF ARRAY>
    
  • Have all the proper ports (3260) been opened up for iSCSI communication? To test port connectivity from the ESXi host to your iSCSI storage array, use the following command:

    nc -z <IP OF ARRAY> 3260
    
  • Are there any firewalls in between your ESXi host and your iSCSI array? If so, ensure that the proper ports have been opened. To check the local firewall on ESXi, use the following command:

    esxcli network firewall ruleset list | grep iSCSI
    
  • Does your network utilize Jumbo Frames? If so, ensure Jumbo Frame configuration has been enabled on all devices the iSCSI packet will traverse. To ensure our vSwitch is configured with Jumbo Frames, use the following command:

    esxcli network vswitch standard list 
    
  • To check that we properly have Jumbo Frames setup throughout the network stack, using the default 9000 MTU size, run the following command on an ESXi host:

    vmkping –s 8972 –d <IP OF ARRAY>
    

Various things to check on ESXi

Tip

You can refer Chapter 1, Understanding vSphere Storage Concepts and Methodologies, and Chapter 3, Troubleshooting Storage Visibility, for more information on this topic.

The following items need to be checked on ESXi when troubleshooting iSCSI connection failures:

  • Ensure that no claim rules have been added to the ESXi runtime utilizing the MASK_PATH plug-in. The following command will list your claim rules:

    esxcli storage core claimrule list
    
  • Ensure that LUN ID is set to a number below 255. Also, check the Disk.MaxLun advanced setting ensuring that your LUN ID isn't higher than the value configured.

Have all CHAP requirements been met?

Tip

You can refer Chapter 3, Troubleshooting Storage Visibility, for more information on this topic.

The following items need to be checked for CHAP when troubleshooting iSCSI connection failures:

  • Double check your CHAP settings

    • Ensure you have entered your CHAP user ID and secret properly in both the iSCSI array and ESXi

    • If using bidirectional CHAP, the CHAP secret must be different from the mutual CHAP secret

  • Double check your target settings

    • By default when you enable CHAP, it is enabled on all iSCSI targets since it is inherited. If using CHAP on some targets and not others, be sure to check your CHAP settings on individual targets and if not required, override the Inherit from parent option.

  • See Further check the logs for iSCSI-related errors and Appendix C, iSCSI Error Codes, to determine if there are any further CHAP related errors occurring

Has there been any advanced settings dealing with iSCSI incorrectly configured?

There are a variety of advanced settings that may affect your connection to your iSCSI storage array:

  • LoginTimeout: By default, it is set to 5 seconds. It specifies that time in seconds that an initiator will wait for login response. If this is set to low or we are experiencing contention, we may need to increase this parameter. The recommended value is 5.

  • RecoveryTimeout: By default, it is set to 10 seconds. It specifies the time in seconds that can elapse while a session recovery is performed. If the recovery hasn't completed within this time, the initiator will terminate the session. If this value is set too low, we could be experiencing path thrashing. If set too high, we could be waiting too long to fail a path. The recommended Value is 10.

  • Header and Data Digests: It is set to Prohibited by default. It can be enabled to ensure data integrity. For troubleshooting, it is recommended to be to Prohibited. The recommended value is Prohibited.

Further check the logs for iSCSI-related errors

Tip

You can refer Chapter 3, Troubleshooting Storage Visibility, for more information on this topic.

The following items need to be considered when checking logs for troubleshooting iSCSI connection failures:

  • We can further view any iSCSI-related errors in the logs by filtering the entire log folder with the following command:

    grep –r iscsid /var/log/* | less
    
  • Refer Appendix C, iSCSI Error Codes, to interpret any reason codes returned

  • Verbose logging can be enabled for troubleshooting purposes by running the following command:

    vmkiscsid -x "insert into internal (key,value) VALUES ('option.LogLevel','999');"