The NetScaler filesystem
The NetScaler code runs on top of FreeBSD as a userspace process, so it pays to understand the underlying file system structure.
Note
A question that comes up often, given the system is based on FreeBSD, is, "Is the system automatically vulnerable to any FreeBSD security issues that (as with any software systems) are reported by advisories?" The answer is, it depends; the NetScaler is a hardened appliance with several of its modules customized to reduce a potential security impact. For a definitive answer, you will need to contact Citrix Support, who would then work with a Security Response team for a validated response.
While the base layout will be familiar to anyone familiar with UNIX-based systems, the files that we would look at when troubleshooting are custom to the NetScaler.
Start by using df. This is also a great way to see how you are doing in terms of disk space:
The command df stands for diskfree, a Unix command to show disk usage statistics. By using the –ah option, we are asking for all the folders to be displayed in a human readable format, with percentages, for easy comprehension.
Let's take a look at the two important ones here for troubleshooting: /flash and /var.
/flash, as you've probably guessed, maps directly to the Flash drive/SSD installed in the NetScaler. This is the most important partition on the NetScaler as it contains the operating system along with the configuration, license, and essentially everything needed to boot the NetScaler.
The /var, which is the largest of partition and equals the hard disk on the NetScaler, contains: logs, crashes, traces, and other items that are to do with the maintenance and monitoring of the NetScaler.
In the case of a VPX, which is a virtual appliance with no physical drives, these folders become references to virtual partitions on the drive. Let's have a brief look at the important subfolders among these.
Folders on /flash
/flash contains the following folders:
/nsconfig/: This contains the NetScaler configuration files (ns.conf.*). Each time you make a configuration change, it does get applied but doesn't get committed to the disk. To commit changes you need to click on Save config. Five such files, each resulting from a "save config", are saved in the/nsconfig/folder. So, you can get back to a last known good configuration if you are in trouble after saving configuration changes.
The best practice of course is to not leave it to chance and use well named backup files. The current versions offer a handy way to do this: navigate to System | Backup/Restore, choose a file name, and select either Basic backup (configuration, location database) or Full (basic backup along with certificates). You can then download the backup.
Note
A copy of these backups is sent to the
/var/ns_sys_backup/folder.The
/nsconfig/folder is also home to other configuration files, most notably that of the routing engine ZebOS:/nsconfig/license: This contains the license files./nsconfig/ssl: This contains the SSL certificates, keys, and requests.ns-root.*andns-server.*: These files come by default; thens-root.*files are used for signing, while thens-server.*files are bound to the internal services, so care must be exercised with any folder cleanup here. Thens-servercertificate is what you are presented with when accessing the NetScaler, and consequently, this is the key-pair that you would change as a best practice with a signed certificate you trust./nsconfig/monitors: This is the folder for anyUSER(script-based) monitors that you upload to the NetScaler.
Note
Monitors provided as
Perlfiles are used when creating a monitor of the typeUSER. Going by the list in the following screenshot, you can guess that these are usually monitors that provide application knowledge beyond basic port or protocol response checks. In newer versions, the home for these files is/netscaler/monitors/;it's when you upload any with modifications that they are stored in/nsconfig/monitors.
Notice that all this while that we've been referring to this all important folder as
/nsconfig/and not/flash/nsconfig/- that's because/nsconfig/is a link to/flash/nsconfig/and they represent the same folder.
Folders on /var
/var/log contains text based logs. Let's look at some of the important ones:
ns.log: This is of paramount importance when troubleshooting, and as you will see during the course of this book, it is a file that we often turn to get a live view in easily readable messages in order to understand what is happening in the background, such as why is that User denied access, or why is the request blocked?Tip
A very handy way to examine this file is to run a tail
–f while you reproduce the issue. You might also find it useful, to demark the entries (I use a series of hyphens) before you begin reproducing the issue, to be able to spot with a bit more ease what you are looking for.messages.*: These are your standard Linux/FreeBSD system logs. This is where I would look first if I were to go back in time to understand when a reported reboot of the unit happened, or when a shell command was run.license.log: This is the log to look at if the licenses fail to apply, such as when thehostIDused during allocation is incorrect./var/nslog: This folder contains the binarynewnslogfiles. While thens.logfiles we just discussed are very easy to read by text and are our preferred go to when dealing in a controlled environment, we most certainly can't digest patterns in a heavily used production unit, and that's wherenewnslogscome in. This is a highly efficient form of logging that tracks the value of each counter (there's a few thousands of them) at every 7 second interval. These logs also serve as a basis for any external tool's plotting patterns, such as traffic, CPU, or errors.
This folder also contains the AppFirewall learning related logs:
/var/coreand/var/crash: This is where the crash files go./var/core: This contains any crashes related to the NetScaler software, and you will almost always have them labeled in the formatNSPPE-0x-xxxxwhere theNSPPEstands for the NetScaler packet engine, the firstxfor the packet engine number, and the rest for the PID; recall what we said about the Packet Engines running as User processes, therefore they will have PIDs./var/crash: This is where any core dumps by the Kernel will go.