PHP uses the Zend Engine's tokenizer at its core to parse and interpret PHP source files. Lucky for us, the tokenizer is also directly accessible via two functions in the language's top-level namespace: token_get_all() and token_get_name().

Tokenization consists of taking input text and breaking it up into meaningful segments. Each segment can optionally carry a label, explanation, or additional detail. Let's look at an example of PHP code being tokenized by the Zend Engine.

<?php
// get the contents of this file into a variable
$thisFile = file_get_contents(__FILE__);
// get the token stack
$tokenStack = token_get_all($thisFile);
$currentLine = 0;
// output each token & look up the corresponding name
foreach ($tokenStack as $token) {
// most tokens are arrays
if (is_array($token)) {
if ($currentLine < $token[2]) {
$currentLine++;
echo "Line $currentLine:\n";
}
echo "\t" . token_name($token[0]) . ': ' . rtrim($token[1]) . "\n";
// some tokens are just strings
} else {
echo "\tString: " . rtrim($token) . "\n";
}
}
?>

The above code snippet runs itself through the tokenizer, which results in the following output:

We formatted our output a littler nicer, but the first token essentially looks like this:

Array(
[0] => 367
[1] => <?php
[2] => 1
)

In this case, 367 is the value of the parser token, which corresponds to T_OPEN_TAG when we look it up with the token_name() function.<?php is the actual text of the token and 1 is the line number on which the token occurs. You can look up the complete list of tokenizer token constants in the online PHP manual, or the following code snippet will list the ones that are defined for your version of PHP.

<?php
// get all constants organized by category
$allTokens = get_defined_constants(true);
// we're only interested in tokenizer constants
print_r($allTokens["tokenizer"]);
?>

As you can see for yourself, tokens contain a lot of information that is useful to programmatically understand what an analyzed portion of code is doing. PHP_CodeSniffer builds upon the existing tokenization extension and built-in tokens by providing additional tokens to provide even finer granularity when examining PHP code.

Now that you know what tokens are, it will be much easier to understand what the individual sniffs are doing. First of all, a sniff registers with the main executable the tokens in which it is interested using the register() method. That way, the main code can hand over execution to the sniff's process() method whenever it encounters such a token. For example, a sniff trying to validate that a code file has the proper PHP opening and/or closing tag might register interest in the T_OPEN_TAG with the parent code. That is exactly what we're doing in the following listing:

<?php
// sniff class definition must implement the
// PHP_CodeSniffer_Sniff interface
class ProjectStandard_Sniffs_Syntax_FullPhpTagsSniff implements PHP_CodeSniffer_Sniff
{
// register for the tokens we're interested in
public function register()
{
return array(T_OPEN_TAG);
}
// process each occurrence of the token in this method
public function process(PHP_CodeSniffer_File $phpcsFile, $stackPtr)
{
$tokens = $phpcsFile->getTokens();
// warn if the opening PHP tag is not the first token in the file
if ($stackPtr != 0) {
$phpcsFile->addWarning('Nothing should precede the PHP open tag.', $stackPtr);
}
// error if full PHP open tag is not used
if ($tokens[$stackPtr]['content'] != '<?php') {
$phpcsFile->addError('Only full PHP opening tags are allowed.', $stackPtr);
}
// all files must have closing tag
if ($token[sizeof($tokens) - 1]['type'] != T_CLOSE_TAG) {
$phpcsFile->addError('All files must end with a closing PHP tag.', $stackPtr);
}
}
sniffssniffswriting}
?>

Let's take a closer look at the process() method, which takes two parameters. The first one is a reference to a PHP_CodeSniffer_File object, which we can use to access the token stack. The second argument is an array index to the current token in the stack.

Armed with that information, we can start validating the code via the token stack. First, we use the PHP_CodeSniffer_File's addWarning() method to display a warning message whenever the very first token is not the PHP open tag. Next, we use the addError() method to display an error message to the user whenever the opening tag doesn't match the string "<?php" since that is the only opening tag that our coding standard allows. Lastly, we display an error message if the last token in the stack is anything other than the closing PHP tag.

That's it. The main phpcs executable does the rest. It tokenizes the input file(s), calls all registered sniffs for each occurrence of their respective token, and displays nicely formatted output to the user.

You may have noticed in the above listing that we used the values of the token's 'content' and 'type' attributes. If you recall, the tokens returned by the standard PHP tokenizer did not have those attributes. Instead, PHP_CodeSniffer adds those and other attributes. Following is a list of token attributes that are always available. Depending on the type of token, additional attributes might be available. You should consult the PHP_CodeSniffer API documentation for details.

We have already seen that we can include sniffs from other coding standards in our own. However, we can take it a step further and make an existing sniff do all the work while still implementing our own standard. For example, the "Generic" coding standard includes a sniff to check for maximum line length. As it happens, the suggested maximum line length is 80 characters— the same as in our own standard. However, the absolute maximum line length is 100; whereas, our standard allows for up to 120 characters per line. Therefore, all we have to do is extend the existing sniff and overwrite the protected property $absoluteLineLimit as in the following listing.

<?php
if (class_exists('Generic_Sniffs_Files_LineLengthSniff', true) === false) {
throw new PHP_CodeSniffer_Exception('Class Generic_Sniffs_Files_LineLengthSniff not found');
}
// class to check line length in number of characters
// note: we're overwriting an existing sniff from the generic coding standard
class Zend_Sniffs_Files_LineLengthSniff extends Generic_Sniffs_Files_LineLengthSniff
{
// we generate an error when exceeding the absolute
// maximum line length
protected $absoluteLineLimit = 120;

}
?>

Even though PHP_CodeSniffer is available, there is no guarantee that individual developers will actually take advantage of it. However, in a team environment, the lead developer can take several steps to make sure the team members adhere to the chosen common standard. First, the code base should be scheduled for an automated check once a day during active development. A simple (insert you favorite scheduler utility here) job can process all the source files and send an email to everybody in the team.

However, it is possible to take things a step further. Assuming that you are using a source code control system, most of these systems provide hooks at various stages of checking out or committing source code. The most commonly used hook is the pre-commit hook. In other words, the source code control system executes any number of user-configurable steps before committing the code. The outcome of these steps impact whether the use is allowed to commit the code or not. In the case of our coding standard, we can configure the pre-commit hook to run any PHP source files being committed through PHP_CodeSniffer and only proceed if no errors and/or warnings are being generated. In essence, this is a way that your team only accepts contributions from individual developers if they adhere to the team's coding standard.

For a detailed example of how to configure the Subversion source code control system with a PHP_CodeSniffer pre-commit hook, please consult the chapter on source code and version control.