This type of kernel rootkit uses the Intel Debug registers as a means to hijack the control flow. A great Phrack paper was written by halfdead on this technique. It is available here:
http://phrack.org/issues/65/8.html.
This technique is often hailed as ultra-stealth because it requires no modification of sys_call_table. Once again, however, there are ways of detecting this type of infection as well.
In many rootkit implementations, sys_call_table and other common infection points do go unmodified, but the int1 handler does not. The call instruction to the do_debug function gets patched to call an alternative do_debug function, as shown in the phrack paper linked earlier. Therefore, detecting this type of rootkit is often as simple as disassembling the int1 handler and looking at the offset of the call do_debug instruction, as follows:
target_address = address_of_call + offset + 5
If target_address has the same value as the do_debug address found in...