The most natural modes of gestures are sadly infeasible as far as normal software is concerned. Humans express themselves best when they are engaged in multimodal communication, as in speaking aided by gestures and facial expressions. There has been quite a bit of progress made in speech recognition and detection of emotions, but making use of the gestural information that people convey in everyday speech requires some creative interpretation and understanding of human languages that is beyond the reach of applications today.
A similar problem arises with using sign language, another very natural human-oriented mode of gestural communication. Signing, like speech, is multimodal; mouthing words along with the hand signs in order to accurately and expressively impart meaning to the symbols used to express one's thoughts. It is, likewise, not terribly useful for directing computers due to the fact that it requires a great deal of contextual information and suffers from the same...