Information theory studies the quantification of information, its storage and communication. We introduce concepts of information entropy and information gain that are used to construct a decision tree using ID3 algorithm.
Information theory
Information entropy
Information entropy of the given data measures the least amount of the information necessary to represent a data item from the given data. The unit of the information entropy is a familiar unit - a bit and a byte, a kilobyte, and so on. The lower the information entropy, the more regular the data is, the more pattern occurs in the data and thus less amount of the information is necessary to represent it. That is why compression tools on the computer can take large text...