My copy of David MacKay's book "Information Theory, Inference and Learning Algorithms" arrived yesterday and I started reading it last night. It looks absolutely fascinating.
I first came across Professor MacKay when I read his other book, "Sustainable Energy Without the Hot Air" and I was struck by how lucid, readable and entertaining it is. There is every reason to expect that his information theory book will be at least as readable.
Why get a book on "Information Theory, Inference and Learning Algorithms"? The main reason is because it should be very useful for my PhD in smart meter disaggregation. One of the god-fathers of disaggregation, George Hart, wrote the following in Hart 1992:
It is insightful to consider the [disaggregation problem] in the context of a communication model. Appliances can be thought of as “transmitters”, inadvertently broadcasting information as a by-product of their operation. The communication “channel” here is the house wiring. Any of the many signatures... may be the “codes” used in this communication scheme. Our task is to design a “receiver” for these codes which can decode them in terms of appliance state-change “messages”.
In other words, the disaggregation problem can be considered in an information-theoretic framework. This conceptual step allows us to take advantage of the tools developed in communication technology (i.e. coding theory).
Also, for a while now I've thought of information theory as one of those "sexy but mysterious" things which I'd love to learn more about. I read "Decoding Reality: The Universe as Quantum Information" a couple of years ago and found it fascinating (although I only understood about half the content of the book).
So, hopefully MacKay's information theory book will be a great self-study book.