Q. Dear Rov, I heard that you are not allowed to touch a Megila without first washing hands, is that really true?
A. Shulchan Aruch (147: 1) rules that one should not touch a Sefer Torah with his bare hands. Remah (ibid) adds that there are opinions that the same applies to other manuscripts of Sifrei Kodesh and although the common Minhag is not to wash hands, one should be stringent when hands were not washed. However, regarding a Sefer Torah, even with washed hands one should not touch the Sefer itself.
Mishna Berura (4) explains that the same would apply to a Megila, since we assume that the usual situation of hands is that they are not kept clean. He quotes similar opinions from Panim Meiros and Hagr'a. Aruch Hashulchan is similarly stringent.
Horav Shlomo Miller's Shlit'a opinion is that there are different traditions, and although many are not careful to wash hands, the ones that follow the traditions of Gaon of Vilna, are indeed careful not touch the parchment of the Megila itself. He explained that this is the source of the Minhag to keep the Megila rolled inside a wooden container, that has also a wood attachment to the scroll, to be able to pull it out without touching it.
Rabbi A. Bartfeld as revised by Horav Dovid Pam, Horav Aharon Miller and Horav Chanoch Ehrentreu Shlit'a