John Lynch has already mentioned this paper on how the evolution of whale hearing unfolds in fossil record over on The Panda's Thumb, but it's fun stuff and I thought I'd discuss some of the data in a little more detail. In particular, take a look at this diagram of the structure of the ear to see how the changes occurred.

Sound transmission mechanisms in land mammals and whales. Diagram of the ear in a generalized land mammal (a), a pakicetid (b), a remingtonocetid/protocetid (c) and a modern odontocete (d). Abbreviations: Coc, cochlea; Dom, dome-shaped depression for periotic; EAM, external auditory meatus; FaPa, fat pad; Inc, incus; Inv, involucrum; Mal, malleus; Man, mandible; MeTy, medial synostosis between periotic and tympanic bone, in cetaceans this synostosis is absent and is homologous to a gap between these bones ("MeTy"); OvW, oval window; Per, periotic bone; PeTy, joint between periotic and tympanic; Sin, air-filled sinuses; Sk, skull; Sta, stapes; TyBo, tympanic bone; TyMe, tympanic membrane; TyPl, tympanic plate.

Start with the top left diagram. This is the ear of a typical modern land mammal. The horn shaped structure angling down towards 8 o'clock is the external auditory meatus (EAM), your ear hole, which leads to the dark gray oval, the tympanic membrane (TyMe), better known as the ear drum. Sound in air travels down the EAM to the TyMe, which vibrates. The vibrations are amplified the the chain of inner ear bones, the malleus (Mal), incus (Inc), and stapes (Sta), or hammer, anvil, and stirrup (I know—the terminology gets a little dense). Finally, the amplified vibrations are transmitted to the cochlea, where they are transduced into localized deflections of hair cells that trigger pitch-specific nerve impulses.

That's the path that works well in the air, but it doesn't work so well in water. Try immersing your head in the bathtub or swimming pool, though, and sounds are immediately dampened; the EAM fills with water that puts pressure on the eardrum, reducing the amplitude. Instead, the vibrations are transmitted through the bones and tissues of the head, vibrating the tympanic bone (TyBo) and by that path the inner ear bones.

The next three diagrams show the progression of changes in the whale lineage. The top right picture (b) is a pakicetid from about 50 million years ago. It's not much different from the generic land mammal, with an EAM, eardrum, etc., but note the one special feature: the tympanic bone isn't connected to the periotic bone (Per), and it's actually thickened into a structure called the involucrum. Basically, the bony structure of the ear is less tightly attached to the skull, and is more free to vibrate in response to sound transmitted through the tissue of the head.

The next step is seen in a group of whales called the remingtonocetid/protocetids, from 43-46 million years ago (c). The ear capsule is even less strongly attached to the skull, and the involucrum is more robust and even more remote from the skull—the whole thing is better at moving freely. The ear drum is reduced and conical in shape, and the malleus is fused to the bone, so although the pieces are all there, it's not going to be particularly effective at capturing sound waves in air. Another feature is a deep groove in the mandible that indicates that these animals had a fat pad (FaPa) in the jaw that would better transmit vibrations from the jaw bone to the ear capsule.

Last (d) is the ear structure of a modern whale. All of the trends of the previous organisms are accentuated: the ear capsule is specialized to receive sounds transmitted through the fat pad, and has completely given up on sounds transmitted through air—the external auditory meatus is closed off and gone, and while the eardrum is present, it's not connected to the external world.

The virtue of this paper is that it illustrates yet another unambiguous transitional serious, showing how over the course of less than seven million years we have a functional transformation of a specialized structure unique to an identifiable lineage.

Nummela S, Thewissen JGM, Bajpal S, Hussain ST, Kumar K (2004) Eocene evolution of whale hearing. Nature 430:776-778.