"Canst thou draw out leviathan with a hook? or his tongue with a cord which thou lettest down?
Canst thou put a hook into his nose? or bore his jaw through with a thorn?" -- Job 41:1-2
"Under the sea,
Under the sea,
Darling it's better
Down where it's wetter;
Take it from me!
Up on the shore they work all day,
Out in the sun they slave away
While we devotin'
Full time to floatin'
Under the sea"
--from "Under the Sea" Alan Menken & Samuel E. Wright, 1989, The Little Mermaid (Walt Disney Productions)
BIG QUESTION: What does the fossil record tell us about the origin of whales?
What Makes a Whale a Whale?
Cetacea: the clade of whales, also containing animals popularly called "dolphins" and "porpoises" (which are just small whales.) They are the most specialized marine mammals, incapable of moving on land (or even of breathing effectively without being buoyed by water). Living whales fall into two major clades:
Much like birds, their anatomy is so transformed relative to other living animals that it is difficult to tell at first where they belong on the family tree. They are indeed mammals, as they suckle their young with milk, have a limited amount of hair, have a four-chambered heart, are endothermic, have single lower jaw bone, etc. However, they are of course very different from other mammals: their skin is mostly hairless; their nostrils have relocated to the tops of their heads (blowholes); even toothed whales only get one set of teeth (not two, as in other mammals); their forelimbs have been transformed into flippers; their hind limbs are extraordinarily vestigial, so that (with the very, very, very rare exception now and then) they do not emerge from the body; their tails end in horizontal soft-tissue flukes, etc. So unravelling their evolutionary history is an interesting paleontological story.
But first, let's consider WHY perfectly good amniotes might want to return to the seas.
There are many groups of extinct marine amniotes: mesosaurid reptiles in the Permian; a whole host of diapsid forms in the Triassic; the pelagic ichthyosaurs, plesiosaurs, and mosasaurs during various parts of the Mesozoic; and so on. In the Cenozoic there were sea sloths and odd desmostylians. So the return to the sea is obviously a successful approach.
All marine amniotes share similar sets of problems:
Given all these problems to be overcome, why even go back to the sea?
A Royal Reptile? Discovery of Stem Whales
Whale fossils have been found throughout history, but the recognition of fossils of animals ancestral to whales took longer. Nevertheless, they were some of the first important paleontological discoveries in the New World. European colonists of what is now the American South (and the native peoples before them) regularly find bones and unusual teeth in abundance. In the early 1800s various officials from southern states sent some of these fossils to the American Philosophical Society in Philadelphia (at the time the premier research institution of the young USA). Scientist Richard Harlan interpreted the teeth as some sort of strange shellfish, but the vertebrae as coming from a giant marine reptile. Thinking it was from a plesiosaur-like marine reptile, he compared these huge vertebrae to those smaller Jurassic ones. In 1834 he named his huge giant Basilosaurus, the "king reptile", and estimated it "could be no less than from eighty to one hundred feet long".
When Sir Richard Owen was given some Basilosaurus specimens to study, he recognized that the teeth were in fact the teeth of this animal (correct), that it was an early primitive whale (correct), and that it was a herbivore (incorrect). (He also tried to give it the new name "Zeuglodon", in blatant violation of the principle of priority! Unfortunately, his preeminence in the field meant that for generations afterwards people incorrectly called this important genus by Owen's name rather than Harlan's).
Later discoveries in the US and Egypt (and elsewhere) gave paleontologists a good understanding of almost the entire skeleton, including the pelvis (not attached to the vertebra) and the femur (thigh bone). Unlike modern whales the teeth at the front of the snout were very different from the premolars and molars, the nostrils are halfway up the snout (unlike in the front of the snout in land mammals, and on top of the head in modern whales) and the body was long and eel-like (although not as eel-like as shown in this picture.)
Other primitive whales were found in the intervening decades, but these were either clearly mysicetes or odontocetes or closely related to Basilosaurus.
Kellogg basically came up with the answer "none of the above", shrugged his shoulders, and said "we don't know". But it turned out that some biologists and paleontologists had found some good evidence of a whale-artiodactyl connection. Ernst Haeckel united whales (including sirenians!) with hippos in a group he called "Obesa" (the "fat ones"), which was itself the sister group to the rest of Artiodactyla. W.H. Flower noticed aspects of the complex stomach and other aspects of the soft tissue uniquely shared between whales and artiodactyls. But these ideas were largely ignored in the 20th Century.
In the 1960s paleontologist Leigh Van Valen proposed that whales had their origins near or among the various groups of carnivorous hoofed mammals of the early Cenozoic: mesonychids, arctocyonids, and the like. The skulls definitely looked like Basilosaurus and other basal whales.
Starting in 1990s, two different fields of research converged on the same answer:
So paleontologists and neontologists alike were convinced that whales are a kind of artiodactyl. And additionally, a great number of new fossils now documented the transition from land animals to protowhales, and from basal whales to the early members of the mysticetes and of the odontocetes.
The latest information puts the sister group to whippomorphs as the entelodonts (also known as "terminator pigs"), large omnivorous-to-carnivorous artiodactyls. (Some of the fossils that Van Valen once thought was related to whale origins turns out to be an entelodont.) Here is a summary of the position of whales among their relatives:
And here is a summary figure of the major transitions in cetacean phylogeny:
(From McGowen et al. 2014 (TREE 29:336-346. doi: 10.1016/j.tree.2014.04.001)
The most primitive stem-whales are Indohyus and the other members of Raoellidae. These Eocene forms are small quadrupedal artiodactyls. They were vaguely deer-like, but share with whales a long heavy tail and ear bones modified to hearing noise underwater. Like the early whales, it is from regions around the ancient Tethys Sea. Isotopic analyses indicate that it mostly lived in the water, but fed from material on land.
The next phase up are the Early Eocene Pakicetidae, such as Pakicetus. In modern taxonomy, this is the base of Cetacea (raoellids plus cetaceans form the Cetaceamorpha). It was a perfectly good walker, but its ears are even better adapted to water hearing than raoellids. Based on its teeth and isotopes, it ate mostly fish and lived in fresh water.
The Early Eocene Ambulocetus is the next known phase. It was larger, but still could walk on land. It fed on land mammals and lived in freshwater (basically a mammalian crocodile). Swimming was by vertical undulation.
The Remingtonocetidae (such as Kutchicetus) are Early-to-Middle Eocene whales in which the legs are reduced. They had a diet of fish, and were fully marine but lived near shore. Their small legs indicate they lived mostly at sea; they swam by vertical undulations of the body and the long tail. It is not known, however, if their tail had flukes or was simply flattened.
The "protocetids" are the paraphyletic grade of Early-to-Middle Eocene coastal marine whales. Some (at least) were still capable of coming out onto land: in fact, a specimen of Maiacetus was found with a late term embryo, showing they were born face-first like land mammals and pinnipeds.. Their nostrils were still at the ends of the snout. Georgiacetus and Rodhocetus show reduced contact of the pelvis to the vertebrae, showing that their legs could not support much weight.
Basilosauridae are a more derived clade of basal whales in which the hindlimbs are detached from the vertebrae, meaning that these were fully marine animals incapable of coming out onto land. Late Eocene Dorudon is an example of a primitive basilosaurid; similar-aged large Cynthiacetus and gigantic Basilosaurus is the most derived. Basilosauridae is the sister taxon to Neoceti (the crown-group whales), together forming the Pelagiceti ("open sea whales"), an apt name for a group of specialized animals which lived far from land.
Note that all the above whale groups (collectively referred to under the paraphyletic term "archaeocetes") occur during the Eocene. This time saw the entire transition from fully-terrestial to fully-marine.
There is still some missing stages between Basilosauridae and Neoceti: we don't know, for instance, if hind limbs became entirely internalized in the concenstor of all neocetes, or if stem-odontocetes and/or stem-odontocetes still had little legs sticking out of their bodies. (Newly-discovered Mystacodon, oldest and most primitive mysticete, has hips that suggest a possible hindlimb: however, the limb itself isn't actually known yet.) Mysticeti is known from the Late Eocene onward; while by-definition Odontoceti must exist then, the oldest known odontocetes are from the mid-Oligocene.
There is a huge diversification of neocetes during the Oligocene; this coincides with the cooling of the oceans as a result of the separation of Antarctica from South America and Australia and the rise of the circum-Antarctic current. (There are similar diversification in other marine groups: almost certainly the cooling seas resulted in an increase in nutrients, and consequently more food in the oceans).
Neocete phylogeny is very well-studied, and there are abundant fossil species known:
(From Geisler et al. 2011. (BMC Evolutionary Biology 11:112. doi: 0.1186/1471-2148-11-112)
(And in all honestly, the number of fossil neocetes named since 2011 has been very high and growing!!)
The stem-odonotocete Simocetus of 32 Ma shows some evidence of space for the 'melon' (the receiver for their echolocation signals), but it would have been poorly developed. Later forms like 28 Ma Cotylocara definitely had good echolocation.
A recently-discovered giant odontocete was the stem-sperm whale Livyatan, nearly as large as Physeter but with massive teeth in the upper and lower jaws: it appears to have hunted other whales.
A series of new discoveries has shown the evolution of baleen. Eocene mysticetes (such as the oldest and basalmost form Mystacodon and more derived Llanocetus had a long snout and basilosaurid-style teeth. Oligocene mammalodontids still had stout teeth, used by Janjucetus to hunt large fish and by others like Mammalodon to suction feed on the sea floor. (In fact, a very recent study suggests based on the sharpness of their teeth that mammalodonts did not use dental suction or filtration; however, Mammalodon itself was not studied.) In Aetiocetus small teeth are present, as are grooves associated with baleen: probably fed on small fish. More derived whales like Parietobalaena were toothless and fully baleened. The precise nature of the ancestral filter feeders (dental filtration, suction feeding, etc.) remains unresolved.
These fossils show the full transition from eating large objects to bulk-feeding of smaller objects to bulk filter-feeding.
Although gigantic size was a trait highlighted for mysticetes, that is actually a fairly new innovation. New research shows that mysticetes bigger than 12-13 m did not evolve until the onset of the Quaternary glaciation. This is likely due to regional increases in prey density (due to stronger upwelling events from stronger winds) compared to the more averaged conditions of warmer seas. (Today, the largest mysticetes are great migrators, going from hemisphere-to-hemisphere chasing the plankton blooms.)
Whales have been an extraordinary success in the history of life. But in the last few centuries increasing technologies have allowed humans to hunt whales on ever-more industrial scales. This nearly brought some species to an end. Whales, as large placental mammals, have long gestation periods and so it takes decades (or centuries) to replenish their populations. Towards that end, there are a number of international organizations dedicated to cetacean conservation, so that Cetacea does not come to an end at our hands (directly or indirectly).