D&D has a wide variety of monsters, inspired by various legends, fictions, and real creatures. Some of these monsters are pretty scary, but there's still a lot of untapped potential in real animals - especially some of the weird extinct ones. D&D has gone into the past for some creatures, but mostly only the well-known ones - dinosaurs and mammoths being the prime examples.
But there's plenty of equally terrifying material in history, and some of it is even scarier and/or weirder than dinosaurs.
Dunkleosteus was a hypercarnivorous armoured fish (hypercarnivorous meaning it ate almost exclusively animals, as opposed to the many carnivores with a mixed diet). You don't typically see very many armoured predators - it's usually a prey defence. So there's scary feature number one.
Number two is its size: up to 33 feet. For comparison, the average full-size school bus is about 36 feet. So picture a hungry fish the size of a school bus - this thing could swallow you whole pretty easily. Of course, you'd probably be bitten in half first.
The really scary thing about dunkleosteus is the biomechanics of its jaw. Instead of teeth, it had sharp, heavy bone plates attached to its jaws - think guillotines instead of fangs. But that's not all - it could also open its jaw in 1/50th of a second, creating a powerful suction that draws prey into its mouth. Dunkleosteus could pull you right into its mouth simply by opening its jaws.
bone plates attached to its jaws - think guillotines instead of fangs. It had one of the most powerful bite forces in history, rivalled only by T. rex, crocodiles, and megalodon.
If that's not a terrifying sea monster, I don't know what is. I'd rather face an angry sea serpent than a hungry dunkleosteus.
...but since I'm currently a DM and not a player, here's a stat block!
Let's give it a cool fantasy name - the guillotine eel. This guy's very dangerous for a level 10 elite - the bite deals above-average damage, and it can pull its enemies into range for that big-damage bite. And the heavily-armoured skull comes into play as a damage-reducing trait: if an attack against its AC just barely hits, it'll deal half damage. You'll need a slightly higher roll to hit a vulnerable spot. Fortunately, though, its reflex and will are much lower than its AC, so if you can attack those you'll be much better off.
That's it for dunkleosteus. Anyone else have any terrifying prehistoric beasts they've used in a D&D game, or would like to see in a game?