Palliser Bay in the Wairarapa. Along the coastline there are many exposures of mudstone from the Hurupi Formation, about 11 to 8 million years old.
These mudstones contain abundant marine shell fossils, but are also known for occasional whale bones. After some time searching, as luck would have it, I found a large piece of bone sticking out of the mud near the base of one of the cliffs.
For an update on additional whale bone discoveries from this locality check out this blog post.