The foundation stone for the harbour was laid in 1913. A literal translation from the Dutch/Afrikaans name "Kalkbaai" is "Lime Bay". This derives from the vast deposits of mussel shells found there, which early settlers burned to make lime for construction. Lime kilns to roast mussel shells are still found along the west coast.
Several famous caves (with names such as 'Boomslang Cave' named after the snake, 'Ronan's Well', after the Walter Scott novel Saint Ronan's Well, and 'Free Drinks Saloon') are located in the mountains above the village. They are of importance to speleologists because they have formed in sandstone. Large cave systems are not often found in this type of chemically unreactive rock.
Kalk Bay is also home to the tiny but locally famous surf spot named "Kalk Bay Reef". This is renowned for heavy barrels and the associated shallow reef. It is best surfed on a big south-easterly swell or a north west wind. In smaller swells low tide makes for better barrels. Southern right whales come here during whale watching seasons, and are often seen playing or resting very close to surfers or piers. Whale activity can often be seen from Boyes Drive, a road that connects the southern suburbs of Cape Town to Kalk Bay and suburbs further south on the Peninsula.