It is the oldest European building in sub-Saharan Africa and was ... The castle is located in a picturesque fishing town along Ghana’s coast, which makes it a great place to spend some time.