Originally owned by the Vanderbilt family, Rough Point eventually fell into the hands ... plus two gardens and bay views. This mansion may not be as grand or as well-known as other area properties ...