Around 1852, the mansion was built in the Italianate style as an early Victorian villa. From 1852 until 1969, the mansion was home to the families of Isaac Kirtland, Benjamin Babb, James C. Neely, Daniel Grant, and Barton Lee Mallory.