Edward Carter was born on August 1, 1796 in Jefferson, West Virginia (then Virginia) to parents Robert and Elizabeth Griggs Carter. He was the first son for the couple, but the fifth of what would eventually be ten children. It is believed that when he was around 15 years old, Edward moved to Bourbon County, Kentucky. He first married his second cousin Sally Timberlake on April 21, 1825 in Jefferson County, West Virginia and together the couple lived in Bourbon County, Kentucky where their daughter Sarah Timberlake Carter was born. Her mother would not live long after her daughters birth, passing away on December 17, 1827.

Edward married for a second time to Elizabeth Stanhope on April 23, 1830 in Fayette County, Kentucky. They had nine children together, including Fanny Carter Headley and Alfred S. Carter. Edward purchased the estate known as Chaumiere des Prairies in Jessamine County, Kentucky and built a new house, attached to part of the original home, on the property. Edward seems to have been very concerned about his children's education, as he not only sent oldest daughter Fanny to a finishing school in Lexington, but also hired a tutor to teach his younger children at home, and helped to found a school in Jessamine County. Edward passed away on August 11, 1851 at the age of 55 and is buried in the Lexington Cemetery.

Documents