Originally known as 'Lords Hill', in 1807 the Asylum for the Education and Instruction of Deaf and Dumb Persons was founded, and for one hundred years, provided services to the deaf. The institution relocated to its present home in West Hartford, whereupon it became the American School for the Deaf. The establishment of the school gave the area its name.

A commemorative statue located at the intersection of Farmington and Asylum Avenues of a young girl standing on oversized open hands represents Alice Cogswell, the first pupil at the school. Designed by Frances Wadsworth, the statue honors Dr. Thomas Hopkins Gallaudet, founder of the school, which was the first in the country.

The Asylum Hill Congregational Church was organized in 1864 by a group of wealthy residents in the neighborhood. Designed in 1865 by Irish-born architect Patrick C. Keely, it is listed in the National Register of Historic Places. Among the more notable neighborhood residents who attended this church is Samuel Clemens, better known as Mark Twain.

Asylum Hill also was home to many educational institutions. The original Hartford High School was designed by architect George Keller, but was demolished in 1963 in favor of construction of Interstate 84. The Hartford Theological Seminary built its headquarters on Broad Street in 1879 and was based there until moving to its current location in the West End Neighborhood.