Most notable of these was likely S. Wadsworth's Inn, operated by Elisha and Sidney Wadsworth. Located at the junction of what is now Albany Avenue and Prospect Avenue, the building still stands today.

From colonial days until the end of the 19th century, Upper Albany was primarily farmland. Throughout most of the 19th century, James Goodwin, a railroad and insurance magnate, owned much of the property along Albany Avenue.

In the 1870's the Hartford Trotting Park was located on the south side of Albany Avenue near the Granby Turnpike (now Blue Hills Avenue). Milford Street cuts through where the park once stood. Among other things, the park was the site of base ball games. The Middletown Mansfields played three games during 1872, and the Hartford Dark Blues considered the park to be their home field prior to the ball park that was built on the Colt family property on Wyllys Street.

The neighborhood built-up quickly once trolley lines were built. Well-established Protestant families were the first to move into the neighborhood, followed by upwardly mobile Irish and Jewish families.

In 1927, Hartford's oldest and largest Conservative congregation, Emmanuel, completed a synagogue on Greenfield street. A year later, Agudas Achim, an Orthodox congregation first organized in 1887 by a group of Rumanian Jews, dedicated their synagogue nearby.

The character of Albany Avenue changed rapid after World War I. It became largely commercial, with some older homes torn down to make way for businesses, and retail store fronts added to other residences. By 1920, most of the property owners in the area were Irish or Jewish because of the close-knit communities that had developed.

After World War II, the number of African-Americans in the neighborhood increased while other groups declined. Today, the neighborhood is predominantly African-American, Puerto Rican and West Caribbean residents.