The white frame building located near the Manor House was once slave quarters and later the cook’s house. Indentured servants worked at the Manor at first, and around 1711 the Jesuits were given slaves.

Fr. George Hunter, S.J., in his spiritual retreat notes of 1749 wrote about slavery. He said charity toward slaves is due from all, particularly their masters. He emphasized that they were to be dealt with in a charitable and Christian manner. A modified form of slavery existed on Jesuit farms until 1830, when sharecropping began.

In the basement of the servant house is an empty well. In 1963, while repairing steps, a parishioner uncovered the upper end of a tunnel which apparently ran to the water. In a record in the Jesuit Archives, a priest reported that the "tunnel had been cemented over when the floor was poured in the slave house." There are many versions of its use: Jesuits were smuggled into the colonies; priests hid from the British; it was used to travel to and from the river wharf without confronting inclement weather; goods were moved from boats and brought up through the tunnel. It is also possible that the tunnel was used by slaves in the "underground railroad."

In February 1999 researchers from Boston College and Western Geophysical Corporation searched for the path of the tunnel. The scientists located the first 20 feet of the tunnel.