Few Americans died. The vast majority of the deaths occurred when the French were starting construction, far fewer (but still over 5000) during American construction. Almost all of the fatalities were laborers from Caribbean islands who lived in terrible conditions and received little medical care.