Submarines
The protected waters of Connecticut and Rhode Island were the site of much of the US Navy's early submarine development efforts, and continue to be even to this day. Many of the Navy's early submarine designs were less than successful, and a number of these boats, some of which were obsolete before they were completed, were used as subjects in weapons tests. If you would like to see a sub closer to home, you may tour the USS Ling in Hackensack.
There are also several German U-boats in the region, most notably the U-853. In the early years of World War II, the u-boats had tremendous success off the East Coast, mainly because practically no effort was made to combat them. These were typically the small Type VIIc and Type IXb submarines, and none of them are sunk off New Jersey or Long Island. The Germans called this 'The Happy Time.'
In the later years, anti-submarine warfare improved tremendously. Although the boats were the larger and more capable Type IXc, they seldom got away without being attacked, and three of them lie in diveable waters, if you call 200-300 feet diveable. And who knows - there may be others.
None of the u-boats that are known to have had action off the New Jersey / New York coastline survived World War II, although the U-123 was raised after the war and put back in service by France. The U-151 of World War I did survive that war, and was sunk as a target in 1921. Also in WWI, the U-156 is assumed to have laid the mine that sank the USS San Diego, and was lost shortly after.