The victims were all children, according to some reports, though police have not confirmed their ages or identities.
Another 24 people were rescued from the water, NBC New York reported. They are thought to have been watching a nearby firework display when the boat sank.
The 34-foot Silverton yacht capsized just after 10 p.m., according to the New York Post.
Police are not yet sure what caused the accident, but believe it could have been due to a sudden thunderstorm as well as the wake created by other boats leaving the bay after the fireworks.
One witness told Newsday that he saw the yacht turn, get hit by a wave and tip over.
Survivors were pulled from the water by coast guards, members of the local police and fire departments, and private boat owners, the Associated Press reported.
Witnesses described the scene as "complete mayhem," telling the Post that other boaters threw life jackets into the water as parents swam in frantic search of their children.
One of the victims' bodies was pulled out of the water last night, a police spokesman told ABC News. The other two were found in the sunken yacht after a long search.
Not all passengers were wearing life jackets, the officer said.
Some of the survivors were taken to local hospitals for treatment, according to Newsday; None is reported to have been seriously hurt.