Mr Wade said he and his wife organised the first event after learning of something similar in Cornwall.
About 50 people turned up in the first year, and it grew to 180 swimmers and about 1,000 spectators in 2023.
“We will miss it,” Mr Wade said. “It is a very popular, big event.”
He said stricter insurance requirements could mean the event was unable to return in the future.
“When we first started, there were no conditions on us at all, we just did it,” he said.
“Gradually, over the years, things like risk assessments and event plans have come in that are highly scrutinised.”
The event was jointly organised by the lifeboat station and a group of local fundraisers.
It raised about £4,000 a year, Mr Wade said.