WASHINGTON, Feb. 4— The National Institutes of Health have quietly initiated a study of cancer deaths among populations near nuclear power plants, according to a letter released today by Senator Edward M. Kennedy.

The letter, which was sent to the Massachusetts Democrat from Dr. James B. Wyngaarden, director of the institutes, said the studies were started as a result of ''leukemia clusters around the Pilgrim power plant in Massachusetts and several plants in the United Kingdom.''

The findings ''have led us to initiate a large-scale evaluation of cancer deaths occurring among persons living near the over 100 reactors operating in the United States,'' said Dr. Wyngaarden.

Most experts in radiation have said that the level of radioactivity near nuclear power plants is too low to be a health hazard. Collaboration With Swedes

Don Ralbovsky, a spokesman for the institutes, said the letter was sent on Jan. 28. He added that Dr. Wyngaarden was out of town and others who could comment on it were not reachable.

In the letter, Dr. Wyngaarden also said the institutes were collaborating in a Swedish study of 40,000 patients who have received low doses of iodine-131 for medical diagnostic reasons. Iodine-131 is described as ''one of the major radioactive isotopes emitted during nuclear power plant operations and from nuclear weapons testing.''

''We have also evaluated descriptive mortality data regarding possible cancer risks in the general population living downwind of the Nevada nuclear test site,'' the letter said. ''While many reported associations are unsupported by these data, a small increase in leukemia in southwest Utah cannot be ruled out at this time.''

A site in Nevada has been used for years for underground testing of nuclear weapons.

The letter said the institutes have ''confirmed that leukemia was increased above expectation'' among military personnel who participated in at least one nuclear weapon test series.

Dr. Wyngaarden said results of the studies are expected within a year. Three Mile Island Effects

Dr. Wyngaarden also said the most serious health effect of the nuclear power plant accident at Three Mile Island, Pa., ''is mental stress to those living near the plant.''

He said the March 28, 1979, accident exposed the 36,000 people living within five miles of the plant to an average dose of two to eight millirems, ''or approximately what might be received from natural background radiation within one or two weeks.''

The Three Mile Island facility, in Middletown, Pa., is owned by General Public Utilities Corporation.

In a reply to Dr. Wyngaarden's letter, Senator Kennedy said he believes the Nuclear Regulatory Commission should delay reopening the Pilgrim plant ''until this study is available.''

Pilgrim, owned by Boston Edison, has been closed since April 1986 because of problems with equipment and management. No date has been set for restarting it.