Scientists believe that periods of sleep called short-wave sleep are important for the consolidation of "memories" acquired during the day.
Just before going to bed, some of the volunteers were given a drug designed to increase levels of acetylcholine, which is a chemical neurotransmitter found in the brain and central nervous system.
The others were left with normal levels of acetylcholine.
In the middle of the night, they were re-tested on a list of words they had been asked to memorise the previous day.
Those with high acetylcholine levels did worse than those who had not been given the drug.
Researchers from a German University found that volunteers with boosted levels of acetylcholine performed less well in night-time memory tests.
The study, published in Proceedings of the National Academy of Sciences, is intriguing for experts interested in Alzheimer's disease.
A lack of the chemical is currently thought to play a role in the illness.
The small study at the University of Lubeck focuses on healthy young men - rather than older subjects or Alzheimer's patients.
The researchers wanted to test the relationship between levels of the brain chemical to their ability to lay down new memories during the night.
While the aim of the study was primarily to test the mechanisms of memory storage, the authors said it might suggest ways of improving the treatment of some Alzheimer's patients.
Many take drugs that boost acetylcholine levels before going to bed because they will then sleep through any unpleasant side-effects.
However, the researchers wrote: "The finding implies that the administration of cholinesterase inhibitors before sleep in Alzheimer's patients should be reconsidered."