If you’re neurotic, you likely have a propensity for being anxious, worrisome, and possibly depressed. You view small dilemmas as major threats and frustrations, you tend to have mood swings, and you’re more likely to experience negative emotions like anger, fear, worry, and jealousy. But neurotic people also have worse memories than people who are more laid-back, according to a new study.

Researchers at the Icahn School of Medicine at Mount Sinai in New York collaborated with scientists in the UK and Switzerland to publish their work in the journal Human Brain Mapping. Their aim was to investigate the relationship between personality and brain plasticity — the ability of the brain to change to better meet cognitive challenges (in short, the brain’s ability to grow and get smarter). The researchers focused primarily on neuroticism — a personality trait that is often painted as charming and sharp in Woody Allen’s films, but one hasn’t been fully researched.

“Neuroticism is universally implicated in making things difficult for people, whatever they might be doing,” Sophia Frangou, lead author of the study, told Smithsonian. “These associations have been known, but there was no mechanism to say why one thing influenced the other.”

The researchers took 40 adult participants and measured brain activity while they completed a memory test, using functional magnetic resonance imaging (fMRI). During the memory test, the participants had to look at a letter sequence on a computer screen, then decide when a current letter matched one from a previous sequence. Afterward, the researchers gave the participants a psychology test known as the NEO-PI-R, which measures the five major personality types: neuroticism, extraversion, openness to experience, agreeableness, and conscientiousness.

Interestingly, they followed that people who scored higher in neuroticism were less efficient, “perhaps because they have the tendency to worry,” Frangou told Smithsonian. People with high levels of conscientiousness scored higher on the test, and completed it more quickly. The fMRI showed that neurotic people had slower neural connections than others.

Why is this the case? Neuroticism and a tendency to worry about all possible negative outcomes might seem like a good way to be careful; but in reality, the tendency to distress can slow you down. “[S]omeone who is less prone to distress and is able to focus on the task at hand will be more likely to get everything on the list and do it more quickly,” Frangou told Smithsonian.

In the past, researchers have linked neuroticism to a variety of health issues, including heart disease, lung disease, high blood pressure, and arthritis. In one recent study, researchers found that people who were extraverted and conscientious, yet more laid-back than neurotic, actually showed greater propensity to have a cleaner bill of health. Neuroticism is typically associated with high levels of stress and anxiety, which increases the stress hormone cortisol. Cortisol in high amounts can impair cognitive function, shorten your telomeres, and increase your risk for chronic diseases. Stress can also have a bad effect on your immune system.

Neuroticism may not have previously been something to talk to a therapist about, but as more research links it to poor health, it might be something you should mitigate in your life. Your memory will thank you for it.

Source: Dima D, Friston K, Stephan K, Frangou S. Neuroticism and conscientiousness respectively constrain and facilitate short-term plasticity within the working memory neural network. Human Brain Mappingi. 2015.