It takes a while to develop your pattern memory. When I first started out, I had problems remembering the patterns in large part because, to me, many rhythms sounded the same or similar. I wasn't aware of the larger melody of things and how my accompaniment fitted in with that, which made it harder to remember things too. (In contrast, these days, when I hear a dundun line, I generally recognize the rhythm and then also know how the hand parts fits into the dundun line to make the complete melody.)
So, the longer you play, the more nuances your brain will recognize, which in turn helps with memory.
One way of remembering things is to repeat them... One way of remembering things is to repeat them... One way of remembering things is to repeat them... One way of remembering things is to repeat them...
I guess you get the message

The more often you play a particular part, the more deeply you commit it to long-term memory. So, play the same thing more often, and it will be easier to remember.
Another thing I do when I learn a new rhythm is to silenty sing the different parts to myself. I do this
a lot, whenever I have a few spare moments. I mentally drum when I'm stopped at a traffic light, when I'm having a shower, before I go to sleep, when I sit down outside for a quiet time, when I'm waiting for the bus, while walking to the shops, etc... This practice probably does more to firmly anchor a rhythm in my memory than anything else.
And the mental exercise helps with dexterity too. If I find the handing for a part counter-intuitive and struggle with it a little, and then sing the rhythm for myself on and off for a day or too, when I go back to playing it again, I have much less trouble. The act of imagining playing of a rhythm uses the same parts of the brain that are used when actually playing the rhythm. (This is a quite well documented effect that has been used for ages in sports and other fields--for example, repeatedly visualising a complex move improves the performance of gymnasts when learning the move. Functional MRI studies have confirmed that the visualisation triggers the same areas of the brain into activity as performing the actual task.)
So, what it boils down to is that, the more time and attention you devote to your drumming, the easier it gets to remember things (which really shouldn't come as a surprise).
Another technique that some players use is to make up a mnemonic for a part. Vocalize the part and listen to what it sounds like. It may bring to mind a line from a poem or a childrens rhyme. If so, you can mentally link the rhyme with the rhythm, which also helps to remember it.
For myself, tactile memory is a big part of remembering. The pattern of movement and what each rhythm feels like in my hands is very important for me. Other people are more visual and write things down, which can help a lot if you are visually inclined.
Yet another aid for memory is to get a cheap voice recorder (if you don't have one in your mobile phone already) and to vocalize a part that you want to remember. All it takes is ten seconds of vocalization to act as a memory aid, so you can retrieve that pattern again whenever you want.
My suggestion is for you to work out which kind of memory works for you (visual, auditory, tactile) and then work to your strength by using a memory aid that matches. And then go and do it a lot!

Cheers,
Michi.