I think it is slow going in the beginning because you are basically learning a new language. After awhile, you will be able to remember the handing for the rhythms more quickly, but even then, it may take awhile to be able to remember and play how it is supposed to sound.
I record each rhythm and listen to them regularly. At this point, I can play about 50 or 60 different patterns from memory and it is because I practice them all. I have actually typed up a list of songs and go through them all about once a week or so. It helps a LOT.
Another thing that helps is that my teacher will have us playing the same song, with it's solo patterns, for several weeks in a row. Then, when he moves on to a new rhythm, we will still go back to the old one for awhile. It keeps it fresh.
Also, in the beginning, I would write down some notations to remember the handing for a piece. As I grew more experienced, I didn't have to do that anymore, but still use the recordings all the time.