I am still wondering at which stage you are now, hazephase. From your earlier post, it seems that you have played for a year now. Do your mates play for roughly the same time or are they already more advanced?
What is the ambition if you tell that you're slower than the rest of the group? If it is rhythms that you cannot follow when the speed increases? Then my advice would be first of all to relax. Once you tense up, you hardly can move any more. And break down parts of the rhythm and do them over and over slow to fast but without creating tension in your body. Obviously, the less you move your hands from one note to the other, the faster you are in the end. Your hands will recognise the movements and it will become automatic and faster over time.
Or is it that others are faster in long, fast roulements? On this I would worry a bit less at that stage, although the same advice holds. Relaxation and wrists are even more important here as you increase mass when playing tensed. And following Sir Isaac
the bigger the mass the more force you need to change direction, which becomes a vicious circle and you break down quickly. Starting up your exercise with just slap roulements may be an idea to develop your technique and stamina, too.
One other interesting approach I read during my earlier piano studies: increasing speed from slow to fast can leave you in a speed lock, as just the movements you use when you are slow have their natural limits. So, the idea presented was to start reverse: infinite speed, i.e. play notes together. Then you develop a different movement by trying to 'slow down'. Sounds strange, but a really different approach in practice. This obviously does not solve all of it. Once you start playing scales you need to cross fingers (31 or 41) etc. For the djembe, you can just play two notes together and the almost infinitely fast form would be the ultra short flam. I never tried, but I wonder whether you could increase speed by approaching roulements via repeated flams? At least you learn a lot about flams