Without a doubt it's harmful. My ears ring all the time. I haven't had my hearing tested lately but I'm pretty certain the results are not going to be encouraging. Djembes were meant to be played outside but most practice, performance and dance classes take place indoors. I've tried earplugs but I don't like the sound. I suppose I won't like being deaf even more. Playing softly presents technique challenges. Playing without bells loses an important element to the music. I've been looking for a middle ground but haven't found it yet.
Just this morning I was considering a setup for practice. Mic'ing the drums through a mic preamp and setting the volume on a pair of headphones to a safe level. Then I don't lose the dynamics of the slap and tone or my hearing.