the sampling rate determines the frequency range. There something in information theory known as the "Nyquist Limit". It states that the highest frequency that can be captured by taking samples at n
Hz is n
So, 44.1kHz sampling rate means that the highest frequency that can be encoded is 22.05kHz, and 96kHz means that the highest frequency that can be encoded is 48kHz.
On the face of it, the higher sampling rate seems pointless, given that the human ear (and only a very young one, at that) can only hear frequencies up to 20kHz. The reason for sampling at higher rates is to make anti-aliasing easier. Basically, to prevent audio artifacts, after analog-to-digital conversion, the analog signal must be fed through a low-pass filter that removes the sampling noise above 20kHz. With a 44.1kHz sampling rate, you have to use quite a steep (1-octave) filter to do this. The problem is that such a steep filter itself introduces artifacts into the audible parts of the signal. By using a higher sampling rate, the sampling noise is further away from the 20kHz audio limit, so it can be removed with a flatter filter that introduces fewer artifacts.
Another advantage of the higher sampling rate is that it increases effective resolution and reduces the noise floor of the final audio signal a little.
Will any of this make a difference in your situation? Almost certainly not
Basically, for mere mortals with a stereo system for mere mortals, a 320kbit/sec MP3 recording is indistinguishable from a CD. You need to be young, have trained hearing, and have a stereo system that costs in the multiple tens of thousands of dollars to be able to detect the difference between a 44.1kHz/16bit recording and a 96kHz/24bit recording. If you are 30 years or older, chances are you could not tell any difference even with the world's best playback equipment. For field recordings, where you cannot control acoustic conditions and you usually have lots of ambient noise, there is even less point in these higher-quality settings.
Personally, I record at 320kbit/sec, which gives me outstanding sound quality while keeping the file size reasonable. In turn, this means that I run out of memory on the recorder much later, that the files transfer to my computer more quickly, and that the size of my backups is smaller. And I can fit far more of the recordings on my iPhone, where storage space is at a premium.