Asking for a particular frame never controls playback and therefore never causes the decoder to change its playback rate. If you ask for a frame 3 seconds in the future the decoder won't seek forward and decode your frame. Decoders generally have a queue of frames to allow them to be out of sync with the presentation system. It also allows the decoder to take a variable amount of time to decode a frame, which is often the case.
I'm not actually sure what happens when you ask for a frame that is too far in the past or the future. I will talk to some of the video people here to try to understand it more.