First time fast, other times slow!!!

My program encounters a strange problem.
I tested it on a powerful computer with 4 NVIDIA GPU (GeForce GTX 970), which is not belong to me.
For the same task, the first time it takes 1 second, then other times it takes 2.5 seconds always.
Every time, the result was the same and correct.
My computer never has this problem.
I have checked the program many times, and pay much attention to the initialization, but nothing was found.
So now I don’t know what should I do for it!
Does any one has any idea?
Thank you!

I think I have found someting.
I dispatched subtasks to 4 devices this way: first, I dispatched 3 subtasks to every device, then after 1 subtask was finished, append a new subtask to that device.
It is usually ok, but not for every computer.
On the computer I have mentioned above, it becomes slow.
When I changed the pre-dispatched subtasks from 3 to 20, it becomes normal.
I want to know how can that be?
Thank you.