Yes providing the physical size of the 125V fuse is the same as the 250V fuse and the circuit voltage the fused will be used is not greater than 125V.
The voltage rating of the fuse is for arc across rating protection in the event of a current overload or a short circuit current fault through the fuse. If the fuse is rated for 125V and installed in a 250V circuit it may not extinguish the arc in the event of an overload in excess of the ampere rating of the fuse, or worse yet a circuit short circuit event.
.