A key point is that as an absolute minimum, an ISR needs to clear the source of the interrupt so it doesn’t just fire again when the ISR ends. If not, what you describe happens, and the ISR is just immediately entered again.
Normally, the ISR will handle the data transfer to/from the device, and perhaps defer to a task to process it.