You don’t say which compiler you are using. Have you checked the WEB documentation page for a similar port (sam7 with the same compiler) to see how an ISR should be written? Different compilers have different requirements as to when and how the task context should be saved and restored.
I’ve taken the AT91SAM7S64 example and created (with small changes, just the include and linker files) a port for the AT91SAM7A3.
I’ve looked to the example of serial.c for the AT91SAM7S64 but somehow it does not seem to be up to date since it does not use __irq in the declaration of the interrupt routine. So most likely this is from IAR 4 and uses the IRQ_Handler_Entry provided in Cstartup.s79 file.