Decent Speed Infusion on a F/P Arch Mage - is it worth it?

I'd still have to swap the gloves out every time to recast it so it wouldn't really make any difference.
I don't ever plan to make new characters (aside from Character Deck and Link Skill which I don't even need to fund) so the other point is irrelevant to me.

You said it was for dojo, you have time to switch it between each set of 5/even between floors. As for how useful decent SI is, the difference of one atk speed isn't that noticeable but its there/will add up to be a lot over all the floors