I had that problem on a memory stick. It would go real slow and look like it would never load.
If I rebooted the PSP, complete cold restart it would be fine for a while and then start doing it again.
I replaced the memory stick and the problem never came again.
But it may be the interface chip in the SDHC to MSPRO duo adapter not the SDHC card itself.
I have several more adapters coming in sometime at the end of the week. Will test and tell you if I can repeat your problem with them.