The psp going down and to the right is doing exactly what it should with no joystick connected
Lets look at the following diagram
When your joystick is in the center and not being pushed, the resistance is balanced to both directions.
If you push the button up, the resistance goes down in the up input, but goes up on the down input.
It is a balancing act, the same goes for the right and left, as you push left the resistance goes down on the
left input but goes up on the right input.
The further you push the button the greater the difference, the faster the cursor moves or the tighter the
turn you make in your game. It is proportional feedback loop.
Now if you remove the joystick things go out of balance and there is maximum resistance on all inputs. Because
of the balance, without a joystick it will always go down to the right.
If you have a loose connection the resistance jumps all over the place and you can not get things balanced
correctly.
So contact pressure to make very good contact is most often the reason a joystick gives people a problem.
If you download the joystick tester on your psp, you can confirm this by leaving the screws on the left side of
your case a little loose and run the program. Take two of your left hand fingers and squeeze the case together
to make better contact and you will see the cursor jump to the middle of the screen.
That little washer is there to make the correct amount of pressure when your case is put together
correctly. If you lose it you will have problems. But I have seen cases that were a problem and all I had to
do was make another circle out of clear report binder folder and use it as an additional shim to make things
tight when the case was closed.
This is a greatly oversimplified version of what is going on, but it gives you the concept of what is happening
and why your games run away when there is not a good connection.