The sticks are going to Analog Inputs on the MCU, and they act as voltage dividers, so with the POTs removed those Inputs are 'floating' and will do whatever they feel like doing, and can sometimes make other parts of the controller act up.
As a quick and dirty test, you can just short the center spot where the POT goes to one of the outside spots with a pair of metal tweezers or a jumper wire, and that Axis should go full tilt in one direction, then short from center to the other outside spot it should go full tilt to the other direction. Never short the 2 outside spots to each other, as they are power and ground for the POT, only center to one, then center to the other.
1 2 3
Connect 2 to 1 and observe what it does. Then remove that jumper and connect 2 to 3 and it should do the opposite. That is, if it went full left or just stayed full left for the 2-1 test, then it should go full right on the 2-3 test. Then do the same test where the other 3 POTs are supposed to be.