I just figured out that I can play the game with a Xbox controller, but when I stop pitching, it goes back down, because it is calibrated to my lap and the plane just straightens out
Welcome to the community!
Unfortunately, Infinite Flight does not support an Xbox controller— I’ve tried it too. Troubleshooting regarding IF and an Xbox controller isn’t exactly possible since the developers have not supported it yet.
For flying, I do not recommend the Xbox controller to control the aircraft because the controls are extremely limited, and also for the issue that you provided above. I’d just stick to the normal device tilt or use a joystick.