Proper sweep for character controller? (PhysX 3.2.4)

I am trying to implement gravity and jumping into my character. My objective is to check if the player is on the ground. What I thought of doing was a sweep test. Instead of navigating the many interfaces of accessing the geometry of the character, I decided to create a capsule geometry myself. This is the code for this:

physics.getScene()->removeActor(*controller->getActor());
PxSweepHit hitInfo;
PxVec3 centerPos;
centerPos.x = controller->getPosition().x;
centerPos.y = controller->getPosition().y;
centerPos.z = controller->getPosition().z;
bool gotHit = physics.getScene()->sweepSingle(PxCapsuleGeometry(0.25+controller->getContactOffset(), 1.25/2), PxTransform(centerPos),PxVec3(0,-1,0), 0.1, PxSceneQueryFlags(PxSceneQueryFlag::eINITIAL_OVERLAP|PxSceneQueryFlag::eINITIAL_OVERLAP_KEEP|PxSceneQueryFlag::eNORMAL), hitInfo);
if(gotHit){
	gravity = 20*Settings::Timestep;
	std::cout << "Ray charles came back true "<<std::endl;
}
physics.getScene()->addActor(*controller->getActor());

As you can see, I remove the character from the scene, so that the geometry sweep test doesn’t collide with the actor of the character. Next I get the position of the character. I do a sweep test. Since the character has a contact offset, I have to adjust for this too in the geometry. The offset adds a 0.1 (default) area around the capsule. Since I am making a capsule geometry, I have to use the half height of the controller, and add the contact offset to the radius(as this will make it the required amount bigger). Next I just simple do the sweep test. I check for already colliding geometry, as it is important to know if the character is colliding with the floor already but not moving.

Now the issue is that this does not work. I am unable to jump, or rather, the gotHit bool stays false, regardless of if I am on the floor or otherwise. There is, however, an exception: I get a sweep test positive if I am against a vertical wall. The normal returns (0,1,0) and I am able to jump as is intended. How do I go about fixing this issue?

Have you tried increasing the sweep distance?
I have just implemented the following code in my project:

// Get the shape corresponding to the controller
PxShape* shape;
actor->getShapes(&shape, 1);

// Get the geometry corresponding to the controller
PxCapsuleGeometry capsuleGeometry;
shape->getCapsuleGeometry(capsuleGeometry);

// Define what parts of PxSweepHit we're interested in
const PxSceneQueryFlags outputFlags = PxSceneQueryFlag::eIMPACT;

// Remove the actor
mScene->removeActor(*actor);

// Perform the sweep
status = mScene->sweepSingle(capsuleGeometry, actor->getGlobalPose(), PxVec3(0, -1, 0), 1.0f, outputFlags, hit);

// Restore the actor
mScene->addActor(*actor);

It works well, but not with a distance < 1.0f. I have not tried to analyze further as this fits to my needs.

For information, I use a different method to know if my character is allowed to jump or not: I save the last time the character has touched the ground (using PxControllerFlag::eCOLLISION_DOWN), and if this time is close enough to the current time I allow the character to jump.
This allows my character to jump even a short time after it has walked over an edge, which is what I wanted.

Also, I suspect removing and adding the actor to the scene may require more computation than excluding the actor from the query by filter.