it seems behave different from the euler angles and has nothing to do with the rotation matrix, but when I change its value, the camera actually rotate?
The camera view vector W (from camera eye point to the look-at point) and the up-vector define a plane.
The camera vector V is in that plane, the camera vector U is perpendicular to that plane.
If you rotate the up-vector around the view vector W, the camera will roll.
Beware of gimbal lock. This is not working when W and up-vector are collinear.