The edge of objects generated with Optix is not smooth

As you can see from the two pictures, the object’s edge is not smooth, How does it happen? What can I do to improve the result?

It is not smooth because you are only taking 1 sample per pixel. You should read up on anti-aliasing techniques if you want to get smooth edges. Most of OptiX sdk examples have a anti-aliasing camera implemented.

Would you please in which SDK example the anti-alsasing cameara is implemented, I have searched most of the examples, but I still can’t find one. Dose it exist in Ray_Generation program or somewhere else?

If you search for “antialias” within the SDK folder, you will get at least the following:

NVIDIA-OptiX-SDK-3.0.1-linux64/SDK$ grep -r "antialias" .
./displacement/  // to provide antialiasing.
./displacement/  // to provide antialiasing.
./sample6/sample6.cpp:    << "  -aa | --antialias                          Use subpixel jittering to perform antialiasing\n"
./sample6/sample6.cpp:    } else if( arg == "-aa" || arg == "--antialias" ) {
./sample6/  // to provide antialiasing.
./sample6/  // to provide antialiasing.
./zoneplate/zoneplate.cpp:    "NVIDIA OptiX antialiasing demo\n"
./zoneplate/zoneplate.cpp:    "  m                   Cycle antialiasing methods\n"
./ocean/  // to provide antialiasing.