I would like to exchange a bit about this problem with CloudXR that we usually get those blurred lines, I think in the know issues for the Unity Plugin it is described as: “Vertical banding distortion: some or all platforms exhibit a “blurriness” in the form of 6-10 regularly spaced vertical bands of distortion”. On Quest 3 I get those lines vertical and horizontal.
For Quest 2 the situation was a bit different, I think I got the lines there only in one direction, and actually after adding foveation plus setting the frame rate to 90 they were completely gone. This was with CloudXR 3.2. Now if under CloudXR 3.2 I do the same with a Quest 3, as soon as I set the frames to 90 which activates this AImageReader Decoder the whole image is always flipped upside down. Only tweaking foveation did not remove the lines for me.
In the tests with CloudXR 4.0 we got so far we did not see any difference, there the AImgeReader Decoder is not activated by default so the lines stay there also if we set frames to 90, if we activate it manually we also get the flipped upside down image. There are other things blocking us from going to 4.0 but I wanted to check at least if that gets better in 4.0 somehow.
Was anybody more successful in getting those lines out of the image?
I just did a clean install of CloudXR 4.0 on my desktop, and tried the Oculus sample app (NOT Unity) on a Meta Quest 3. The “lines” issues is very clear, and when I play with the foveation (-f), scale factor (-m) and other settings, I start to perceive a “grid” of lines, which on close inspection seem to be some kind of offset of one block-column/block-row of pixels relative to its neighbors, and the grid size varies with these parameters.
I did some debugging with our video and image dump flags (-ccb, -csb, -d), and if I view the dumped raw HEVC streams in e.g. VLC, I don’t see the issue, but although it takes a minute to find it in a static image, I do see it in the dumped PNGs.
I cleared away all my flags except IP address, and added -enable-ir-decoder. This switches from Android’s MediaCodec to Android’s AImageDecoder. We previously recommended not using this., Suddenly it all looks great, no lines at all. Some folks have reported vertically inverted images, I don’t get that. I’ve confirmed at least one other Meta Quest 3 user with a totally different setup than mine gets this result as well.
This is on the native Oculus VR sample app. I have not tested AImageReader with Unity, the vertical flip thing is an easy fix if that’s where you’re seeing it. I will do that next.
If I enable the AImageReader decoder in the Unity client (therefore CloudXR 4.0 client library and SteamVR server), I get:
A vertically flipped image
None of the blurry grid lines
Checking our code, it turns out that the Oculus sample client handles this case, in which the AImageReader actually returns a texture that is flipped relative to what MediaCodec returns. For now, I went into the CxruReprojection.shader and added this line:
// Temp hack to vertically invert output
uv.y=1.0-uv.y;
to the top of the float2 ReprojectUV(float2 uv,float4x4 tV,float4x4 tVS){} function. With this, I get:
No blurry lines
Correctly oriented frames
noticeable jumpy/laggy display
I ran several tests, and with MediaCodec, no lag. With AImageReader, substantial jittery lag in displayed pixels. And not the acceptable reprojected lag, actual head pose lag that can induce nausea. At the same time, there have been reports on this forum of significant / intolerable lag when using a Unity client to connect to a cloud instance, but not from a local instance.
This appears, for the moment, to be a deeper bug in the Unity plugin which will take some time to debug.
All of that said, to get back to the main topic of this thread for a minute: the blurry lines bug.
This appeared in CloudXR on Quest platforms somewhere between 12 and 18 months ago, starting with one of the CloudXR 3.2 releases. I suspect that something changed in how we are encoding that is not properly handled by MediaCodec on the versions of Android used on the Quest. It would appear that AImageReader handles this better, but for a time, we did not suggest using it, because we had stability issues with it on the Quest Android OS, which was very old at the time, though since the release of Quest 3 has moved forward a bit.
This is my working theory: Changes to use advanced h265 codec features to improve streaming quality/latency/performance resulted in this unexpected artifact, but only under older Android devices using MediaCodec. The more modern and supported AImageReader decodes them just fine.
Hi!
Thanks for the detailed answer! Actually I can confirm that it works with AImageReader for me as well, as soon as I turn off the foveation. As soon as I activate some foveation the image is flipped upside down again. So I would assume that somewhere in the code there should be a reason for that, I will check :)
so my understanding was that per default foveation is not active but strongly recommended
(at least docu still says this: )
-foveation N -f N Enable foveated scaling at given percentage scale.
Values from 25-100 as a percentage scale factor on original frame to destination post-foveated frame size. 0 to disable (or to override an options text file value from the command line). Defaults to 0.
We strongly recommend that you always enable for VR use. Suggested value is 50 percent, using -f 50 or explicitly preset in client code in foveatedScaleFactor)|
But yes for me if I set any foveation value I get the inverted image, otherwise not.
You are correct! I may be remembering an older version of CloudXR or just misremembering.
I can confirm the behavior you describe exactly. What is more, I have actually found the bug in the CloudXR library code on Android that causes this bug, and fixed it. When we release a bug-fix build (currently targeted for mid-July), this will be included.
For now, the workaround is:
enable AImageReader via the -enable-ir-decoder flag (or, in code, use cxrDebugFlags_EnableAImageReaderDecoder).
disable foveation, either by using the default value of 0, or by explicitly setting foveation to 0 or 100
Great thanks! Could you maybe already include the code for the fix here, so we could just through it into our current client? Would still be great to use the combination of foveation and the AImageReader.