One more observation, when using mp4 file as a input, and omxh264dec to decode with nvoverlaysink as video sink, the video plays out fine ( i.e. the chroma is displayed correctly). Also the omxh264dec outputs raw video in YUV 420 Semi Planar format, which is the same format I am trying to use. I tried to dump raw video after decode and use it with the pipeline I have mentioned above. I still see the same distorted video.
When working with raw files, in my experience it is necessary to add the blocksize parameter to the filesrc. Furthermore it may be necessary to transfer the raw video to the NVMM memory before sending it to nvoverlaysink.