hello apa,
please check Tool for EKB Generation section for reference.
since you’re using a custom key, you’ll also need to perform gen_ekb.py to create a new EKS image; please also updating EKS image accordingly for image flash.
furthermore,
here’s similar discussion thread, Topic 270934, which give steps to enable disk encryption with a custom key,
it’s tested/verified the ROOTFS_ENC functionality had worked normally.