How to use OpenSSL RSA in trusty?

zjfsharp,
Here is more update for your reference,
You are correct. The sample can not be compiled with the OpenSSL that had been configured with embedded settings.
As the downstream was upgraded the OpenSSL to 1.1.1g, the error message is different but still can’t be compiled with the APIs,
PEM_read_bio_RSA_PUBKEY
PEM_read_bio_RSAPrivateKey

Here is a suggestion to get the RSA key,

  1. To generate RSA key pairs for DER format on the host and store that on the device. This can be done using custom EKB format and flash it on the partition.

  2. In the Tursty TA,

  3. RSA encryption of private key

1. Load the private key of DER format into memory
2. EVP_PKEY_new (create a key structure)
3. d2i_PrivateKey (setting up the private key from memory into the key structure)
4. EVP_PKEY_get1_RSA (get the RSA structure)
5. RSA_private_encrypt/decrypt (perform the RSA private key encryption)
  1. RSA encryption of public key
1. Load the public key of DER format into memory
2. EVP_PKEY_new (create a key structure)
3. d2i_PublicKey (setting up the public key from memory into the key structure)
4. EVP_PKEY_get1_RSA (get the RSA structure)
5. RSA_public_encrypt/decrypt (perform the RSA public key encryption)