Should I import my secret recovery phrase from Rabby Wallet in ZERO or the other way around?

No. You should not import any seed phrases at all, but rather synchronize your ZERO with Rabby Wallet. While it is possible to import your Rabby Wallet seed phrase in ZERO or to import your ZERO seed phrase into Rabby Wallet, this will provide no extra security.

Rabby Wallet is a so-called 'hot' wallet: the private keys for a wallet are generated and stored in a browser or app that is connected to the internet. On the ZERO, the private key is generated by a unique procedure and stored 100% offline using hardware made for this purpose.

To use the Rabby Wallet integration you should sync your ZERO with Rabby Wallet. You don't even need to have a wallet made in Rabby. Then your ZERO accounts will be available in Rabby Wallet.
See Synchronize ZERO with Rabby Wallet.