It's safe! When you sign in with guild.xyz, you only are validating ownership of the wallet address. The Guild.xyz bot does not get any permissions to perform transactions, and does not have any way of withdrawing anything from your wallet.