How can I find out where this strange variable in my interactive shell came from?

Some variables are set by programs that run before bash, and this FAQ can't help you with those. For the variables that are set by bash reading a dot file, if you are not root, you can use bash's trace mode:

$ PS4='+ $BASH_SOURCE:$FUNCNAME:$LINENO:' bash -lxc : 2>&1 | grep CVS_RSH
+ /home/greg/.profile::44:export CVS_RSH=ssh
+ /home/greg/.profile::44:CVS_RSH=ssh

In this example, the variable we were looking for is CVS_RSH, and it's being set in /home/greg/.profile on line 44. The same technique can be used to track down anything that comes from a dot file -- aliases, functions, undesired shopt settings, and so on.

This will not work if you're root, at least not in bash 4.3 or newer, because those versions of bash refuse to honor an inherited PS4 variable when running as root, for security reasons. PS4 may contain command substitutions, which would be executed upon expansion.