If a variable straight line $x\cos \alpha +y\sin \alpha =p$ which is a chord of the hyperbola $\dfrac{{{x}^{2}}}{{{a}^{2}}}-\dfrac{{{y}^{2}}}{{{b}^{2}}}=1$(b > 0) subtend a right angle at the center of the hyperbola, then it always touches a fixed circle whose:
\[\begin{align}
  & A.\text{ Radius is }\dfrac{ab}{\sqrt{b-2a}} \\
 & B.\text{ Radius is }\dfrac{ab}{\sqrt{{{b}^{2}}-{{a}^{2}}}} \\
 & C.\text{ Center is }\left( 0,0 \right) \\
 & D.\text{ Center is at the center of the hyperbola} \\
\end{align}\]

Hint: In this question, we are given that variable straight line $x\cos \alpha +y\sin \alpha =p$ is a chord of the hyperbola $\dfrac{{{x}^{2}}}{{{a}^{2}}}-\dfrac{{{y}^{2}}}{{{b}^{2}}}=1$(b > 0) and it subtend right angle at center of hyperbola. We have to find the radius and center of the fixed circle always touching it. Here we will first obtain the equation of the pair of straight line passing through origin and point of intersection of variable chord and hyperbola and then apply the condition of $\text{coefficient of }{{x}^{2}}+\text{coefficient of }{{y}^{2}}=0$ to obtain the value of p. Since p is length of perpendicular from origin on the given variable line, it will become equal to radius of circle and thus radius and origin can be obtained.

Complete step by step answer:
We are given the equation of hyperbola as
\[\dfrac{{{x}^{2}}}{{{a}^{2}}}-\dfrac{{{y}^{2}}}{{{b}^{2}}}=1\left( b\text{ }>\text{ }0 \right)\cdots \cdots \cdots \cdots \left( 1 \right)\]
Equation of variable straight line which is chord to hyperbola is given as
\[x\cos \alpha +y\sin \alpha =p\cdots \cdots \cdots \cdots \left( 2 \right)\]
Now, let us find equation of straight line passing through origin and points of intersection of variable chord and hyperbola using (1) and (2), from (2)
\[\dfrac{x\cos \alpha +y\sin \alpha }{p}=1\]
Putting this values of 1 in (1) we get:
\[\dfrac{{{x}^{2}}}{{{a}^{2}}}-\dfrac{{{y}^{2}}}{{{b}^{2}}}={{\left( \dfrac{x\cos \alpha +y\sin \alpha }{p} \right)}^{2}}\]
As we know, ${{\left( a+b \right)}^{2}}={{a}^{2}}+{{b}^{2}}+2ab$ so applying that, we get:
\[\dfrac{{{x}^{2}}}{{{a}^{2}}}-\dfrac{{{y}^{2}}}{{{b}^{2}}}=\dfrac{{{x}^{2}}{{\cos }^{2}}\alpha }{{{p}^{2}}}+\dfrac{{{y}^{2}}{{\sin }^{2}}\alpha }{{{p}^{2}}}+\dfrac{2x\cos \alpha (y\sin \alpha) }{p^2}\]
Rearranging the terms we get:
\[{{x}^{2}}\left( \dfrac{1}{{{a}^{2}}}-\dfrac{{{\cos }^{2}}\alpha }{{{p}^{2}}} \right)-{{y}^{2}}\left( \dfrac{1}{{{b}^{2}}}-\dfrac{{{\sin }^{2}}\alpha }{{{p}^{2}}} \right)-\dfrac{2x\cos \alpha (y\sin \alpha )}{p^2}=0\]
This is the equation of a pair of straight lines passing through origin and points of intersection of variable chord and hyperbola.
As we are given that, these pair of line form right angle at center of hyperbola, therefore, $\text{coefficient of }{{x}^{2}}+\text{coefficient of }{{y}^{2}}$ will be equal to 0.
Coefficient of ${{x}^{2}}$ is given as $\left( \dfrac{1}{{{a}^{2}}}-\dfrac{{{\cos }^{2}}\alpha }{{{p}^{2}}} \right)$.
Coefficient of ${{y}^{2}}$ is given as $\left( \dfrac{1}{{{b}^{2}}}-\dfrac{{{\sin }^{2}}\alpha }{{{p}^{2}}} \right)$.
Using these coefficient we get:
\[\begin{align}
  & \dfrac{1}{{{a}^{2}}}-\dfrac{{{\cos }^{2}}\alpha }{{{p}^{2}}}-\left( \dfrac{1}{{{b}^{2}}}-\dfrac{{{\sin }^{2}}\alpha }{{{p}^{2}}} \right)=0 \\
 & \Rightarrow \dfrac{1}{{{a}^{2}}}-\dfrac{1}{{{b}^{2}}}-\dfrac{{{\cos }^{2}}\alpha }{{{p}^{2}}}-\dfrac{{{\sin }^{2}}\alpha }{{{p}^{2}}}=0 \\
 & \Rightarrow \dfrac{1}{{{a}^{2}}}-\dfrac{1}{{{b}^{2}}}-\left( \dfrac{{{\cos }^{2}}\alpha +{{\sin }^{2}}\alpha }{{{p}^{2}}} \right)=0 \\
\end{align}\]
We know, ${{\cos }^{2}}\theta +{{\sin }^{2}}\theta =1$ therefore equation becomes
\[\begin{align}
  & \Rightarrow \dfrac{1}{{{a}^{2}}}-\dfrac{1}{{{b}^{2}}}-\dfrac{1}{{{p}^{2}}}=0 \\
 & \Rightarrow \dfrac{1}{{{p}^{2}}}=\dfrac{1}{{{a}^{2}}}-\dfrac{1}{{{b}^{2}}} \\
\end{align}\]
Now, let us calculate value of p, we get:
\[\Rightarrow \dfrac{1}{{{p}^{2}}}=\dfrac{{{b}^{2}}-{{a}^{2}}}{{{a}^{2}}{{b}^{2}}}\]
Taking reciprocal both sides
\[\Rightarrow \dfrac{1}{{{p}^{2}}}=\dfrac{{{a}^{2}}{{b}^{2}}}{{{b}^{2}}-{{a}^{2}}}\]
Taking square root on both sides, we get:
\[\Rightarrow p=\dfrac{ab}{\sqrt{{{b}^{2}}-{{a}^{2}}}}\]
Now, we know that p is the length of perpendicular from origin on the line $x\cos \alpha +y\sin \alpha =p$ therefore, this line touches the circle with center at origin and radius equal to p.
So, radius becomes equal to $\dfrac{ab}{\sqrt{{{b}^{2}}-{{a}^{2}}}}$.
Hence, center of the circle is origin that is (0,0) and also center of hyperbola is at origin and radius of circle is $\dfrac{ab}{\sqrt{{{b}^{2}}-{{a}^{2}}}}$.

So, the correct answer is “Option B, C and D”.

Note: Students should carefully put values from one equation to another as calculations are complex in these sums. Students should keep in mind the property that, $\text{coefficient of }{{x}^{2}}+\text{coefficient of }{{y}^{2}}=0$. If a pair of lines subtend the right angle to origin while calculating the value of p, we did not consider the negative value because radius cannot be negative.