A general equation of second degree \[a{{x}^{2}}+2hxy+b{{y}^{2}}+2gx+2fy+c=0\] represents a pair of straight lines if
1. \[{{h}^{2}}=2ab\]
2. \[{{h}^{2}}>2ab\]
3. \[\left| \begin{matrix}
   a & h & g \\
   h & b & f \\
   g & f & c \\
\end{matrix} \right|=0\]
4. None of these

Hint: At first we take the general equation that is \[S=a{{x}^{2}}+2hxy+b{{y}^{2}}+2gx+2fy+c=0\] which represents a straight line for that we have to consider two line equation \[{{l}_{1}}x\text{ }+\text{ }{{m}_{1}}y\text{ }+~{{n}_{1~}}=~0\text{ }\]and\[{{l}_{2}}x\text{ }+~{{m}_{2}}y\text{ }+~{{n}_{2~}}=~0\]. Therefore, \[({{l}_{1}}x\text{ }+\text{ }{{m}_{1}}y\text{ }+~{{n}_{1~}})({{l}_{2}}x\text{ }+~{{m}_{2}}y\text{ }+~n)=0\] and simplify further in this problem.

Complete step by step answer:
Suppose \[S=0\] represents pair of straight line but there are two line of equation are:
\[{{l}_{1}}x\text{ }+\text{ }{{m}_{1}}y\text{ }+~{{n}_{1~}}=0---(1)\]
\[{{l}_{2}}x\text{ }+~{{m}_{2}}y\text{ }+~{{n}_{2~}}=~0---(2)\]
It represents a pair of straight line that means
\[S=({{l}_{1}}x\text{ }+\text{ }{{m}_{1}}y\text{ }+~{{n}_{1~}})({{l}_{2}}x\text{ }+~{{m}_{2}}y\text{ }+~n_2)\]
By simplifying this above equation we get:
\[S={{l}_{1}}{{l}_{2}}{{x}^{2}}\text{+}{{l}_{1}}~{{m}_{2}}xy+{{l}_{1}}{{n}_{2}}x\text{+}{{l}_{2}}{{m}_{1}}xy+{{m}_{1}}{{m}_{2}}{{y}^{2}}\text{+}{{m}_{1}}{{n}_{2}}y\text{+}{{l}_{2}}{{n}_{1~}}x+{{m}_{2}}{{n}_{1~}}y\text{ }+~{{n}_{2}}{{n}_{1~}}\]
By rearranging the term we get:
\[S={{l}_{1}}{{l}_{2}}{{x}^{2}}+{{m}_{1}}{{m}_{2}}{{y}^{2}}+{{l}_{1}}~{{m}_{2}}xy+{{l}_{2}}{{m}_{1}}xy+{{l}_{1}}{{n}_{2}}x+{{l}_{2}}{{n}_{1~}}x+{{m}_{1}}{{n}_{2}}y+{{m}_{2}}{{n}_{1~}}y+~{{n}_{2}}{{n}_{1~}}\]
By solving further we get:
\[S={{l}_{1}}{{l}_{2}}{{x}^{2}}+{{m}_{1}}{{m}_{2}}{{y}^{2}}+({{l}_{1}}~{{m}_{2}}+{{l}_{2}}{{m}_{1}})xy+({{l}_{1}}{{n}_{2}}+{{l}_{2}}{{n}_{1~}})x+({{m}_{1}}{{n}_{2}}+{{m}_{2}}{{n}_{1~}})y+~{{n}_{2}}{{n}_{1~}}--(3)\]
By comparing the general equation that is \[S=a{{x}^{2}}+2hxy+b{{y}^{2}}+2gx+2fy+c\] with equation \[(3)\]
We get,
\[a={{l}_{1}}{{l}_{2}},\,\,b={{l}_{1}}{{l}_{2}},\,\,2h={{l}_{1}}~{{m}_{2}}+{{l}_{2}}{{m}_{1}},\,\,c={{n}_{2}}{{n}_{1~}},\,\,2g={{l}_{1}}{{n}_{2}}+{{l}_{2}}{{n}_{1~}},\,\,2f={{m}_{1}}{{n}_{2}}+{{m}_{2}}{{n}_{1~}}\]
Now, \[8fgh\] can be written as \[(2f)(2g)(2h)\]
\[\therefore 8fgh=(2f)(2g)(2h)\]
Substitute the value of h, g and f we get;
\[\begin{align}
  & 8fgh=({{m}_{1}}{{n}_{2}}+{{m}_{2}}{{n}_{1~}})({{l}_{1}}{{n}_{2}}+{{l}_{2}}{{n}_{1~}})({{l}_{1}}~{{m}_{2}}+{{l}_{2}}{{m}_{1}}) \\
 & \\
\end{align}\]
After simplifying the bracket we get:
\[8fgh={{l}_{1}}{{l}_{2}}({{m}_{1}}^{2}{{n}_{2}}^{2}+{{m}_{2}}^{2}{{n}_{1~}}^{2})+{{m}_{1}}{{m}_{2}}({{l}_{1}}^{2}{{n}_{2}}^{2}+{{l}_{2}}^{2}{{n}_{1~}}^{2})+{{n}_{1~}}{{n}_{2~}}({{l}_{1}}^{2}~{{m}_{2}}^{2}+{{l}_{2}}^{2}{{m}_{1}}^{2})+2{{l}_{1}}{{l}_{2}}{{m}_{1}}{{m}_{2}}{{n}_{1~}}{{n}_{2~}}\]
By arranging the term we get:
\[8fgh={{l}_{1}}{{l}_{2}}\left[ ({{m}_{1}}^{2}{{n}_{2}}^{2}+{{m}_{2}}^{2}{{n}_{1~}}^{2}+2{{m}_{1}}{{m}_{2}}{{n}_{1~}}{{n}_{2~}})-2{{m}_{1}}{{m}_{2}}{{n}_{1~}}{{n}_{2~}} \right]+{{m}_{1}}{{m}_{2}}\left[ ({{l}_{1}}^{2}{{n}_{2}}^{2}+{{l}_{2}}^{2}{{n}_{1~}}^{2}+2{{l}_{1}}{{l}_{2}}{{n}_{1~}}{{n}_{2~}})-2{{l}_{1}}{{l}_{2}}{{n}_{1~}}{{n}_{2~}} \right]+{{n}_{1~}}{{n}_{2~}}\left[ ({{l}_{1}}^{2}~{{m}_{2}}^{2}+{{l}_{2}}^{2}{{m}_{1}}^{2}+2{{l}_{1}}{{l}_{2}}{{m}_{1}}{{m}_{2}}{{n}_{1~}})-2{{l}_{1}}{{l}_{2}}{{m}_{1}}{{m}_{2}}{{n}_{1~}} \right]+2{{l}_{1}}{{l}_{2}}{{m}_{1}}{{m}_{2}}{{n}_{1~}}{{n}_{2~}}\]
As we know the value of \[a,\,\,b,\,\,c\,\,\]substitute in these equation we get:
And apply the formula for \[{{(a+b)}^{2}}={{a}^{2}}+2ab+{{b}^{2}}\]
\[8fgh=a\left[ {{({{m}_{1}}{{n}_{2}}+{{m}_{2}}{{n}_{1~}})}^{2}}-2{{m}_{1}}{{m}_{2}}{{n}_{1~}}{{n}_{2~}} \right]+b\left[ {{({{l}_{1}}{{n}_{2}}+{{l}_{2}}{{n}_{1~}})}^{2}}-2{{l}_{1}}{{l}_{2}}{{n}_{1~}}{{n}_{2~}} \right]+c\left[ {{({{l}_{1}}~{{m}_{2}}+{{l}_{2}}{{m}_{1}})}^{2}}-2{{l}_{1}}{{l}_{2}}{{m}_{1}}{{m}_{2}}{{n}_{1~}} \right]+2{{l}_{1}}{{l}_{2}}{{m}_{1}}{{m}_{2}}{{n}_{1~}}{{n}_{2~}}\]
By substituting the value of \[h,\,\,f,\,\,g\,\,\]in this above equation:
\[8fgh=a(4{{f}^{2}}-2bc)+b(4{{g}^{2}}-2ca)+c(4{{h}^{2}}-2ab)+2abc\]
By simplifying this we get:
\[8fgh=4(a{{f}^{2}}+b{{g}^{2}}+c{{h}^{2}}-abc)\]
Divide by 4 on both sides we get:
\[2fgh=a{{f}^{2}}+b{{g}^{2}}+c{{h}^{2}}-abc\]
By rearranging the term we get:
\[2fgh=a{{f}^{2}}+b{{g}^{2}}+c{{h}^{2}}-abc\] or \[abc+2fgh-a{{f}^{2}}-b{{g}^{2}}-c{{h}^{2}}=0\]
If we notice this above equation then it is a determinant of the matrix which is generally it is represented as \[\Delta \] that is the above equation becomes \[\Delta =0\]
This is the required condition for \[S=a{{x}^{2}}+2hxy+b{{y}^{2}}+2gx+2fy+c=0\] represents a straight line
\[\Delta =0\] can also be written as:
\[\left| \begin{matrix}
   a & h & g \\
   h & b & f \\
   g & f & c \\
\end{matrix} \right|=0\]
Because \[\Delta =\left| \begin{matrix}
   a & h & g \\
   h & b & f \\
   g & f & c \\
\end{matrix} \right|=abc+2fgh-a{{f}^{2}}-b{{g}^{2}}-c{{h}^{2}}=0\]

So, the correct answer is “Option 3”.

Note: In this particular problem we have to remember that \[S=0\] only for the equation which represents a pair of straight lines. And its determinant will be 0. So, while simplifying the brackets, don't make silly mistakes. So the above solution is referred for such types of problems.