Let \[a, b\] and \[c\] be the distinct non-negative numbers. If the vectors \[a \widehat i + a \widehat j + c \widehat k, \widehat i + \widehat k\] and \[c \widehat i + c \widehat j + b \widehat k\] lie in a plane. Then which of the following statement is true for \[c\]?
A. The harmonic mean of \[a\] and \[b\]
B. equal to zero
C. The arithmetic mean of \[a\] and \[b\]
D. The geometric mean of \[a\] and \[b\]

Hint: First, apply the determinant condition of the coplanar vectors. Solve the determinant of the coefficients of the given vectors and get the equation that contains \[a, b\] and \[c\]. Observe the situations of \[a, b\] and \[c\] in the equation and verify the statement.

Formula used:
Three vectors \[x, y\] and \[z\] lie in a plane, then they are coplanar and their scalar triple product is zero.
 Means, \[\left[ {x y z} \right] = 0\].
If three numbers \[p, q\] and \[r\] are in the geometric progression, then \[{q^2} = pr\].

Complete step by step solution:
Given:
vectors \[a \widehat i + a \widehat j + c \widehat k, \widehat i + \widehat k\] and \[c \widehat i + c \widehat j + b \widehat k\] lie in a same plane.
Since vectors lie in the same plane. So, they are coplanar.
Hence, their scalar triple product is zero.
Then,
\[\left| {\begin{array}{*{20}{c}}a&a&c\\1&0&1\\c&c&b\end{array}} \right| = 0\]
Solve the above determinant.
\[a\left( {0 - c} \right) - a\left( {b - c} \right) + c\left( {c - 0} \right) = 0\]
\[ \Rightarrow - ac - ab + ac + {c^2} = 0\]
\[ \Rightarrow - ab + {c^2} = 0\]
\[ \Rightarrow {c^2} = ab\]
We know that if three numbers \[p, q\] and \[r\] are in the geometric progression, then \[{q^2} = pr\].
Clearly, from the above equation, we get
\[a, c\] and \[b\] are in the geometric progression.
Thus, \[c\] is the geometric mean of \[a\] and \[b\].
Hence the correct option is D.

Note: Students often get confused about the coplanarity of the vectors in 3-dimensional geometry.
If three vectors \[\overrightarrow a :{a_1}\widehat i + {b_1}\widehat j + {c_1}\widehat k\] , \[\overrightarrow b :{a_2}\widehat i + {b_2}\widehat j + {c_2}\widehat k\] and \[\overrightarrow c :{a_3}\widehat i + {b_3}\widehat j + {c_3}\widehat k\] are coplanar or lie in a plane, then \[\left[ {\overrightarrow a \overrightarrow b \overrightarrow c } \right] = 0\].
Means \[\left| {\begin{array}{*{20}{c}}{{a_1}}&{{b_1}}&{{c_1}}\\{{a_2}}&{{b_2}}&{{c_2}}\\{{a_3}}&{{b_3}}&{{c_3}}\end{array}} \right| = 0\]