I think it might be easier for me to just go through this problem and you ask any questions you have about what I've done.

First, we need to show that $P$ is a group under multiplication. For this to be true $P$ has to satisfy the 4 group axioms.

1. Closure -------

Given 2 elements of $X,Y \in P$ we have to show that $X \cdot Y \in P$

$

\begin{pmatrix}

x_1 & x_2 & x_3\\

0 & x_4 & x_5 \\

0 & x_6 & x_7

\end{pmatrix}

\begin{pmatrix}

y_1 & y_2 & y_3\\

0 & y_4 & y_5 \\

0 & y_6 & y_7

\end{pmatrix}

=

\begin{pmatrix}

x_1 y_1 & * & * \\

0 & x_4 y_4 + y_5 y_6 & x_4 y_5 + x_5 y_7 \\

0 & x_6 y_4 + x_7 y_6 & x_6 y_5 + x_7y_7

\end{pmatrix}

=XY

$

For $XY$ to be in $P$ we have to show that

$\text{det}

\begin{pmatrix}

x_4 y_4 + y_5 y_6 & x_4 y_5 + x_5 y_7 \\

x_6 y_4 + x_7 y_6 & x_6 y_5 + x_7y_7

\end{pmatrix}

\neq 0

$

$\text{det}

\begin{pmatrix}

x_4 y_4 + y_5 y_6 & x_4 y_5 + x_5 y_7 \\

x_6 y_4 + x_7 y_6 & x_6 y_5 + x_7y_7

\end{pmatrix}

= (x_4 y_4 + y_5 y_6)(x_6 y_5 + x_7y_7) - (x_4 y_5 + x_5 y_7)(x_6 y_4 + x_7 y_6)

$

After multiplying this out and pulling like terms you'll find that

$

(x_4 y_4 + y_5 y_6)(x_6 y_5 + x_7y_7) - (x_4 y_5 + x_5 y_7)(x_6 y_4 + x_7 y_6) = (x_4x_7 - x_5x_6)(y_4y_7-y_5y_6) =

$

$

\text{det}

\begin{pmatrix}

x_4 & x_5 \\

x_6 & x_7

\end{pmatrix}

\text{det}

\begin{pmatrix}

y_4 & y_5 \\

y_6 & y_7

\end{pmatrix}

$

$X$ and $Y$ are in $P$, so by definition these determinants are nonzero, so their product is nonzero.

Since both $X$ and $Y$ are in $P$, $x_1 \neq 0$ and $y_1 \neq 0$ so $x_1 y_1 \neq 0$. Fields are closed under addition and multiplication so all the elements of $XY$ are in $\mathbb{F}$.

Lastly, since

$\text{det}

\begin{pmatrix}

x_4 y_4 + y_5 y_6 & x_4 y_5 + x_5 y_7 \\

x_6 y_4 + x_7 y_6 & x_6 y_5 + x_7y_7

\end{pmatrix}

=

\text{det}

\begin{pmatrix}

x_4 & x_5 \\

x_6 & x_7

\end{pmatrix}

\text{det}

\begin{pmatrix}

y_4 & y_5 \\

y_6 & y_7

\end{pmatrix} \neq 0

$

We have that $XY$ must also be in $P$

2. Associativity ----------

Inherent in matrix multiplication.

3. Identity --------------

The identity in matrix multiplication,

$

\begin{pmatrix}

1 & 0 & 0\\

0 & 1 & 0 \\

0 & 0 & 1

\end{pmatrix}

$

Is in $P$, I'll leave that to you to check.

4. Inverse ------------------

We need to show every element of $P$ has a multiplicative inverse.

For a matrix to have an inverse, it needs to be invertible, we means it's determinant is nonzero. We'll use $X$ as an arbitrary element of $P$ again.

$\text{det}

\begin{pmatrix}

x_1 & x_2 & x_3\\

0 & x_4 & x_5 \\

0 & x_6 & x_7

\end{pmatrix}

=

x_1 \text{det}

\begin{pmatrix}

x_4 & x_5 \\

x_6 & x_7

\end{pmatrix}

-0 + 0

$

Since $x_1 \neq 0$ and that determinant is nonzero, all the elements of $P$ are invertible.

Since all 4 axioms are satisfied, $P$ is a group under matrix multiplication.