A decomposition of a representation $R:G\to GL(n,\mathbf{C})$ is a splitting ${\mathbf{C}}^{n}={V}_{1}\oplus \mathrm{\cdots}\oplus {V}_{k}$ where each ${V}_{i}\subset {\mathbf{C}}^{n}$ is a subrepresentation of ${\mathbf{C}}^{n}$ , that is $R(g)v\in {V}_{i}$ whenever $v\in {V}_{i}$ . In this case, each matrix $R(g)$ is blockdiagonal (if we write it with respect to a basis of vectors from ${V}_{1},\mathrm{\dots},{V}_{k}$ , ordered so that basis vectors from ${V}_{i}$ before basis vectors from ${V}_{j}$ if $$ ): $$R(g)=\left(\begin{array}{ccc}\hfill {R(g)}_{{V}_{1}}\hfill & \hfill 0\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill \mathrm{\ddots}\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill 0\hfill & \hfill {R(g)}_{{V}_{k}}\hfill \end{array}\right).$$ We will write this as $R={R}_{{V}_{1}}\oplus {\mathrm{\cdots}R}_{{V}_{k}}$ .
Complete reducibility
Complete reducibility
Decomposition
One of the key techniques for studying representations is to break them down into smaller subrepresentations.
In such a decomposition, we would like the pieces ${V}_{i}$ to be as ``small'' as possible, because then our matrices will be concentrated very close to the diagonal and most entries will be zero. These ``smallest pieces'' are called irreducible representations:
A subrepresentation $V\subset {\mathbf{C}}^{n}$
is called
Orthogonal complements: idea
So we would like to decompose our representations as direct sums of irreducible subrepresentations; in this case we say our representation is completely reducible. That's not always possible (we'll see an example in one of the exercises). However, we will focus on groups for which it is possible, namely the compact groups (matrix groups where the matrix entries are bounded). As a first step, we prove:
If ${\mathbf{C}}^{n}$ admits an invariant Hermitian inner product for the representation $R:G\to GL(n,\mathbf{C})$ then $R$ can be decomposed into irreducible summands.
I'll give you the idea of the proof before defining what an invariant Hermitian inner product is; suffice it to say that it's something like a dot product.
If ${\mathbf{C}}^{n}$ is not irreducible then it contains a subrepresentation $U$ . The orthogonal complement ${U}^{\u27c2}$ of $U$ with respect to the Hermitian inner product will also be a subrepresentation and ${\mathbf{C}}^{n}=U\oplus {U}^{\u27c2}$ . So if ${\mathbf{C}}^{n}$ is not itself irreducible then it can be decomposed as a direct sum of subrepresentations. Applying the same reasoning to the summands, if either is not irreducible, we can decompose further; and so on and so on.
Eventually this process terminates because the dimension of the summands decreases each time you decompose. Either you hit an irreducible summand, or you keep going all the way down and find a 1dimensional summand, but 1dimensional representations are automatically irreducible: they have no proper subspaces, let alone proper subrepresentations.
I'm not saying you always break up into 1dimensional pieces, but that these provide a "safetyblanket": if you get all the way down to 1d then you're guaranteed to be irreducible.
To complete the proof, it remains to:

define the term "invariant Hermitian inner product",

define the orthogonal complement of a subspace with respect to an invariant Hermitian inner product,

prove that the orthogonal complement of a subrepresentation is a subrepresentation.
Hermitian inner products
A Hermitian inner product is a map $\u27e8,\u27e9:{\mathbf{C}}^{n}\times {\mathbf{C}}^{n}\to \mathbf{C}$ (i.e. it eats two complex vectors $v$ and $w$ and returns a complex number $\u27e8v,w\u27e9$ ) such that:

$\u27e8v,v\u27e9$ is real and positive unless $v=0$ .

$\u27e8v,u\u27e9=\overline{\u27e8u,v\u27e9}$ for all $u,v\in {\mathbf{C}}^{n}$ ,

$\u27e8u,a{v}_{1}+b{v}_{2}\u27e9=a\u27e8u,{v}_{1}\u27e9+b\u27e8u,{v}_{2}\u27e9$ for all $u,v\in {\mathbf{C}}^{n}$ and $a,b\in \mathbf{C}$ ,

$\u27e8a{u}_{1}+b{u}_{2},v\u27e9=\overline{a}\u27e8{u}_{1},v\u27e9+\overline{b}\u27e8{u}_{2},v\u27e9$ .
The final condition actually follows from (1) and (2), so we don't really need to take it as an axiom.
This notion is supposed to be a replacement of "dot product" that works well with complex vectors. The problem with just taking the dot product $v\cdot w$ of complex vectors is that $v\cdot v=\sum {v}_{k}^{2}$ is a complex number, and we would like the length of $v$ to be $\sqrt{v\cdot v}$ , which would then also be a complex number. If instead we take $\u27e8v,v\u27e9=\sum {\overline{v}}_{k}{v}_{k}$ then we get a real number which is positive unless $v=0$ . The axioms above are intended to capture the important properties of the "standard Hermitian inner product" $\u27e8v,w\u27e9=\sum {\overline{v}}_{k}{w}_{k}$ .
When orthogonal complements are subrepresentations
A Hermitian inner product $\u27e8,\u27e9$ is invariant for a representation $R:G\to GL(n,\mathbf{C})$ if $$\u27e8R(g)v,R(g)w\u27e9=\u27e8v,w\u27e9$$ for all $g\in G$ and $v,w\in {\mathbf{C}}^{n}$ .
Given a representation $R:G\to GL(n,\mathbf{C})$
, a subrepresentation $U\subset {\mathbf{C}}^{n}$
, and an invariant Hermitian inner product on ${\mathbf{C}}^{n}$
, the
If $w\in {U}^{\u27c2}$ , we want to show that $R(g)w\in {U}^{\u27c2}$ for all $g\in G$ . To see this, we need to compute $\u27e8u,R(g)w\u27e9$ and see that it's zero.
Using invariance, we get $$\u27e8u,R(g)w\u27e9=\u27e8R({g}^{1})u,R({g}^{1})R(g)w\u27e9=\u27e8R({g}^{1})u,w\u27e9.$$ Since $U$ is a subrepresentation, $R({g}^{1})u\in U$ . Since $w\in {U}^{\u27c2}$ , we therefore get $\u27e8R({g}^{1})u,w\u27e9=0$ as desired.
Preclass exercise
Let $(\mathbf{R},+)$ denote the group of real numbers with addition. Why is the representation $R(x)=\left(\begin{array}{cc}\hfill 1\hfill & \hfill x\hfill \\ \hfill 0\hfill & \hfill 1\hfill \end{array}\right)$ not irreducible? Can you find a decomposition of it?