An automorphism is a way of relabeling the elements of a group that the multiplication table can't detect. Every group has at least one (the identity map), and most have more. On this page we'll collect all of them into a group of their own, $\operatorname{Aut}(G)$, and find out that conjugation hands us a big supply of them for free. Then we'll compute $\operatorname{Aut}(G)$ for a handful of groups, and use it to prove that some orders only allow Abelian groups.
On Homomorphisms, an automorphism of $G$ was an isomorphism $\phi : G \rightarrow G$. We met a few: $k \mapsto ak$ on $\mathbb{Z}/n\mathbb{Z}$ when $(a, n) = 1$, inversion $g \mapsto g^{-1}$ (only when $G$ is Abelian), and conjugation $\tau_g(h) = ghg^{-1}$, which is an automorphism of every group.
An automorphism of $G$ is a bijective homomorphism from $G$ to itself. Instead of looking at them one at a time, put them all in one set.
The automorphism group of a group $G$ is $\operatorname{Aut}(G) = \{\sigma : G \rightarrow G \mid \sigma \text{ is an automorphism}\}$, with composition of maps as the operation.
The name promises a group, so let's check that we've earned it.
$\operatorname{Aut}(G)$ is a subgroup of $S_G$, the group of all bijections $G \rightarrow G$.
Proof. $\operatorname{Aut}(G) \subseteq S_G$, and $\operatorname{id}_G \in \operatorname{Aut}(G)$. If $\sigma, \rho \in \operatorname{Aut}(G)$ then $\sigma \circ \rho$ is a bijection, and $\forall g, h \in G$, $(\sigma \circ \rho)(gh) = \sigma(\rho(g)\rho(h)) = \sigma(\rho(g))\,\sigma(\rho(h))$, so $\sigma \circ \rho \in \operatorname{Aut}(G)$. Finally, the inverse of an isomorphism is an isomorphism (Property 4 on Homomorphisms), so $\sigma^{-1} \in \operatorname{Aut}(G)$. $\blacksquare$
An automorphism has to send the identity to the identity, and it has to send each element to one of the same order. Those two facts are most of what you need to hunt automorphisms down by hand.
On Group Actions, every action $G \curvearrowright A$ came with its permutation representation $\varphi : G \rightarrow S_A$, $\varphi(g) = \pi_g$, where $\pi_g(a) = g \cdot a$. That page's Sneak Peek mentioned one more action, $G$ acting on itself by conjugation, which The Class Equation & Burnside's Lemma studies orbit by orbit. Here's the version where $G$ conjugates a normal subgroup.
If $H \unlhd G$, then conjugating an element of $H$ by any $g \in G$ lands back in $H$. So each $g$ shuffles $H$, and the shuffle respects multiplication.
If $H \unlhd G$, then:
Proof.
Part (ii) is the same computation that showed $\tau_g$ is an automorphism on Homomorphisms, just restricted to $H$. And because $\operatorname{Aut}(H) \le S_H$, the permutation representation $\varphi : G \rightarrow S_H$ lands inside $\operatorname{Aut}(H)$, so we may as well think of it as a homomorphism $\varphi : G \rightarrow \operatorname{Aut}(H)$.
Now feed this to the 1st Isomorphism Theorem: $G/\ker \varphi$ is isomorphic to the image, so $G/C_G(H)$ sits inside $\operatorname{Aut}(H)$. Even better, we don't need $H$ to be normal in all of $G$.
If $G$ is a group and $H \le G$, then $N_G(H)/C_G(H)$ is isomorphic to a subgroup of $\operatorname{Aut}(H)$.
Proof. Every subgroup is normal in its own normalizer: $\forall g \in N_G(H)$, $gHg^{-1} = H$, so $H \unlhd N_G(H)$. Apply the proposition with $N_G(H)$ in the role of $G$. It gives a homomorphism $\varphi : N_G(H) \rightarrow \operatorname{Aut}(H)$ with kernel $$C_{N_G(H)}(H) = C_G(H) \cap N_G(H) = C_G(H),$$ because $C_G(H) \subseteq N_G(H)$ (Normal Subgroups, First Observations). By the 1st Isomorphism Theorem, $N_G(H)/C_G(H) \cong \varphi(N_G(H)) \le \operatorname{Aut}(H)$. $\blacksquare$
You'll often see this corollary stated with the hypothesis $H \unlhd G$. That isn't wrong, it's just more than you need. All the proof uses is $H \unlhd N_G(H)$, which is true for every subgroup. When $H \unlhd G$ you get $N_G(H) = G$, and the corollary reads $G/C_G(H) \hookrightarrow \operatorname{Aut}(H)$. That special case is the one Example 7 below uses.
Take $H = G$ in the proposition. $G$ is always normal in itself, so $G$ acts on itself by conjugation, and each $g$ gives the automorphism $\tau_g(h) = ghg^{-1}$.
Let $G$ act on itself by conjugation and let $\varphi : G \rightarrow \operatorname{Aut}(G)$ be the associated permutation representation. Define $\operatorname{Inn}(G) = \operatorname{Im}(\varphi) = \{\tau_g : g \in G\}$, the group of inner automorphisms of $G$.
Inner automorphisms: the call is coming from inside the group.
By part (iii), $\ker \varphi = C_G(G) = Z(G)$, the center. The 1st Isomorphism Theorem does the rest.
$\operatorname{Inn}(G) \cong G/Z(G)$.
So two elements $g, g'$ give the same inner automorphism exactly when $gZ(G) = g'Z(G)$. Multiplying $g$ by something central doesn't change how it conjugates: if $z \in Z(G)$ then $(gz)h(gz)^{-1} = g(zhz^{-1})g^{-1} = ghg^{-1}$. In the widget below, $G = D_8 = \langle r, s \mid r^4 = s^2 = 1, rs = sr^{-1} \rangle$, whose center is $\langle r^2 \rangle = \{1, r^2\}$ (Example 6 below checks this). Tap any $g$ and compare $\tau_g$ with $\tau_{gr^2}$.
Eight elements, only four different pictures, one for each coset of $Z(D_8)$. The elements each $\tau_g$ leaves alone (shaded) are exactly the ones that commute with $g$, which is $C_{D_8}(g)$. And the two elements whose picture has no arrows at all, $1$ and $r^2$, make up the kernel, which is the center.
Inner automorphisms are the easy ones to find. The rest of $\operatorname{Aut}(G)$ takes more work, and Abelian groups are the extreme case: no inner automorphisms at all beyond the identity.
Suppose $G$ is Abelian. Find $\operatorname{Inn}(G)$.
$Z(G) = G$, so $\operatorname{Inn}(G) \cong G/Z(G) = G/G = \{1\}$. Directly: $\tau_g(h) = ghg^{-1} = hgg^{-1} = h$, so every $\tau_g$ is the identity map.
So for an Abelian group, every interesting automorphism is an outer one, and we have to find them some other way. Cyclic groups are the place to start, because a homomorphism out of a cyclic group is pinned down by where it sends a generator.
Let $G = \mathbb{Z}/n\mathbb{Z}$. Show that $\operatorname{Aut}(G) \cong (\mathbb{Z}/n\mathbb{Z})^\times$.
Proof. Any homomorphism $\tau : \mathbb{Z}/n\mathbb{Z} \rightarrow \mathbb{Z}/n\mathbb{Z}$ is uniquely determined by $\tau(1)$ (Property 6a on Homomorphisms), and it is $\tau(k) = k\,\tau(1)$. Such a homomorphism is an isomorphism $\iff (\tau(1), n) = 1$ (the multiplication-by-$a$ example on Homomorphisms). Therefore there are $|(\mathbb{Z}/n\mathbb{Z})^\times|$ distinct automorphisms, one for each primitive residue class.
Furthermore, the map $\gamma : \operatorname{Aut}(\mathbb{Z}/n\mathbb{Z}) \rightarrow (\mathbb{Z}/n\mathbb{Z})^\times$ defined by $\gamma(\tau) = \tau(1)$ is an isomorphism. It's a bijection by what we just said. For the homomorphism property, let $\tau_1, \tau_2 \in \operatorname{Aut}(\mathbb{Z}/n\mathbb{Z})$. Then $$\gamma(\tau_1 \tau_2) = (\tau_1 \tau_2)(1) = \tau_1(\tau_2(1)) = \tau_1(\underbrace{1 + 1 + \dots + 1}_{\tau_2(1)\text{ times}}) = \underbrace{\tau_1(1) + \dots + \tau_1(1)}_{\tau_2(1)\text{ times}} = \tau_1(1)\,\tau_2(1). \ \checkmark \ \blacksquare$$
$|(\mathbb{Z}/n\mathbb{Z})^\times|$ is Euler's phi function from Modular Arithmetic, usually written $\varphi(n)$. On this page $\varphi$ is already busy being a permutation representation, so we'll write the totient out as $|(\mathbb{Z}/n\mathbb{Z})^\times|$. Either way, $|\operatorname{Aut}(\mathbb{Z}/n\mathbb{Z})|$ is the number of $a \in \{1, \dots, n\}$ with $(a, n) = 1$, and for a prime $p$ that's $p - 1$.
Notice what the example says and what it doesn't: $\mathbb{Z}/n\mathbb{Z}$ is cyclic, but its automorphism group doesn't have to be.
Find $\operatorname{Aut}(\mathbb{Z}/8\mathbb{Z})$. Is it cyclic?
By Example 2, $\operatorname{Aut}(\mathbb{Z}/8\mathbb{Z}) \cong (\mathbb{Z}/8\mathbb{Z})^\times = \{1, 3, 5, 7\}$ under multiplication mod $8$. The four automorphisms are $k \mapsto k$, $k \mapsto 3k$, $k \mapsto 5k$, $k \mapsto 7k$. Now square everything: $$1^2 = 1, \quad 3^2 = 9 \equiv 1, \quad 5^2 = 25 \equiv 1, \quad 7^2 = 49 \equiv 1 \pmod 8.$$ A group of order $4$ with no element of order $4$ isn't $C_4$, so $\operatorname{Aut}(\mathbb{Z}/8\mathbb{Z}) \cong V_4$. Not cyclic.
When is $\operatorname{Aut}(\mathbb{Z}/n\mathbb{Z}) \cong (\mathbb{Z}/n\mathbb{Z})^\times$ cyclic? Cyclic Groups quoted the Primitive Root Theorem: exactly when $n = 1, 2, 4, p^k$, or $2p^k$. So $\operatorname{Aut}(\mathbb{Z}/p\mathbb{Z}) \cong C_{p-1}$ for every prime $p$, and $n = 8$ is the first case that fails.
Next, products of copies of $\mathbb{Z}/p\mathbb{Z}$. Here the automorphisms turn out to be matrices. Write $\mathbb{F}_p$ for $\mathbb{Z}/p\mathbb{Z}$ when we're thinking of it as a field (a number system where you can add, subtract, multiply, and divide by anything nonzero), which it is, since every nonzero class mod $p$ has an inverse.
The general linear group $GL_n(\mathbb{F}_p) = \{n \times n \text{ matrices } A \text{ with entries in } \mathbb{F}_p \text{ and } \det(A) \neq 0\}$, under matrix multiplication.
Let $G = \mathbb{Z}/p\mathbb{Z} \times \dots \times \mathbb{Z}/p\mathbb{Z}$ ($n$ times). Show that $\operatorname{Aut}(G) \cong GL_n(\mathbb{F}_p)$, and that $$|GL_n(\mathbb{F}_p)| = (p^n - 1)(p^n - p)(p^n - p^2) \cdots (p^n - p^{n-1}).$$
Think of $G$ (written additively) as $\mathbb{F}_p^n$, the column vectors of length $n$ over $\mathbb{F}_p$. Then automorphisms of $G$ are exactly the invertible linear maps, which are exactly the invertible matrices.
Proof. Let $\sigma \in \operatorname{Aut}(G)$. For $k \in \{0, 1, \dots, p-1\}$ and $v \in G$, $\sigma(kv) = \sigma(v + \dots + v) = \sigma(v) + \dots + \sigma(v) = k\sigma(v)$. Every scalar in $\mathbb{F}_p$ is one of these $k$, so $\sigma$ is $\mathbb{F}_p$-linear. A linear bijection $\mathbb{F}_p^n \rightarrow \mathbb{F}_p^n$ is multiplication by a unique invertible matrix $A_\sigma$ (column $i$ is $\sigma(e_i)$), and invertible means $\det(A_\sigma) \neq 0$. Conversely, every $A \in GL_n(\mathbb{F}_p)$ gives a bijection $v \mapsto Av$ with $A(v + w) = Av + Aw$, which is an automorphism. Since $A_{\sigma \circ \rho} = A_\sigma A_\rho$, the bijection $\sigma \mapsto A_\sigma$ is an isomorphism $\operatorname{Aut}(G) \cong GL_n(\mathbb{F}_p)$.
Counting: $A$ is invertible exactly when its columns are linearly independent. Choose them one at a time. Column $1$ can be any nonzero vector: $p^n - 1$ choices. Column $2$ can be anything outside the span of column $1$, a $1$-dimensional space with $p$ vectors: $p^n - p$ choices. Column $k + 1$ avoids the $k$-dimensional span of the first $k$ columns, which has $p^k$ vectors: $p^n - p^k$ choices. Multiply. $\blacksquare$
$|GL_3(\mathbb{F}_2)| = 7 \cdot 6 \cdot 4 = 168$. That number is suspiciously famous. (It's the order of the second-smallest non-Abelian simple group, and this is that group.)
The smallest interesting case is $n = 2$, $p = 2$, which is a group we know very well.
Let $V_4 = \{e, a, b, c\}$ be the Klein 4-group. Show that $\operatorname{Aut}(V_4) \cong S_3$.
By hand. Every $\sigma \in \operatorname{Aut}(V_4)$ fixes $e$, so it permutes $\{a, b, c\}$. Conversely, let $\sigma$ be any bijection of $V_4$ with $\sigma(e) = e$. In $V_4$, the whole multiplication table is: $e$ is the identity, $x^2 = e$, and the product of two different elements of $\{a, b, c\}$ is the third. Check $\sigma(xy) = \sigma(x)\sigma(y)$:
So $\operatorname{Aut}(V_4)$ is exactly the set of permutations of $\{a, b, c\}$, and restricting $\sigma$ to $\{a, b, c\}$ is an isomorphism $\operatorname{Aut}(V_4) \cong S_{\{a, b, c\}} \cong S_3$.
By Example 4. $V_4 \cong \mathbb{Z}/2\mathbb{Z} \times \mathbb{Z}/2\mathbb{Z}$, so $\operatorname{Aut}(V_4) \cong GL_2(\mathbb{F}_2)$, which has order $(2^2 - 1)(2^2 - 2) = 3 \cdot 2 = 6$. It's non-Abelian (it contains $\begin{pmatrix} 0 & 1 \\ 1 & 0 \end{pmatrix}$ and $\begin{pmatrix} 1 & 1 \\ 0 & 1 \end{pmatrix}$, which don't commute), and the only non-Abelian group of order $6$ is $S_3$. $\checkmark$
Below, $e$ sits in the middle and $a, b, c$ sit on the corners of a triangle. Each edge carries one rule of the multiplication table: the two corners multiply to the third. Rearrange the tiles and the rules re-check themselves.
Any shuffle of the corners keeps every rule, which is the "by hand" proof in pictures: the triangle's six symmetries are $\operatorname{Aut}(V_4)$. Moving $e$ breaks rules immediately, because an automorphism has nowhere else to send the identity.
Can I take $\operatorname{Aut}$ of $\operatorname{Aut}$ of $\operatorname{Aut}$ forever? $\operatorname{Aut}(\mathbb{Z}/8\mathbb{Z}) \cong V_4$, then $\operatorname{Aut}(V_4) \cong S_3$, then $\operatorname{Aut}(S_3) \cong S_3$, then $\operatorname{Aut}(S_3) \cong S_3$, then... I'm stuck in a loop and it's somehow my fault.
Back to non-Abelian groups, where the inner automorphisms finally show up. Here's the computation behind the $D_8$ widget.
Let $G = D_8 = \langle r, s \mid r^4 = s^2 = 1, rs = sr^{-1} \rangle$. Find $Z(G)$ and $\operatorname{Inn}(G)$.
$Z(G) = \langle r^2 \rangle$. First, $r, r^{-1}, s \notin Z(G)$, since $sr \neq rs$ (that would force $r = r^{-1}$). Similarly no reflection $sr^i$ commutes with $r$. But $r^2 \in Z(G)$: it commutes with $r$, and $sr^2 = r^{-2}s = r^2 s$.
So $|\operatorname{Inn}(G)| = |G/Z(G)| = 8/2 = 4$. And $G/Z(G) \not\cong \mathbb{Z}/4\mathbb{Z}$, otherwise $G/Z(G)$ would be cyclic and $G$ would be Abelian. So $\operatorname{Inn}(G) \cong \mathbb{Z}/2\mathbb{Z} \times \mathbb{Z}/2\mathbb{Z}$. $\checkmark$
"$G/Z(G)$ cyclic $\implies$ $G$ Abelian" is the exercise from Quotient Groups. In the language of this page: a non-Abelian group can never have a cyclic group of inner automorphisms.
Now the payoff. Aut groups of small things are small, and the corollary squeezes a quotient of $G$ into one. That's enough to rule out non-Abelian groups of some orders entirely.
Suppose $n = pq$, where $p < q$ are prime and $p \nmid q - 1$. Show that there are no non-Abelian groups of order $n$.
Proof. Suppose $G$ is a non-Abelian group of order $pq$.
Since $|N_G(H)/C_G(H)| = |G/H| = p$, and $|\operatorname{Aut}(H)| = |\operatorname{Aut}(\mathbb{Z}/q\mathbb{Z})| = |(\mathbb{Z}/q\mathbb{Z})^\times| = q - 1$, Lagrange says we must have $p \mid q - 1$. $\Rightarrow\!\Leftarrow$ $\blacksquare$
Example 7 leaned on two theorems from earlier pages. Cauchy's theorem, from Finite Abelian Groups, produced the subgroup $H$ of order $q$. The smallest prime index theorem, from Group Actions, made $H$ normal for free.
Combine Example 7 with the classification of finite Abelian groups and you get a clean statement.
Show that every group of order $15$ is cyclic.
$15 = 3 \cdot 5$ with $3 < 5$ prime, and $3 \nmid 5 - 1 = 4$. By Example 7, every group $G$ of order $15$ is Abelian. By Cauchy's theorem, $G$ has elements $x$ of order $3$ and $y$ of order $5$. They commute, and $\gcd(3, 5) = 1$, so $|xy| = 15$. Therefore $G = \langle xy \rangle \cong \mathbb{Z}/15\mathbb{Z}$.
The same argument works for any $pq$ with $p \nmid q - 1$: $15$, $33$, $35$, $51$, and so on are all "cyclic only" orders. What about orders like $6 = 2 \cdot 3$ and $21 = 3 \cdot 7$, where $p \mid q - 1$? There, $S_3$ shows a non-Abelian group can exist.
When $p \mid q - 1$, $\operatorname{Aut}(\mathbb{Z}/q\mathbb{Z}) \cong C_{q-1}$ has a subgroup of order $p$, and that's exactly the room a non-Abelian group needs. On Semidirect Products we'll use a homomorphism $\mathbb{Z}/p\mathbb{Z} \rightarrow \operatorname{Aut}(\mathbb{Z}/q\mathbb{Z})$ to build one, and settle this case.
The first two are warm-ups with the formula $\operatorname{Inn}(G) \cong G/Z(G)$. The next two show how $\operatorname{Aut}(G)$ and $\operatorname{Inn}(G)$ fit together.
Find $\operatorname{Inn}(S_3)$ and $\operatorname{Inn}(Q_8)$.
$S_3$: no non-identity element commutes with everything. Each transposition fails to commute with the other two transpositions (e.g. $(1\ 2)(1\ 3) = (1\ 3\ 2) \neq (1\ 2\ 3) = (1\ 3)(1\ 2)$), and each $3$-cycle fails to commute with every transposition (e.g. $(1\ 2\ 3)(1\ 2) = (1\ 3) \neq (2\ 3) = (1\ 2)(1\ 2\ 3)$). So $Z(S_3) = \{e\}$, and $\operatorname{Inn}(S_3) \cong S_3/\{e\} \cong S_3$.
$Q_8$: $Z(Q_8) = \{\pm 1\}$, and $Q_8/\{\pm 1\} \cong V_4$ (Example 7 on Quotient Groups). So $\operatorname{Inn}(Q_8) \cong V_4$.
Notice $\operatorname{Inn}(D_8) \cong \operatorname{Inn}(Q_8) \cong V_4$, even though $D_8 \not\cong Q_8$. Inner automorphism groups forget a lot.
Show that $\operatorname{Inn}(G) \unlhd \operatorname{Aut}(G)$. Hint: for $\sigma \in \operatorname{Aut}(G)$, compute $\sigma \circ \tau_g \circ \sigma^{-1}$.
Proof. $\operatorname{Inn}(G)$ is the image of the homomorphism $\varphi : G \rightarrow \operatorname{Aut}(G)$, so it's a subgroup. Let $\sigma \in \operatorname{Aut}(G)$ and $g \in G$. $\forall x \in G$, $$(\sigma \circ \tau_g \circ \sigma^{-1})(x) = \sigma\big(g\,\sigma^{-1}(x)\,g^{-1}\big) = \sigma(g)\,\sigma(\sigma^{-1}(x))\,\sigma(g)^{-1} = \sigma(g)\,x\,\sigma(g)^{-1} = \tau_{\sigma(g)}(x),$$ using that $\sigma$ is a homomorphism (so $\sigma(g^{-1}) = \sigma(g)^{-1}$). So $\sigma \tau_g \sigma^{-1} = \tau_{\sigma(g)} \in \operatorname{Inn}(G)$, and $\operatorname{Inn}(G) \unlhd \operatorname{Aut}(G)$. $\blacksquare$
Show that if $\operatorname{Aut}(G)$ is cyclic, then $G$ is Abelian.
Proof. $\operatorname{Inn}(G) \le \operatorname{Aut}(G)$, and every subgroup of a cyclic group is cyclic, so $\operatorname{Inn}(G)$ is cyclic. Since $G/Z(G) \cong \operatorname{Inn}(G)$, $G/Z(G)$ is cyclic, so $G$ is Abelian (the exercise on Quotient Groups). $\blacksquare$
The last one is Example 7 in action.
Show that every group of order $33$ is cyclic, and that every group of order $35$ is cyclic. Why doesn't the same argument work for order $21$?
$33 = 3 \cdot 11$ and $3 \nmid 10$; $35 = 5 \cdot 7$ and $5 \nmid 6$. By Example 7, any group $G$ of either order is Abelian. By Cauchy's theorem it has elements $x$ of order $p$ and $y$ of order $q$, which commute and have coprime orders, so $|xy| = pq$ and $G = \langle xy \rangle$ is cyclic.
For $21 = 3 \cdot 7$, $3 \mid 6 = 7 - 1$, so the last step of Example 7 gives no contradiction: a subgroup of order $3$ inside $\operatorname{Aut}(\mathbb{Z}/7\mathbb{Z}) \cong C_6$ is perfectly possible. (And there really is a non-Abelian group of order $21$.)
We found $\operatorname{Inn}(D_8) \cong V_4$, which has order $4$. The full automorphism group is twice as big, and it's a group you've met before.
Let $D_8 = \langle r, s \mid r^4 = s^2 = 1, rs = sr^{-1} \rangle$. Prove that $\operatorname{Aut}(D_8) \cong D_8$.
Hint: an automorphism is determined by where it sends $r$ and $s$. Count the possibilities, then find an automorphism of order $4$ and one of order $2$.
Proof. At most $8$. Let $\sigma \in \operatorname{Aut}(D_8)$. Since $D_8 = \langle r, s \rangle$, $\sigma$ is determined by $\sigma(r)$ and $\sigma(s)$ (Property 6a on Homomorphisms). $\sigma(r)$ has order $4$, so $\sigma(r) \in \{r, r^3\}$. $\sigma(s)$ has order $2$, and $\sigma(r), \sigma(s)$ must generate $D_8$; since $\langle r, r^2 \rangle = \langle r \rangle \neq D_8$, $\sigma(s)$ is a reflection: $\sigma(s) \in \{s, sr, sr^2, sr^3\}$. That's at most $2 \cdot 4 = 8$ automorphisms.
Exactly $8$. Fix $a \in \{1, 3\}$ and $i \in \{0, 1, 2, 3\}$, and send $r \mapsto r^a$, $s \mapsto sr^i$. These satisfy the relations of the presentation: $(r^a)^4 = 1$; $(sr^i)^2 = sr^i s r^i = s s r^{-i} r^i = 1$; and $r^a (sr^i) = s r^{-a} r^i = (sr^i) r^{-a} = (sr^i)(r^a)^{-1}$. By Property 6b on Homomorphisms, this extends to a homomorphism $\sigma_{a,i} : D_8 \rightarrow D_8$. Its image contains $r^a$ (so all of $\langle r \rangle$) and the reflection $sr^i$, so it's all of $D_8$. A surjection from a finite set to itself is a bijection, so $\sigma_{a,i} \in \operatorname{Aut}(D_8)$, and $|\operatorname{Aut}(D_8)| = 8$.
Which group of order $8$. Let $\alpha = \sigma_{1,1}$ ($r \mapsto r$, $s \mapsto sr$) and $\beta = \sigma_{3,0}$ ($r \mapsto r^{-1}$, $s \mapsto s$). Then $\alpha^k(s) = sr^k$, so $\alpha$ has order $4$, and $\beta$ has order $2$. Also $\beta \notin \langle \alpha \rangle$, since every power of $\alpha$ fixes $r$. Check $\beta \alpha \beta$ on the generators: $$r \overset{\beta}{\mapsto} r^{-1} \overset{\alpha}{\mapsto} r^{-1} \overset{\beta}{\mapsto} r, \qquad s \overset{\beta}{\mapsto} s \overset{\alpha}{\mapsto} sr \overset{\beta}{\mapsto} sr^{-1}.$$ That's $\alpha^{-1}$ ($r \mapsto r$, $s \mapsto sr^{-1}$), so $\beta\alpha = \alpha^{-1}\beta$. So $\langle \alpha \rangle$ has index $2$, $\operatorname{Aut}(D_8) = \langle \alpha, \beta \rangle$, and $\alpha, \beta$ satisfy $\alpha^4 = \beta^2 = 1$, $\alpha\beta = \beta\alpha^{-1}$, the presentation of $D_8$. By Property 6b, $r \mapsto \alpha$, $s \mapsto \beta$ gives a homomorphism $D_8 \rightarrow \operatorname{Aut}(D_8)$; it's onto, and both groups have order $8$, so $\operatorname{Aut}(D_8) \cong D_8$. $\blacksquare$
Inside it, $\operatorname{Inn}(D_8) = \{\sigma_{1,0}, \sigma_{1,2}, \sigma_{3,0}, \sigma_{3,2}\}$ is a normal $V_4$: $\tau_r$ fixes $r$ and sends $s \mapsto rsr^{-1} = sr^{-2} = sr^2$, and $\tau_s$ sends $r \mapsto srs^{-1} = r^{-1}$ and fixes $s$. The automorphism $\alpha$ is not inner: it's an outer automorphism, swapping the two kinds of reflection (through vertices and through edges).