Q:

Determinant associated with a group action

Let $G$ be a finite group and $S$ be a finite set, with $G$ acting on $S$. I consider indeterminates $x_g$ indexed by $g\in G$ and form the matrix of the group action $A\in M_{S\times S}$. Its entries are indexed by pairs $(s,t)$ of elements of $S$. By definition, $a_{st}$ is the sum of the $x_g$'s over the group elements such that $s^g=t$.

What are the irreducible factors of $\det A$ ?

Without loss of generality, we may assume that the action is transitive, because otherwise $A$ is block diagonal. Just sort out the elements of $S$ by orbits. We may also assume that the action is faithful.
This determinant has an obvious factor $\ell:=\sum_{g\in G}x_g$, because $\ell$ is the sum of the entries of every row (or of every column). What does it mean about the action that the quotient $(\det A)/\ell$ be irreducible ?
Notice that if $S=G$ and $G$ acts by multipication, this determinant is that considered by Dedekind and Frobenius, which led the latter to the theory of representations.

A:

Should it not still be the case that this determinant is the product of the irreducible factors
of the group determinant associated to the irreducible constituents of the given permutation
representation, each factor occurring with the multiplicity with which the associated irreducible
character occurs in the given permutation character. Furthermore, the irreducibility of
(det A)/$\ell$ would seem to force the given permutation action to be doubly transitive.
Isn't your matrix $A$ just $\sum_{g \in G} x_{g}(g\sigma)$, where $ g \sigma$ is the permutation matrix by which $g$ acts. Therefore, if we choose a (wlog) unitary matrix $M$ such that 
$M^{-1}(G\sigma)M$ breaks into irreducible (as complex representations) blocks, this gives at least
one factor of ${\rm det} A$  (as polynomial in the $x_{g}$'s) for each irreducible
constituent (counting multiplicities) of the given representation. So this seems to show
that the irreducibility of ${\rm det} A/\ell$ forces double transitivity on the permutation action.
The harder part, I think, is to prove that the polynomials obtained by the determinant of the
irreducible constituents remains irreducible as a polynomial in the $x_{g}$'s, but I think
that Frobenius proved that when working with the full group determinant
(by which I mean ${\rm det}(\sum_{g \in G}x_{g}(g \rho))$, where $\rho$ is the regular 
representation of $G$). 

A:

I wish to elaborate a bit on Geoffrey Robinson's answer:
Given a representation $D\colon G\to M_{n\times n}(F)$ which is similar to a representation in upper block form
$$ D \sim \begin{pmatrix} 
             D_1 & * \\
                 & D_2
          \end{pmatrix},
$$
then the determinant factors into
$$ \Theta_D(x):= \det\left( \sum_{g} x_g D(g) \right) = \det\left( \sum_g x_g D_1(g) \right) 
                               \det\left( \sum_g x_g D_2(g) \right) .$$
Thus $\Theta_D$ factors accordingly to the composition factors of $D$, even if the characteristic of $F$ divides $|G|$.  
If $D$ is absolutely irreducible, then $\Theta_D$ is irreducible (see here), and inequivalent representations yield different irreducible polynomials. In particular, your $\det(A)/l$ is irreducible over $\mathbb{C}$ iff the action is doubly transitive.  
The coefficients of $\Theta_D(x)$ as polynomial in the $x_g$'s generate the same field as the values of the character of $D$, and of course $\Theta_D(x)$ remains irreducible over that smaller field (for $D$ absolutely irreducible). If $D$ is irreducible over $F$, but not absolutely irreducible, then $D \cong m(D_1 \oplus \dotsb \oplus D_r)$, where the $D_i$ are algebraically conjugate over $F$, and $\Theta_{D_1}(x)\cdot \dotsm \cdot \Theta_{D_r}(x) $ is irreducible over $F$. Thus in your situation, $\det(A)/l$ is irreducible as a polynomial over $\mathbb{Q}$ iff the permutation module 
$$ \mathbb{Q}S \cong \mathbb{1} \oplus( \text{rationally irreducible module}). $$
This is a slight weakening of double transitivity. There is a paper about this situation:

Dixon, John D., Permutation representations and rational irreducibility, Bull. Austral. Math. Soc. 71 (2005), p. 493-503 (link).  

There are only few examples where this occurs and the action is not doubly transitive, but there is, to the best of my knowledge, no classification.

