Q:

What does the (*) in (int (*)[30]) mean?

What does (int (*)[30]) mean in C? For instance, in:
int (*b)[30] = (int (*) [30]) malloc(30 * sizeof(int [20]));

A:

It means, roughly, "is a pointer".
int (*b)[30]

This means "b is a pointer to an array of 30 integers".
(int (*) [30])

This means "cast to a pointer to an array of 30 integers".

A:

int (*b)[30] = (int (*) [30]) malloc(30 * sizeof(int [20]));

Breaking it down:
      b        -- b
    (*b)       -- is a pointer
    (*b)[30]   -- to a 30-element array
int (*b)[30]   -- of int.

In both declarations and expressions, postfix operators like [] have higher precedence than unary operators like *, so T *a[] is interpreted as T *(a[]); IOW, a is an array of pointer to T.  To designate a as a pointer to an array, we have to force the grouping T (*a)[].  
Simlilarly, the cast expression (int (*) [30]) means "treat the pointer value returned by malloc as a pointer to a 30-element array of int".  Note that, technically speaking, the cast expression is superfluous and should be removed.  
The malloc call itself seems very wrong.  You're allocating 30 instances of a 20-element array of int, but assigning the result to a pointer to a 30-element array of int; that's going to cause problems.  Assuming you're trying to allocate a N x 30 matrix of int, the following would be safer:
int (*b)[30] = malloc(N * sizeof *b); 

The type of the expression *b is int [30], so sizeof *b is the same as sizeof (int [30]).   

