Q:

Trying to define a matrix by a vector

I am trying to define a matrix with vectors with the following syntax:
typedef int vect[dim];
typedef int vect mat[dim];

In the end, I want to have a matrix using two vectors, although I get the following errors:

variably modified 'vect' at file scope typedef int vect[dim];
expected '=', ',', ';', 'asm' or '__attribute__' before 'mat' typedef
  int vect mat[dim];

A:

This typedef definition
typedef int vect mat[dim];

is invalid because the type specifier intoccurs two times due to this typedef
typedef int vect[dim];

You should write
typedef vect mat[dim];

Secondly (the C Standard, 6.7.8 Type deﬁnitions)

2 If a typedef name speciﬁes a variably modiﬁed type then it shall
  have block scope.

However it seems you defined a variably modified type in the typedef in a file scope. So the compiler should issue an error.
If you need a typedef with a variably modified type then define it in a block scope for example in the beginning of a function where it is need to be used.
Here is a demonstrative program.
#include <stdio.h>

void f( size_t dim )
{
    for ( ; dim != 0; --dim )
    {
        typedef int vect[dim];
        typedef vect mat[dim];

        mat m;

        printf( "sizeof( m ) = %zu\n", sizeof( m ) );
    }
}

int main(void) 
{
    f( 5 );

    return 0;
}

Its output is
sizeof( m ) = 100
sizeof( m ) = 64
sizeof( m ) = 36
sizeof( m ) = 16
sizeof( m ) = 4

Or another example.
#include <stdio.h>

void fill( size_t dim, int m[][dim] )
{
    for ( size_t i = 0; i < dim; i++ )
    {
        for ( size_t j = 0; j < dim; j++ )
        {
            m[i][j] = i * dim + j;
        }
    }
}

void output( size_t dim, int m[][dim] )
{
    for ( size_t i = 0; i < dim; i++ )
    {
        for ( size_t j = 0; j < dim; j++ )
        {
            printf( "%2d ", m[i][j] );
        }
        putchar( '\n' );
    }
}

int main(void) 
{
    printf( "Enter the dimension of a square matrix: " );
    size_t dim;

    scanf( "%zu", &dim );

    typedef int vect[dim];
    typedef vect mat[dim];

    mat m;

    fill( dim, m );
    output( dim, m );

    return 0;
}

The program output might look like
Enter the dimension of a square matrix: 4
 0  1  2  3 
 4  5  6  7 
 8  9 10 11 
12 13 14 15 

