In AC framework, since we need to work with the \(Q\) -function and the policy, which are \(|\mathcal {S}||\mathcal {A}|\)  dimensional objects, the algorithm becomes intractable when the size of the state-action space is large [1]}. To overcome this difficulty, in this work we consider using linear function approximation for both the policy and the \(Q\) -function. Specifically, let \(\lbrace \phi _i\rbrace _{1\le i\le d}\)  be a set of basis functions, where \(\phi _i\in \mathbb {R}^{|\mathcal {S}||\mathcal {A}|}\)  for all \(i\) . Without loss of generality, we assume that \(\phi _i\) , \(1\le i\le d\) , are linearly independent and are normalized so that \(\Vert \phi (s,a)\Vert _1\le 1\)  for all \((s,a)\) , where \(\phi (s,a)=[\phi _1(s,a),\cdots ,\phi _d(s,a)]\)  is the feature associated with state-action pair \((s,a)\) . Let \(\Phi =[\phi _1,\cdots ,\phi _d]\)  be the feature matrix. We parameterize the policy and the \(Q\) -function using compatible function approximation [2]}. In particular, we use softmax parametrization for the policy, i.e., \(\pi _\theta (a|s)=\frac{\exp (\phi (s,a)^\top \theta )}{\sum _{a^{\prime }\in \mathcal {A}}\exp (\phi (s,a^{\prime })^\top \theta )}\)  for all \((s,a)\) , where \(\theta \in \mathbb {R}^{d}\)  is the parameter. As for the \(Q\) -function, we approximate it from the linear sub-space given by \(\mathcal {Q}=\lbrace Q_w=\Phi w\mid w\in \mathbb {R}^d\rbrace \) , where \(w\in \mathbb {R}^{d}\)  is the corresponding parameter. By doing this, we now only need to work with \(d\) -dimensional objects (i.e., \(w\)  for the \(Q\) -function and \(\theta \)  for the policy), where \(d\)  is usually chosen to be much smaller than \(|\mathcal {S}||\mathcal {A}|\) .
