“be closely integrated with the statistical description of the dataset”.  VizAI automatically generates a number of alternative visualizations for the given dataset, evaluates them on how well they (statistically) represent the underlying data and then selects the best performing one. The evaluation of the visualization itself is done using a deep convolutional neural network, based on an ensemble of standard ConvNets, that extracts features from the visualization which can be used to predict the aggregate statistics of the data. For example, in Figure REF  the underlying data was plotted using scatter, line and density plots (we used Chart Studio by Plotly for plotting these, although our work can be easily used with any other plotting software). Of these, VizAI ranked the density plot as the best and this was further confirmed by our crowdsourcing evaluation (details are in Section ).
Note that our focus in this paper is on generating visualizations of numerical datasets rather than datasets with varied data types, such as textual, categorical, temporal, etc., across data fields. Often these data types are transformed as part of data preprocessing into an appropriate numerical representation before their visualization [1]}, [2]}. VizAI can be adapted to incorporate these preprocessing steps as well. With numerical data, commonly used visualizations are line, scatter, area, bar and pie charts, and more sophisticated settings require density plots.
Apart from developing a visualization recommendation framework, our work also goes towards developing a systematic performance evaluation measure of various visualizations. In fact, it is an important topic among visualization researchers [3]}. We believe that our approach of extracting statistical features of the underlying data mimics the human perception of these visualizations, and offers a framework that can be extended to take into account aspects like choice of colorschemes for the plots and so on.
Contributions
In summary, we make the following key contributions in this paper:
We present a novel framework for automatically evaluating the “goodness” of a visualization in terms of the accuracy of extracting various statistics of the underlying data, just from the visualization alone. Our approach does not rely on axis or key labels in the plot, or any other numerical/textual content that may be embedded in the plot.

Based on the above, we present VizAI, an intelligent visualization recommendation system based on generative-discriminative framework. It uses a deep convolutional neural network for generating various statistical features from the plot, and a discriminator that selects the plot with the lowest error in predicting aggregate statistics of the underlying data.

We conduct a detailed evaluation, using real-world (from Plot.ly) visualizations along with a visualization learning and benchmarking repository, VizNet [4]} and the results show that VizAI offers high quality recommendations nearly as good as human evaluators.
Organization
The rest of the paper is organized as follows: in Section , we discuss work related to the ideas pursued in this paper. These range from studies on human perception of visualizations, machine perception of visualizations, and various automated visualization recommendation systems. Next, in Section , we present our key hypothesis regarding how to evaluate the quality of visualizations, and formulate our problem statement. Various aspects of VizAI system, including the statistical features and the CNN for extracting these features from visualizations, are described in detail in Section . It is followed by the details of our experimental framework in Section , and the results over real-world datasets in Section . We finally conclude and outline future directions of research in Section .

Related Work
Our work focuses on utilizing deep convolutional neural networks trained to emulate human perception of visualizations for automatic selection of best visualization. In this section we review prior work in each of these areas.
Human Perception of Visualizations
Early works experimentally demonstrated that aggregate statistics of data, means and correlations respectively could be perceived with high accuracy and compared across classes by humans from data visualizations [5]}, [6]}.
Subsequent analysis of chart perception through extensive human studies found that both individual features as well as aggregate of data could be inferred from good visualizations  [7]}. They varied the number of columns and the entropy of data, showing that for a small number of columns humans can perceive differences and even individual values of aggregates like averages with low error rates.
Using a large-scale crowdsourced study, [8]} demonstrated that human performance (accuracy as well as time taken) in tasks like finding aggregates (sum for example) of data as well as extremums is highly affected by the type of visualization (scatter, bar, table etc).
Our work builds on these insights to model the perception of aggregates and extremums to help in identifying the most appropriate visualization form for the given data.

CNNs for Data Visualization Understanding
Deep CNNs are extremely popular for various learning tasks over images [9]}, [10]}. Considering data visualizations as images, they have been applied for tasks such as visual QA, focus tracking etc. For example, [11]} use a combination of CNN and OCR output for question answering on visualizations. As a solution to the problem of automatically learning visual importance i.e., the places in an infographic a user is likely to focus on, [12]} used a fully convolutional network for creating importance maps.
Recently [13]} did an evaluation of CNNs as applied to visualizations. They tried to regress elementary quantitative features like position, length and area from the images of visualizations presented to the network. They suggest that though CNNs can regress quantities for well constrained problems, they would require extensive training to generalize across tasks.

Visualization recommendation
There have been many recent efforts towards automating the visualization recommendation process for a given dataset. We briefly survey four of the most recent works that have shown significant promise in this direction.
Data2Vis  [1]}
tackles the visualization generation problem as a language translation problem where data specifications are mapped to visualization specifications. It uses an LSTM based neural translation model to learn a mapping between JSON encoded data and a Vega-lite visualization specification. The model is trained using \(4,300\)  Vega-lite examples. Their model was in an elementary way able to learn the vocabulary and syntax for a valid visualization specification, appropriate transformations (count, bins, mean) and how to use common data selection patterns that occur within data visualizations.
VizML [15]}
formulates visualization recommendation as making design choices. It learns these choices from a large corpus of one million dataset-visualization pairs collected from Plot.ly public studio of visualizations. They use 841 hand-coded features to train their recommendation networks to make encoding-level and visualization-level design choices. VizML verified that certain steps in a visualization generation pipeline can be learned well using neural networks.
DeepEye  [2]}
combines rule based visualization generation with models trained to 1) classify a visualization `good' or `bad' using a decision tree classifier and 2) rank lists of visualizations using a ranking neural network. The DeepEye corpus consists visualizations drawn from 42 public datasets. 100 students annotated these visualizations as good/bad. These annotations, combined with 14 features for each column pair and rule based heuristics are used to train their models.
Draco [17]}
tackles the visualization recommendation in a more formal framework. It uses Answer Set Programming (ASP), a contraint logic programming to specify and generate visualizations with soft weights for constraints learnt from training data.
Problem Formulation
The visualization of a dataset is a collection of visual elements used to communicate the data in a distortion-free manner. While there are many aspects of visually communicating the contents of a dataset, including the choice of colors, domain-specific imagery such as maps, overlays, etc., our work is based on a fairly simple and fundamental premise that one should be able to derive the core statistical features of the underlying dataset accurately from the visualization [18]}, [19]}. Since finding the best visualization is a normative question, we make the following assumptions and later validate them through user studies:

Humans prefer visualizations from which they can perceive features about the underlying data accurately.

The modern neural network based models, trained to predict data features from plots, naturally emulate human perception to some extent. That is, if the model makes larger errors in estimating statistical features from one visualization over from another visualization of the same data, we expect that humans when asked to perceive the same statistics end up making corresponding scale of errorsIt should be noted that we do not expect the same exact amount of statistical errors between human and our model, but rather their relative scale..

With these assumptions our problem of finding the best visualization is reduced to the task of designing and training a model which satisfies the above criteria and finding a set of features.

VizAI Framework
<FIGURE>VizAI consists of the following three phases:

True Statistics Computation and Candidate Generation: We compute a set of statistics from the underlying data tables and use them as features that need to be estimated by our model from the given visualizations. We keep these statistics to be easily computable and, more importantly, easily interpretable by our users. Next, we programmatically generate a number of visualizations of each data table using a visualization generator such as Plotly, gnuplot, Vegalite etc.

Statistical-Feature Regression: This is the core model building stage where visualizations are input as images to a supervised Convolution Neural Network (CNN) regression model that extracts (regresses) the statistical features from these images, and trains to minimize the loss between the true statistics and the extracted statistical features. Note that during training, we optimize the statistics regression for each visualization type separately. Thus, adding another visualization type has no impact on the models built on earlier visualization types.

Visualization Selection: Finally, we compute the loss between the true statistics computed in the first stage and the extracted statistics by the model trained the second stage. The visualization that is considered to be performing best in terms of this loss is recommended for the given data table.

The overall model architecture is depicted in Figure REF . In the rest of the section, we discuss each of these parts of VizAI in detail.
True Statistics Computation and Candidate Visualization Generation
<TABLE>We tried several approaches of selecting a set of statistics as features to extract from data tables. We initially experimented with table embeddings[20]} and other neural feature extraction approaches but their major disadvantage was that these features are not understood or interpretable by end users. Since understanding the various aspects of why a particular visualization is good is relevant for data analysts, for us the interpretability is an essential requirement[21]}. Thus finally we settled on a set of simple and commonly used features that most people working with data have some understanding of. These are listed in REF . Note that these are consistent with the recommendation of focusing on extremes, medians and relationships [19]}. Note that these core statistics can be further enriched as required by specific application settings.
Although the data tables we currently experiment with consist of only two columns to be visualized, in a general setting we will have to deal with tables with variable number columns.
If only individual column-level statistics are used as features, then each data table can result in different feature dimensionality. To overcome this, we introduce the use of 2-stage feature aggregation. First, column wise statistics (mentioned in the first column of REF ) are calculated for each column. Then a data-table level aggregation is carried out using the metrics mentioned in the second column of REF . Thus we use two levels of features, e.g., `max of mean' is the maximum of the mean values of each column. Along with this, we also use Pearson's correlation coefficient, \(r\) , between any two columns X and Y of our table, written as:
\(r =\frac{ \sum _{i=1}^{n}(x_i-\bar{x})(y_i-\bar{y}) }{\sqrt{\sum _{i=1}^{n}(x_i-\bar{x})^2}\sqrt{\sum _{i=1}^{n}(y_i-\bar{y})^2}},\) 
where \(x_i\)  and \(y_i\)  are the \(i^{th}\)  row for each column and \(\bar{x}\)  and \(\bar{y}\)  represents the mean of each column. The generated correlation coefficient in unique for each table as we have only 2 columns in current setup of tables in the dataset. This can be easily modified to performing a table level aggregation of the correlation coefficient to accommodate tables with variable number of columns. As a result, in the current setup each data-table is represented using the same 26 statistical features which are used by the next stage of model building.
Visualization Candidate Generation. Next, we generate a number of different visualizations for each data table and feed them to a convolutional neural network (CNN) layer coupled with a regressor that extracts statistics from each of these visualizations. As we already mentioned we make use of Plot.ly Chart Studio https://plotly.com/chart-studio/ for generating visualizations of the table to keep our visualizations consistent with the publicly available datasets from VizNet repository (details in Section ). These visualizations are input to the CNN + regressor model during training, and also as candidate visualizations to select from during the test/inference.

Statistical-Feature Regression
<FIGURE>The visualizations are given as image input to the CNN model that generates their embeddings. We train a multivariate regressor to extract various statistical features of the underlying data.
We experimented with multiple state of the art CNN's in our experiments such as AlexNet [23]}, VGG19 [24]} along with ResNet18 [10]} and ResNet50 [10]}. As the input images as well as the output labels are quite different from the standard datasets that were used to train these CNNs, simply training the regressor yielded quite poor results. Instead, we simply fine-tune the final classifier layers of CNNs and train the regression layer for our training dataset.
Somewhat surprisingly we found that larger models (in terms of number of parameters) do not correspond to improved performance in our setting. In Figure REF , we illustrate the performance of different CNNs in terms of their error in extracting various statistics for the dataset we collected from Plotly (details in Section ). It is immediately clear that larger models such as ResNet18 and ResNet50 give huge losses on the test dataset for many statistical features. It may perhaps be due to the issue of over-fitting because of the number of free parameters. On the other hand, AlexNet and VGG19 show consistently better performance across all the statistics we focus on. In our experiments we settled on using AlexNet and VGG19 which were our top-2 best performing CNNs. Additionally, we also experimented with an AdaBoost ensemble of these two models, which we represent as (AlexNet+VGG19).
The output embedding from the convolutional layers passes through a fully connected regression layer trained to output the values of statistical features for the given visualization. For the regressor, we used weighted smooth L1 loss between the predicted vector and the features extracted from the data table. The model is trained in a supervised fashion (using the true statistics computed from the previous phase) to predict the statistical features of each visualization.

Visualization Selection
At the time of evaluation, we take a data table as our input, extract it's data features for use as true values and produce different types of plot images for it. The regression model is run on each of these to obtain predicted feature values. Based on how close these values are to the actual features of the data, each plot is assigned a score. The best plots is chosen by our model using this score. This not only helps in selecting the best plot for some given data, but also helps in realizing how good a visualization is by analyzing the score. In our experiments, we consider two such types of scoring functions.

Normalised L1 Loss - For our setup, we define the well known normalised L1 loss as
\(l^{L1}(\hat{y}, y) = \sum _{i=1}^{26}\frac{|\hat{y_i} - y_i|}{\overline{t_i}}\) 
where \(\hat{y}\)  is the features predicted by our CNN Encoder, \(y\)  is the true value of all the feature and \(\overline{t_i}\)  is the average value of the \(i^{th}\)  feature from the training dataset.

Top-K Closest Loss -
We define the Top-K Closest loss as the cumulative L1 loss for only the Top-K Statistical features for each plot type prediction closest to true values.
\(l^{topK}(\hat{y}, y) = \min {\sum _{i\in tup}(\frac{|\hat{y_i} - y_i|}{\overline{t_i}})} \quad \forall {tup \in \lbrace 1,2...,26\rbrace ^k}\) 
Evidently, this will be nothing but the sum of normalised L1 Loss for the k features whose prediction is closest to true feature values relative to the true value.

We will use Normalised L1 Loss, Top-5 and Top-10 Closest Loss in our experiments.

Advantages of VizAI
Based on the literature survey done in section REF , we identify two major limitations of the existing recommendation systems and discuss how VizAI can solve these limitations :

Rule-based recommendation systems like  [27]}, [2]} have obvious limitations when it comes to adapting to new data tables and new types of visualizations. Also machine learning based models such as  [1]}, [15]} require complete re-training of the complete dataset in such a setup. VizAI is highly flexible and only requires additional training on new data to adapt to the new setup.

None of these works provide quantification on how well the recommended plots are able to depict the data they were created on. Based on our previous studies, we know the importance of such a quantification, especially for visualisations of data in the numeric domain. Overcoming this limitation, VizAI tries to generate a score for each visualization type based on it's ability to depict data statistics easily.


Datasets
As we mentioned earlier, finding the best visualization for a data table is a normative question, user studies are necessary to evaluate the overall performance of recommendations. For this purpose, we conduct experiments using publicly available Plotly data-visualization pairs and a subset from VizNet collection to carefully examine the performance of various choices of statistical-feature regression models, and the loss models used in the visualization selection. We generate plots using the Chart Studio Python API for all of these datasets to maintain uniformity.
Plotly Dataset
Plot.ly [31]} is an online service used by a large number of people to generate various types of plots of data. We wrote our own crawler which extracted data-visualization pair from the Plot.ly raw feed. We extracted data-visualization pairs of the following plot types: (i) scatter, (ii) line, (iii) density. We filtered out data tables with categorical and text columns. For line and scatter plots with more than one dataseries in the same plot, we separated up to 5 of them and treated them as different data-visualization pairs.
After the preprocessing, the dataset had a total of 24547 data-visualization pairs, the distribution of the data is shown in Table REF . We used min-max normalization within a column to keep the range of x and y in [0,1]. Each visualization type was split in 80:10:10 ratio into training, validation and test respectively.
As we described in Section , our statistics-extraction models based on VGG19, AlexNet, and their ensemble were trained, validated and tested on this dataset. Figure REF  shows the statistical feature extraction loss we observed on Plotly test data. Note that we make use of only the data tables in this stage to compute the feature extraction loss.
<TABLE>
VizNet
VizNet [4]} is a large-scale repository of data as used in practice, compiled from the web, open data repositories, and online visualization platforms. VizNet repository contains more than 23 visualization types, and a set of unique data tables and corresponding visualization to represent the data table – thus, the visualization type used is considered the gold label for the data table.
We extract 2,042 data table and visualization pairs from the plot.ly fraction of VizNet to build our test dataset. We extract only those instances where (i) the size of data table (i.e., the number of rows) \(\ge 5\) , and, (ii) the gold labels are either of scatter or line plots as VizNet does not contain density plots as gold label. For each data table extracted, we generate 3 plots of type line, scatter and density. We conducted several experiments and crowd-sourced judgments on this data to validate the performance of our model.

Crowd Sourcing Framework
For evaluation, we employed a Turkle[33]} instance running on a university server, offering features similar to Amazon Mechanical Turk (AMT) platform. Since the evaluation required expert users who could understand various statistical properties of the data we are interested in inferring, we did not run these experiments directly on AMT. Participants in our evaluation were CS graduate students and faculty members in our university. We defined the following three tasks designed to study our hypothesis about whether statistical features of the underlying data tables are reproduced by the recommended visualizations:

Complex Statistics Inference (CSI): Given the mean and standard deviation values of X and Y columns of some data table, the users must select the plot which represents them the best. Similar to the above task, all plots given as choices in every task depict the same data whose statistics are shown to the users (but not the data table itself). The user has to choose on which plots it is easiest, doable and impossible for this task as shown in Figure REF .

Fraction Above Threshold (FT): We provide the users with a single plot per question and ask them to estimate the fraction of points in the given plot which have y (or x) value \(> 0.5\) . While Answering these questions, It might be trivial to count points for some plots and answer the question so we instructed evaluators to estimate the fraction by looking at the plot without counting the number of points one by one.

<FIGURE><FIGURE>We assign points to each plot type for each data table on the basis of answers given by users in the crowdsourcing experiment for VizNet data as shown in Table REF . For CSI task, the points are awarded based on the choice indicating the ease of statistics inference from the plot. In the FT task, where the evaluators have to estimate the fraction, the points are awarded based on the relative error made by them from the true value of the statistic – upto 20% error we award 2 points, upto 40% error we award 1 point, and 0 points are awarded for errors beyond 40%. The cumulative points are calculated for each plot type for each data table and the plot type with most number of points is chosen as the overall preferred choice of plot type.
<TABLE>
Experimental Results
Statistics Extraction Accuracy
As already mentioned, the statistics extraction model consisting of CNN followed by a fully connected regressor was trained and tested using Plotly dataset. Figure REF  presents the heatmaps of statistics extraction error with different choices of CNNs including our AlexNet+VGG19 ensemble. In Figure REF  we show the loss in the Skew column aggregation over each of the table level aggregations given in Table REF . Similarly, Figure REF  plots the loss in estimating Pearson's correlation coefficient between X and Y columns of the data table from the visualization alone. As one can observe, the larger CNN models (ResNet18 and ResNet50) perform much poorer than smaller models, and our (VGG19+AlexNet) ensemble further improves on statistics extraction accuracy.
<FIGURE><FIGURE>
VizAI Performance on VizNet
<TABLE>We now turn our attention to evaluating the performance of VizAI on real-world data from VizNet [4]} based on a crowd-sourced experiment. We selected samples from VizNet which are 2-Dimensional and have gold labels as either line or scatter plots (note that VizNet does not have density plots). We generated the same three plots for each sample for the crowd sourcing experiment. We removed the labels of the data and just treated the two columns as X and Y.
<TABLE>We asked each participants in the crowdsourcing to complete a total of 245 SI and FT (both for Y and X axis) tasks, on a randomly selected 35 tables from the test set. We aggregate points based on the scheme described earlier (Table REF ) for CSI and FT tasks. The plot type with maximum points is chosen as the preferred plot type. The resulting count of preferred plot type for each visualization type is summarized in Table REF . Note that the sum of these counts is greater than 35 due to ties in the points between two or more plot types. It is interesting to observe that although VizNet did not have density plots, there were a significant instances where crowd evaluators shown preference to density plots over line chart. In our subsequent evaluation, we used our crowd sourced plot types as gold labels.
We experiment using all combinations of statistics extraction models (AlexNet, VGG19 or (VGG19+AlexNet) ensemble) and the visualization selection (L1-loss, Top-5 Closest and Top-10 Closest loss) models. The results are summarized in Table REF . From these results, we can evidently see that the use of the (VGG19+AlexNet) ensemble, followed by the use of Top-5 Closest loss in the visualization selection performs the best in terms of accuracy and F1-scores of individual plot types as well as aggregated weighted F1-score using the counts from Table REF .

Issues with VizNet Gold Labels
As we observed deviations in crowdsourced gold labels we collected from the gold labels given in VizNet dataset, we felt it prudent to investigate this issue further. We first evaluate our best performing combination – (VGG19+AlexNet) with Top-5 closest loss – when using the gold labels as given in VizNet. Table REF  shows that the use of VizNet gold label results in much worse performance. However, we believe that this is indicative of the data quality issues one would expect when crawling at large (as observed recently in [35]}). We further highlight this issue with an example drawn from our crowdsourcing experiment shown in Figure REF . While VizNet gold label was the line plot, two evaluators for this task clearly preferred the density plot (and one evaluator chose the scatter plot).
<FIGURE><TABLE>
Conclusions
In this work we presented VizAI, a visualization recommendation system based upon machine prediction of statistics of the data from different plot images.
Our experiments demonstrate that our model makes intelligent visualization predictions which are highly favored by end users. The results of our crowd-sourced human study reveals interesting aspects behind human perception of visualizations.
From a practical perspective, the approach taken by VizAI can easily adapt to new visualization paradigms without significant cost compared to VizML which requires complete retraining to accommodate for the new plot type. Further, it is possible to incorporate domain-specific details like focus on the quality of certain statistics, color preferences, different visualization libraries, etc., into VizAI.
Limitations
A limitation of our work is that we make use of an algorithmic system, viz., deep convolutional neural network (refer Section ), as a proxy for human visual perception to determine the extent to which a visualization is representing various data statistics. However, it is well-known in visualization literature that human visual perception is highly context dependent (e.g., it is subject to optical illusions), and varies greatly from person to person. Naturally, CNNs are not expected to fully mimic the human perception of visualizations. We believe that further research is required, especially in complex visualizations of semantically rich data. Despite this, VizAI offers a unique and flexible approach to recommending visualizations for a dataset.

Future Work
We plan to pursue research in extending VizAI along following three directions:

Incorporation of more plot variables such as - which columns to plot, what goes on which axis and more plot types into our prediction framework. For simplicity and easy testing, we chose only to predict plot types but for usability purposes it would be desirable to have multiple decision variables inferred automatically.

Applications of our model, which predicts human perceived data features from plot images and compares them with actual data features, in different tasks like identifying misleading visualizations (for detecting fake news for example) and helping analysts to interpret plot visualizations.

More robust scale and data statistic invariance in our feature prediction model. Right now we use a simple normalization procedure to deal with different scales in our input data features. An approach based on learning scale invariant models should improve performance and make the model more adaptable.

