From the above example, we can see that the following challenges in proposing a new solution.
Firstly, it is essential to learn richer semantics of columns based on natural language domain.
To this end, we require a more powerful approach to learn the column representation so as to capture richer information instead of relying on simple methods like the average over bag of word embeddings utilized in previous studies [1]}, [2]} or even the similarity of the word embedding distributions [3]}.
Secondly, we argue that it is crucial to utilize the contextual information within a table to learn the representation of each column, which is ignored by previous studies. Even proposals for capturing relationship semantics do not use contextual information to learn column representations [4]}.
Hence, it is difficult for them to identify the false alignment of columns between Table A and Table C in the above example.
Finally, due to the large volume of data lake tables, it is also a great challenge to develop a scalable and memory-efficient solution.
