Publications
2024
- NAACL 2024FindingsSUQL: Conversational Search over Structured and Unstructured Data with Large Language ModelsShicheng Liu, Jialiang Xu, Wesley Tjangnaka, Sina Semnani, Chen Jie Yu, and 1 more author2024
Many knowledge sources consist of both structured information such as relational databases as well as unstructured free text. Building a conversational interface to such data sources is challenging. This paper introduces SUQL, Structured and Unstructured Query Language, the first formal executable representation that naturally covers compositions of structured and unstructured data queries. Specifically, it augments SQL with several free-text primitives to form a precise, succinct, and expressive representation. This paper also presents a conversational search agent based on large language models, including a few-shot contextual semantic parser for SUQL. To validate our approach, we introduce a dataset consisting of crowdsourced questions and conversations about real restaurants. Over 51% of the questions in the dataset require both structured and unstructured data, suggesting that it is a common phenomenon. We show that our few-shot conversational agent based on SUQL finds an entity satisfying all user requirements 89.3% of the time, compared to just 65.0% for a strong and commonly used baseline.
@misc{liu2024suql, title = {{SUQL}: Conversational Search over Structured and Unstructured Data with Large Language Models}, author = {Liu, Shicheng and Xu, Jialiang and Tjangnaka, Wesley and Semnani, Sina and Yu, Chen Jie and Lam, Monica}, year = {2024}, url = {https://openreview.net/forum?id=DoSQeeVlUO}, acc_type = {Findings} }
2023
- LM-Switch: Lightweight Language Model Conditioning in Word Embedding SpaceChi Han, Jialiang Xu, Manling Li, Yi Fung, Chenkai Sun, and 3 more authors2023
In recent years, large language models (LMs) have achieved remarkable progress across various natural language processing tasks. As pre-training and fine-tuning are costly and might negatively impact model performance, it is desired to efficiently adapt an existing model to different conditions such as styles, sentiments or narratives, when facing different audiences or scenarios. However, efficient adaptation of a language model to diverse conditions remains an open challenge. This work is inspired by the observation that text conditions are often associated with selection of certain words in a context. Therefore we introduce LM-Switch, a theoretically grounded, lightweight and simple method for generative language model conditioning. We begin by investigating the effect of conditions in Hidden Markov Models (HMMs), and establish a theoretical connection with language model. Our finding suggests that condition shifts in HMMs are associated with linear transformations in word embeddings. LM-Switch is then designed to deploy a learnable linear factor in the word embedding space for language model conditioning. We show that LM-Switch can model diverse tasks, and achieves comparable or better performance compared with state-of-the-art baselines in LM detoxification and generation control, despite requiring no more than 1% of parameters compared with baselines and little extra time overhead compared with base LMs. It is also able to learn from as few as a few sentences or one document. Moreover, a learned LM-Switch can be transferred to other LMs of different sizes, achieving a detoxification performance similar to the best baseline. We will make our code available to the research community following publication.
@misc{han2023lmswitch, title = {LM-Switch: Lightweight Language Model Conditioning in Word Embedding Space}, author = {Han, Chi and Xu, Jialiang and Li, Manling and Fung, Yi and Sun, Chenkai and Jiang, Nan and Abdelzaher, Tarek and Ji, Heng}, year = {2023}, eprint = {2305.12798}, archiveprefix = {arXiv}, primaryclass = {cs.CL}, } - ACL 2023FindingsAnaMeta: A Table Understanding Dataset of Field Metadata Knowledge Shared by Multi-dimensional Data Analysis TasksXinyi He, Mengyu Zhou, Mingjie Zhou, Jialiang Xu, Xiao Lv, and 5 more authorsJul 2023
Tabular data analysis is performed everyday across various domains. It requires an accurate understanding of field semantics to correctly operate on table fields and find common patterns in daily analysis. In this paper, we introduce the AnaMeta dataset, a collection of 467k tables with derived supervision labels for four types of commonly used field metadata: measure/dimension dichotomy, common field roles, semantic field type, and default aggregation function. We evaluate a wide range of models for inferring metadata as the benchmark. We also propose a multi-encoder framework, called KDF, which improves the metadata understanding capability of tabular models by incorporating distribution and knowledge information. Furthermore, we propose four interfaces for incorporating field metadata into downstream analysis tasks.
- InfoPattern: Unveiling Information Propagation Patterns in Social MediaChi Han, Jialiang Xu, Manling Li, Hanning Zhang, Tarek Abdelzaher, and 1 more authorJul 2023
Social media play a significant role in shaping public opinion and influencing ideological communities through information propagation. Our demo InfoPattern centers on the interplay between language and human ideology. The demo (Code: this https URL ) is capable of: (1) red teaming to simulate adversary responses from opposite ideology communities; (2) stance detection to identify the underlying political sentiments in each message; (3) information propagation graph discovery to reveal the evolution of claims across various communities over time. (Live Demo: this https URL )
@misc{han2023infopattern, title = {InfoPattern: Unveiling Information Propagation Patterns in Social Media}, author = {Han, Chi and Xu, Jialiang and Li, Manling and Zhang, Hanning and Abdelzaher, Tarek and Ji, Heng}, year = {2023}, eprint = {2311.15642}, archiveprefix = {arXiv}, primaryclass = {cs.SI}, }
2022
- Towards Robust Numerical Question Answering: Diagnosing Numerical Capabilities of NLP SystemsJialiang Xu, Mengyu Zhou, Xinyi He, Shi Han, and Dongmei ZhangJul 2022
Numerical Question Answering is the task of answering questions that require numerical capabilities. Previous works introduce general adversarial attacks to Numerical Question Answering, while not systematically exploring numerical capabilities specific to the topic. In this paper, we propose to conduct numerical capability diagnosis on a series of Numerical Question Answering systems and datasets. A series of numerical capabilities are highlighted, and corresponding dataset perturbations are designed. Empirical results indicate that existing systems are severely challenged by these perturbations. E.g., Graph2Tree experienced a 53.83% absolute accuracy drop against the “Extra” perturbation on ASDiv-a, and BART experienced 13.80% accuracy drop against the “Language” perturbation on the numerical subset of DROP. As a counteracting approach, we also investigate the effectiveness of applying perturbations as data augmentation to relieve systems’ lack of robust numerical capabilities. With experiment analysis and empirical studies, it is demonstrated that Numerical Question Answering with robust numerical capabilities is still to a large extent an open question. We discuss future directions of Numerical Question Answering and summarize guidelines on future dataset collection and system design.
doi = {10.48550/ARXIV.2211.07455}, author = {Xu, Jialiang and Zhou, Mengyu and He, Xinyi and Han, Shi and Zhang, Dongmei}, keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences}, title = {Towards Robust Numerical Question Answering: Diagnosing Numerical Capabilities of NLP Systems}, publisher = {arXiv}, year = {2022}, copyright = {arXiv.org perpetual, non-exclusive license}, } - LUNA: Language Understanding with Number Augmentations on Transformers via Number Plugins and Pre-trainingHongwei Han*, Jialiang Xu*, Mengyu Zhou, Yijia Shao, Shi Han, and 1 more authorJul 2022
Transformers are widely used in NLP tasks. However, current approaches to leveraging transformers to understand language expose one weak spot: Number understanding. In some scenarios, numbers frequently occur, especially in semi-structured data like tables. But current approaches to rich-number tasks with transformer-based language models abandon or lose some of the numeracy information - e.g., breaking numbers into sub-word tokens - which leads to many number-related errors. In this paper, we propose the LUNA framework which improves the numerical reasoning and calculation capabilities of transformer-based language models. With the number plugin of NumTok and NumBed, LUNA represents each number as a whole to model input. With number pre-training, including regression loss and model distillation, LUNA bridges the gap between number and vocabulary embeddings. To the best of our knowledge, this is the first work that explicitly injects numeracy capability into language models using Number Plugins. Besides evaluating toy models on toy tasks, we evaluate LUNA on three large-scale transformer models (RoBERTa, BERT, TabBERT) over three different downstream tasks (TATQA, TabFact, CrediTrans), and observe the performances of language models are constantly improved by LUNA. The augmented models also improve the official baseline of TAT-QA (EM: 50.15 -> 59.58) and achieve SOTA performance on CrediTrans (F1 = 86.17).
doi = {10.48550/ARXIV.2212.02691}, author = {Han*, Hongwei and Xu*, Jialiang and Zhou, Mengyu and Shao, Yijia and Han, Shi and Zhang, Dongmei}, keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences}, title = {LUNA: Language Understanding with Number Augmentations on Transformers via Number Plugins and Pre-training}, publisher = {arXiv}, year = {2022}, copyright = {arXiv.org perpetual, non-exclusive license}, }