Revisiting Pre-Trained Models for Chinese Natural Language Processing

Bidirectional Encoder Representations from Transformers (BERT) has shown marvelous improvements across various NLP tasks, and consecutive variants have been proposed to further improve the performance of the pre-trained language models. In this paper, we target on revisiting Chinese pre-trained language models to examine their effectiveness in a non-English language and release the Chinese pre-trained language model series to the community. We also propose a simple but effective model called MacBERT, which improves upon RoBERTa in several ways, especially the masking strategy that adopts MLM as correction (Mac). We carried out extensive experiments on eight Chinese NLP tasks to revisit the existing pre-trained language models as well as the proposed MacBERT. Experimental results show that MacBERT could achieve state-of-the-art performances on many NLP tasks, and we also ablate details with several findings that may help future research. Resources available: https://github.com/ymcui/MacBERTComment: 12 pages, to appear at Findings of EMNLP 202

A Span-Extraction Dataset for Chinese Machine Reading Comprehension

Machine Reading Comprehension (MRC) has become enormously popular recently and has attracted a lot of attention. However, the existing reading comprehension datasets are mostly in English. In this paper, we introduce a Span-Extraction dataset for Chinese machine reading comprehension to add language diversities in this area. The dataset is composed by near 20,000 real questions annotated on Wikipedia paragraphs by human experts. We also annotated a challenge set which contains the questions that need comprehensive understanding and multi-sentence inference throughout the context. We present several baseline systems as well as anonymous submissions for demonstrating the difficulties in this dataset. With the release of the dataset, we hosted the Second Evaluation Workshop on Chinese Machine Reading Comprehension (CMRC 2018). We hope the release of the dataset could further accelerate the Chinese machine reading comprehension research. Resources are available: https://github.com/ymcui/cmrc2018Comment: 6 pages, accepted as a conference paper at EMNLP-IJCNLP 2019 (short paper

Suppression of blow-up in 3-D Keller-Segel model via Couette flow in whole space

In this paper, we study the 3-D parabolic-parabolic and parabolic-elliptic Keller-Segel models with Couette flow in $\mathbb{R}^3$. We prove that the blow-up phenomenon of solution can be suppressed by enhanced dissipation of large Couette flows. Here we develop Green's function method to describe the enhanced dissipation via a more precise space-time structure and obtain the global existence together with pointwise estimates of the solutions. The result of this paper shows that the enhanced dissipation exists for all frequencies in the case of whole space and it is reason that we obtain global existence for 3-D Keller-Segel models here. It is totally different from the case with the periodic spatial variable $x$ in [2,10]. This paper provides a new methodology to capture dissipation enhancement and also a surprising result which shows a totally new mechanism.Comment: 22 pag

IDOL: Indicator-oriented Logic Pre-training for Logical Reasoning

In the field of machine reading comprehension (MRC), existing systems have surpassed the average performance of human beings in many tasks like SQuAD. However, there is still a long way to go when it comes to logical reasoning. Although some methods for it have been put forward, they either are designed in a quite complicated way or rely too much on external structures. In this paper, we proposed IDOL (InDicator-Oriented Logic Pre-training), an easy-to-understand but highly effective further pre-training task which logically strengthens the pre-trained models with the help of 6 types of logical indicators and a logically rich dataset LGP (LoGic Pre-training). IDOL achieves state-of-the-art performance on ReClor and LogiQA, the two most representative benchmarks in logical reasoning MRC, and is proven to be capable of generalizing to different pre-trained models and other types of MRC benchmarks like RACE and SQuAD 2.0 while keeping competitive general language understanding ability through testing on tasks in GLUE. Besides, at the beginning of the era of large language models, we take several of them like ChatGPT into comparison and find that IDOL still shows its advantage.Comment: Accepted to the Findings of ACL 202

Tanshinone IIa protects retinal endothelial cells against mitochondrial fission induced by methylglyoxal through glyoxalase 1

Advanced glycation end products (AGEs) play an important role in the onset of diabetic retinopathy. Therefore, in the current study, we investigate whether and how Tanshinone IIa (Tan IIa) from Salvia miltiorrhiza protects bovine retinal endothelial cells (BRECs) against methylglyoxal (MGO) mediated cell dysfunction. The results showed that MGO reduced cell viability in dose dependent manner. The treatment of Tan IIa (50 μM) significantly improved cell viability induced by MGO in BRECs. MGO increased cellular reactive oxygen species formation and cellular nitric oxide (NO) level; enhanced nox1 and iNOS mRNA levels; inhibited prdx1 mRNA level. The treatment of Tan IIa effectually ameliorated cellular oxidative stress. Exposure of MGO resulted in mitochondrial fission and decrease of opa1 and mfn1. No significant difference in mRNA levels of mfn2 and drp1 was detected between MGO and medium. Tan IIa reduced mitochondrial fragmentation, enhanced the mRNA levels of mfn1 and opa1 in MGO cultured BRECs. The short time exposure of cellular antioxidatants, dimethylthiourea (10 mM) and tiron (10 mM) had no effect on mitochondrial fission although they ameliorated cellular reactive oxygen species level. Moreover, overexpression of glyoxalase 1 (GLO1) increased key proteins of mitochondrial fusion, including opa1 and mfn1 in BRECs cultured with MGO. However, inhibition of GLO1 by siRNA abolished the effect of Tan IIa on induction of mitochondrial fusion in MGO cultured BRECs. In conclusion, MGO caused the injury of retinal endothelial cells through induction of mitochondrial dysfunction and mitochondrial fission, the treatment of Tan IIa ameliorated mitochondrial dysfunction and fission induced by AGEs through enhancing GLO1

Convolutional Spatial Attention Model for Reading Comprehension with Multiple-Choice Questions

Machine Reading Comprehension (MRC) with multiple-choice questions requires the machine to read given passage and select the correct answer among several candidates. In this paper, we propose a novel approach called Convolutional Spatial Attention (CSA) model which can better handle the MRC with multiple-choice questions. The proposed model could fully extract the mutual information among the passage, question, and the candidates, to form the enriched representations. Furthermore, to merge various attention results, we propose to use convolutional operation to dynamically summarize the attention values within the different size of regions. Experimental results show that the proposed model could give substantial improvements over various state-of-the-art systems on both RACE and SemEval-2018 Task11 datasets.Comment: 8 pages. Accepted as a conference paper at AAAI-19 Technical Trac