Citation :

AnNing, Mazida Ahmad, Huda lbrahim, Wang Zhuoxian, Xu Jie, 2024. "Pre-Trained Models in Natural Language Processing: Progress after BERT" ESP International Journal of Communication Engineering & Electronics Technology (ESP- IJCEET)  Volume 2, Issue 1 : 35-46

.

Abstract :

With the continuous development of the natural language processing field, the pre-training model has achieved remarkable results in text processing tasks. As a representative, the BERT model achieves leading performance on multiple natural language processing tasks by pre-training and fine-tuning. However, with the deepening of the research, the BERT model has also exposed some problems, such as the high training cost and the large model scale. Therefore, the investigators proposed a series of improved models, such as XLNet, RoBERTa, ALBERT, and ELECTRA, to address the shortcomings of the BERT model. These models innovate in model architecture, training strategies, and optimization methods, and achieve better performance. In addition, the researchers proposed a variety of pre-trained models, such as fine-tuning, domain adaptation, transfer learning, and multi-task learning, to further improve the performance of the model on specific tasks. However, pre-trained models still face challenges such as high training costs and poor model interpretability. Therefore, future research directions can focus on reducing training costs, improving model interpretability, and further optimizing performance on specific tasks. This paper summarizes the development and application of pre-trained models in natural language processing, introduces some important advances after BERT, and explores the challenges of pre-trained models and future development directions.

References :

[1] Li Qiong, Chen Jingxian, Wu Yuan, Lv Lingling, Ying Haifeng, Zhu Wenhua, Xu Jiayue, Ruan Ming, Guo Yuanbiao, Zhu Weirong, Yin Shipeng, Zheng Lan. Pharmacological study of the mechanism of colorectal cancer liver metastasis based on TCGA database [J]. Journal of Gansu University of Traditional Chinese Medicine, 2023,40 (03): 1-11.
[2] LI Zhi-hao,HU Jun-wei,LI Xu,CHEN Yue-lai. Mechanism progress of acupuncture and moxibustion for chronic prostatitis: Progress in the research mechanism of acupuncture in the treatment of chronic prostatitis [J].World Journal of Acupuncture – Moxibustion,2021,31(4).
[3] Julia D. Hammer, Christopher Palma, KeriAnn Rubin, Alice Flarend,Yann Shiou Ong,Chrysta Ghent,Timothy Gleason,Scott McDonald,Brandon Botzer,Tanya Furman.Evaluating a learning progression for the solar system: Progress along gravity and dynamical properties dimensions[J].Science Education,2020,104(3).
[4] Ponna. Research on the intelligent identification method of chemical bond energy data for scientific papers [D]. University of Chinese Academy of Sciences (Literature and Intelligence Center of Chinese Academy of Sciences), 2020.
[5] Zhang WeiYu,Zhou JianHua,Wang HuanRui,Mu Qing,Wang Qi,Xu KeXin,Xu Tao,Hu Hao.[Research Progress of the Roles of Ubiquitination/Deubiquitination in Androgen Receptor Abnormalities and Prostate Cancer].[J].Zhongguo yi xue ke xue yuan xue bao.Acta Academiae Medicinae Sinicae,2020,42(2).
[6] Benefit of Tolvaptan on Time to End-stage Renal Disease (ESRD) for Patients with Rapidly Progressing Autosomal Dominant Polycystic Kidney Disease (ADPKD): A Disease Progression Model[J].American Journal of Kidney Diseases,2020,75(4).
[7] Tao Yaoye,Wang Jianguo, Xu Xiao.Emerging and Innovative Theranostic Approaches for Mesoporous Silica Nanoparticles in Hepatocellular Carcinoma: Current Status and Advances.[J].Frontiers in bioengineering and biotechnology,2020,8.
[8] Sofia Ajeganova, Thomas Gustafsson, Linnea Lindberg, Ingiäld Hafström, Johan Frostegåd.P159 Similar progression of carotid intima-media thickness in 7-year surveillance of patients with mild SLE and controls, but this progression in patients is still promoted by dyslipidemia, hypertension, history of lupus nephritis and a higher prednisone usage[J].Lupus Science & Medicine,2020,7(Suppl 1).
[9] Agriculture; Findings from Anhui University Advance Knowledge in Agriculture (Recent advances in Raman technology with applications in agriculture, food, and biosystems: A review)[J].Journal of Engineering,2020.
[10] Disease Attributes - Disease Progression; Researchers from Fuwai Hospital Report on Findings in Disease Progression (Elevated Plasma Beta-hydroxybutyrate Predicts Adverse Outcomes and Disease Progression In Patients With Arrhythmogenic Cardiomyopathy)[J].News of Science,2020.

Keywords :

Pre-Training Model, BERT, XLNet, RoBERTa, ALBERT, ELECTRA, Improvement Method, Challenge, Outlook.