Home Journal of Social Computing Article
PDF (607.8 KB)
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript
Open Access

Impact of Large Language Models on Scholarly Publication Titles and Abstracts: A Comparative Analysis

Department of Cyber and Computing, Wrexham University, Wrexham, LL11 2AW, UK
School of Computing and Computing, STEM Faculty, The Open University, Milton Keynes, MK7 6AA, UK
Show Author Information

Abstract

Artificial Intelligence (AI) tools become essential across industries, distinguishing AI-generated from human-authored text is increasingly challenging. This study assesses the coherence of AI-generated titles and corresponding abstracts in anticipation of rising AI-assisted document production. Our main goal is to examine the correlation between original and AI-generated titles, emphasizing semantic depth and similarity measures, particularly in the context of Large Language Models (LLMs). We argue that LLMs have transformed research focus, dissemination, and citation patterns across five selected knowledge areas: Business Administration and Management (BAM), Computer Science and Information Technology (CS), Engineering and Material Science (EMS), Medicine and Healthcare (MH), and Psychology and Behavioral Sciences (PBS). We collected 15 000 titles and abstracts, narrowing the selection to 2000 through a rigorous multi-stage screening process adhering to our study’s criteria. Result shows that there is insufficient evidence to suggest that LLM outperforms human authors in article title generation or articles from the LLM era demonstrates a marked difference in semantic richness and readability compared to those from the pre-LLM. Instead, it asserts that LLM is a valuable tool and can assist researchers in generating titles. With LLM’s assistance, the researcher ensures that the content is reflective of the finalized abstract and core research themes, potentially increasing the impact and accessibility and readability of the academic work.

Keywords

semantic alignmentcosine similaritypatternslarge language modelstechnology impact

References

[1]

H. Zohny, J. McMillan, and M. King, Ethics of generative AI, Journal of Medical Ethics, vol. 49, no. 2, pp. 79–80, 2023.

Crossref Google Scholar
[2]

M. Z. M. Hurmuz, S. M. Jansen-Kosterink, I. Flierman, S. del Signore, G. Zia, S. del Signore, and B. Fard, Are social robots the solution for shortages in rehabilitation care? Assessing the acceptance of nurses and patients of a social robot, Comput. Hum. Behav. Artif. Hum., vol. 1, no. 2, p. 100017, 2023.

Crossref Google Scholar
[3]
C. R. Yeh, C. J. Wong, W. V. Chang, and C. S. Lai, Labor displacement in artificial intelligence era: A systematic literature review, Taiwan Journal of East Asian Studies, vol. 17, no. 2, pp. 25–75, 2020.
[4]

Z. Tekic and J. Füller, Managing innovation in the era of AI, Technol. Soc., vol. 73, p. 102254, 2023.

Crossref Google Scholar
[5]
S. Salter, P. L. Teh, and R. Hebblewhite, Human-created and AI-generated text: What’s left to uncover? in Intelligent Computing, K. Arai, ed. Cham. Switzerland: Springer, 2024, pp. 74–80.
Crossref
[6]
M. Jakesch, M. French, X. Ma, J. T. Hancock, and M. Naaman, AI-mediated communication: How the perception that profile text was written by AI affects trustworthiness, in Proc. 2019 CHI Conf. Human Factors in Computing Systems, Glasgow, UK, 2019, pp. 1−13.
Crossref
[7]

P. M. Mah, I. Skalna, and J. Muzam, Natural language processing and artificial intelligence for enterprise management in the era of industry 4.0, Appl. Sci., vol. 12, no. 18, p. 9207, 2022.

Crossref Google Scholar
[8]

R. Tolosana, S. Romero-Tapiador, R. Vera-Rodriguez, E. Gonzalez-Sosa, and J. Fierrez, DeepFakes detection across generations: Analysis of facial regions, fusion, and performance evaluation, Eng. Appl. Artif. Intell., vol. 110, p. 104673, 2022.

Crossref Google Scholar
[9]

K. Somoray and D. J. Miller, Providing detection strategies to improve human detection of deepfakes: An experimental study, Comput. Hum. Behav., vol. 149, p. 107917, 2023.

Crossref Google Scholar
[10]

N. Serki and S. Bolkan, The effect of clarity on learning: Impacting motivation through cognitive load, Commun. Educ., vol. 73, no. 1, pp. 29–45, 2024.

Crossref Google Scholar
[11]
B. L. Frye, Should using an AI text generator to produce academic writing be plagiarism? https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4292283, 2022.
[12]

Z. Elyoseph, D. Hadar-Shoval, K. Asraf, and M. Lvovsky, ChatGPT outperforms humans in emotional awareness evaluations, Front. Psychol., vol. 14, p. 1199058, 2023.

Crossref Google Scholar
[13]
A. Pegoraro, K. Kumari, H. Fereidooni, and A. R. Sadeghi, To ChatGPT, or not to ChatGPT: That is the question! arXiv preprint arXiv: 2304.01487, 2023.
[14]
N. Rodrigues, M. Burch, L. Di Silvestro, and D. Weiskopf, A visual analytics approach for word relevances in multiple texts, in Proc. 21st Int. Conf. Information Visualisation (IV), London, UK, 2017, pp. 1–7.
Crossref
[15]

M. S. Tullu, Writing the title and abstract for a research paper: Being concise, precise, and meticulous is the key, Saudi J. Anaesth., vol. 13, no. Suppl1, pp. S12–S17, 2019.

Crossref Google Scholar
[16]

H. H. Thorp, ChatGPT is fun, but not an author, Science, vol. 379, no. 6630, p. 313, 2023.

Crossref Google Scholar
[17]
T. Holtgraves, Implicit communication of emotions via written text messages, Computers in Human Behavior Reports, vol. 7, p. 100219, 2022.
Crossref
[18]
W. Liang, Y. Zhang, H. Cao, B. Wang, D. Ding, X. Yang, K. Vodrahalli, S. He, D. Smith, Y. Yin, et al., Can large language models provide useful feedback on research papers? A large-scale empirical analysis, arXiv preprint arXiv: 2310.01783, 2023.
Crossref
[19]
P. L. Teh, P. Rayson, I. Pak, S. Piao, J. S. Y. Ho, A. Moore, and Y. Cheah, Textual variations affect human judgements of sentiment values, Electron. Commer Res. Appl., vol. 53, p. 101149, 2022.
Crossref
[20]
S. Jiang, R. E. Robertson, and C. Wilson, Reasoning about political bias in content moderation, in Proc. 34th AAAI Conference on Artificial Intelligence (AAAI-20), New York, NY, USA, 2020, pp. 13669–13672.
Crossref
[21]

Q. Zhou, B. Li, L. Han, and M. Jou, Talking to a bot or a wall? How chatbots vs. human agents affect anticipated communication quality, Computers in Human Behavior, vol. 143, p. 107674, 2023.

Crossref Google Scholar
[22]
M. Khalil and E. Er, Will ChatGPT get you caught? Rethinking of plagiarism detection, in Learning and Collaboration Technologies, P. Zaphiris and A. Ioannou, eds. Cham, Switzerland: Springer, 2023, pp. 475–487.
Crossref
[23]
K. Cao, C. Chen, S. Baltes, C. Treude, and X. Chen, Automated query reformulation for efficient search based on query logs from stack overflow, in Proc. 43rd Int. Conf. Software Engineering, Madrid, Spain, 2021, pp. 1273–1285.
Crossref
[24]
B. D. Lund and T. Wang, Chatting about ChatGPT: How may AI and GPT impact academia and libraries? https://ssrn.com/abstract=4333415, 2023.
Crossref
[25]
S. Syed and M. Spruit, Full-text or abstract? Examining topic coherence scores using latent dirichlet allocation, in Proc. IEEE Int. Conf. Data Science and Advanced Analytics (DSAA), Tokyo, Japan, 2017, pp. 165–174.
Crossref
[26]
V. P. E. Reyes, Exploring the use of the phoneme frequency scale method in determining word difficulty levels and readability scores, in Proc. 2019 7th Int. Conf. Information and Education Technology, Aizu-Wakamatsu, Japan, 2019, pp. 284–288.
Crossref
[27]
S. Albitar, S. Fournier, and B. Espinasse, An effective TF/IDF-based text-to-text semantic similarity measure for text classification, in Web Information Systems Engineering—WISE 2014, B. Benatallah, A. Bestavros, Y. Manolopoulos, A. Vakali, and Y. Zhang, eds. Cham, Switzerland: Springer, 2014, pp. 105–114.
Crossref
[28]
X. Ao, X. Yu, D. Liu, and H. Tian, News keywords extraction algorithm based on TextRank and classified TF-IDF, in Proc. Int. Wireless Communications and Mobile Computing (IWCMC), Limassol, Cyprus, 2020, pp. 1364–1369.
Crossref
[29]
Y. Li and B. Shen, Research on sentiment analysis of microblogging based on LSA and TF-IDF, in Proc. 3rd IEEE Int. Conf. Computer and Communications (ICCC), Chengdu, China, 2017, pp. 2584–2588.
Crossref
[30]
B. Guo, X. Zhang, Z. Wang, M. Jiang, J. Nie, Y. Ding, J. Yue, and Y. Wu, How close is ChatGPT to human experts? Comparison corpus, evaluation, and detection, arXiv preprint arXiv: 2301.07597, 2023.
[31]
Y. Ma, J. Liu, F. Yi, Q. Cheng, Y. Huang, W. Lu, and X. Liu, AI vs. human—Differentiation analysis of scientific content generation, arXiv preprint arXiv: 2301.10416, 2023.
[32]

R. Saptono, H. Prasetyo, and A. Irawan, Combination of cosine similarity method and conditional probability for plagiarism detection in the thesis documents vector space model, Journal of Telecommunication, vol. 10, nos. 2–4, pp. 139–143, 2018.

Google Scholar
[33]
M. Ali Fauzi, D. C. Utomo, B. D. Setiawan, and E. S. Pramukantoro, Automatic essay scoring system using N-gram and cosine similarity for gamification based E-learning, in Proc. Int. Conf. Advances in Image Processing, Bangkok, Thailand, 2017, pp. 151–155.
Crossref
[34]

S. Tata and J. M. Patel, Estimating the selectivity of TF-IDF based cosine similarity predicates, SIGMOD Rec., vol. 36, no. 4, pp. 75–80, 2007.

Crossref Google Scholar
[35]
S. T. Yuan and J. Sun, Ontology-based structured cosine similarity in speech document summarization, in Proc. IEEE/WIC/ACM Int. Conf. Web Intelligence (WI’04), Beijing, China, 2004, pp. 508–513.
[36]
K. Jain, S. Gandhi, S. Singhal, and S. Rajput, Semantic image captioning using cosine similarity ranking with semantic search, in Proc. 2023 15th Int. Conf. Contemporary Computing, Noida, India, 2023, pp. 220–223.
Crossref
[37]
S. Akter, A. E. Siam, K. M. M. Monir, M. H. K. Mehedi, and A. A. Rasel, Bengali movie recommendation system using K nearest neighbor and cosine similarity, in Proc. 2023 9th Int. Conf. Computer Technology Applications, Vienna, Austria, 2023, pp. 25–29.
Crossref
[38]

L. S. Lo, The art and science of prompt engineering: A new literacy in the information age, Internet Ref. Serv. Q., vol. 27, no. 4, pp. 203–210, 2023.

Crossref Google Scholar
[39]

T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Language models are few-shot learners, Adv. Neural Inf. Process. Syst., vol. 33, pp. 1877–1901, 2020.

Google Scholar
[40]

D. Peters, K. Vold, D. Robinson, and R. A. Calvo, Responsible AI—Two frameworks for ethical design practice, IEEE Trans. Technol. Soc., vol. 1, no. 1, pp. 34–47, 2020.

Crossref Google Scholar
[41]
I. Poola, Overcoming ChatGPTs inaccuracies with pre-trained AI prompt engineering sequencing process, International Journal of Technology and Emerging Sciences, vol. 3, no. 3, pp. 16–19. 2023.
[42]
A. W. Harzing, The Publish or Perish Book: Your Guide to Effective and Responsible Citation Analysis. Melbourne, Australia: Tarma Software Research Pty Limited, 2010.
[43]

H. Huang, D. Zhu, and X. Wang, Evaluating scientific impact of publications: Combining citation polarity and purpose, Scientometrics, vol. 127, no. 9, pp. 5257–5281, 2022.

Crossref Google Scholar
[44]

B. I. Hutchins, X. Yuan, J. M. Anderson, and G. M. Santangelo, Relative citation ratio (RCR): A new metric that uses citation rates to measure influence at the article level, PLoS Biol., vol. 14, no. 9, p. e1002541, 2016.

Crossref Google Scholar
[45]
D. W. Aksnes, L. Langfeldt, and P. Wouters, Citations, citation indicators, and research quality: An overview of basic concepts and theories, SAGE Open, vol. 9, no. 1, p. 2158244019829575, 2019.
Crossref
[46]

M. R. Dougherty and Z. Horne, Citation counts and journal impact factors do not capture some indicators of research quality in the behavioural and brain sciences, R. Soc. Open Sci., vol. 9, no. 8, p. 220334, 2022.

Crossref Google Scholar
[47]

A. P. Akella, H. Alhoori, P. R. Kondamudi, C. Freeman, and H. Zhou, Early indicators of scientific impact: Predicting citations with altmetrics, Journal of Informatics, vol. 15, no. 2, p. 101128, 2021.

Crossref Google Scholar
[48]
F. Rahutomo, T. Kitasuka, and M. Aritsugi, Semantic cosine similarity, in Proc. 7th International Student Conference on Advanced Science and Technology ICAST, Seoul, Republic of Korea, 2012, p. 1.
[49]
T. Schmohl, A. Watanabe, N. Fröhlich, and D. Herzberg, How artificial intelligence can improve the academic writing of students, https://conference.pixel-online.net/library_scheda.php?id_abs=4769, 2020.
[50]
X. Yu, Y. Qi, K. Chen, G. Chen, X. Yang, P. Zhu, W. Zhang, and N. Yu, GPT paternity test: GPT generated text detection with GPT genetic inheritance, arXiv preprint arXiv: 2305.12519, 2023.
[51]

G. Le Mens, B. Kovács, M. T. Hannan, and G. Pros, Uncovering the semantics of concepts using GPT-4, Proc. Natl. Acad. Sci. U.S.A., vol. 120, no. 49, p. e2309350120, 2023.

Crossref Google Scholar
Journal of Social Computing
Volume 5 Issue 2,
June 2024
Pages 105-121
DOI: 10.23919/JSC.2024.0011
Cite this article:
Teh PL, Uwasomba CF. Impact of Large Language Models on Scholarly Publication Titles and Abstracts: A Comparative Analysis. Journal of Social Computing, 2024, 5(2): 105-121. https://doi.org/10.23919/JSC.2024.0011
Metrics & Citations  
Article History
Copyright
Rights and Permissions
Return