0% Complete
English
صفحه اصلی
/
پانزدهمین کنفرانس بین المللی فناوری اطلاعات و دانش
Evaluating LLMs in Persian News Summarization
نویسندگان :
Arya VarastehNezhad
1
Reza Tavasoli
2
Mostafa Masumi
3
Seyed Soroush Majd
4
Mehrnoush Shamsfard
5
1- University of Tehran
2- University of South Carolina
3- Sharif University of Technology
4- shahid beheshti university
5- shahid beheshti university
کلمات کلیدی :
Text Summarization،Large Language Models،Persian News،LLM Evaluation،Natural Language Processing،Artificial Intelligence
چکیده :
This study evaluates the performance of eight Large Language Models (LLMs) in Persian news summarization: GPT-4o, Claude-3.5-Sonnet, Gemini-Pro-1.5, Llama-3.1-405B, Command-R, Mistral-Large-2, DeepSeek V2.5, and Gemma-2-9B. We assess these models across five news categories: Economy, International, Sports, Technology, and Social, using the pn_summary dataset. Our evaluation employs multiple metrics, including BERTScore and ROUGE, across two input conditions: article-only and article-with-title. Results show that Llama-3.1-405b performed best against reference summaries in the article-only setting, achieving the highest BERTScore F1 (50.60) and ROUGE-L (33.96) scores. Notably, including article titles helped models produce summaries which aligned more closely to the reference summary, increasing the average BERTScore F1 from 48.31 to 50.16 across most models. Moreover, when comparing generated summaries to original articles, Mistral-Large-2 led with a BERTScore F1 of 48.09. In category-specific analysis, Mistral-Large-2 consistently outperformed the reference summaries across all news categories, with the most significant improvement in the Economic category. This study provides valuable insights into the current capabilities of LLMs for Persian summarization, highlighting their potential and the impact of input structure on performance. Our findings contribute to the growing body of research on multilingual summarization and have practical implications for Persian language processing applications.
لیست مقالات
لیست مقالات بایگانی شده
AI-based Message Spam Classification Framework for Secure Autonomous Vehicles Communication
Riya Upadhyay - Mili Virani - Lakshit Pathak - Rajesh Gupta - Sudeep Tanwar - Hossein Shahinzadeh
Enhancing Software Effort Estimation with an Integrated Approach of Particle Swarm Optimization and Genetic Algorithms in Analogy-based Method
Ehsan Nasr - Keyvan Mohebbi
Beyond One-Hot: CatBoost for Heating and Cooling Load Prediction
Shayan Naghizadeh - Mohammad Saeed Rajabi - Ehsan Nazerfard
Design and modeling of a waiter robot
Amin Mohammadnejad - Hami Tourajizadeh
ساخت پیکره برچسب خورده گزارش های آسیب شناسی
مسلم سمیعی پاقلعه - مهرنوش شمس فرد
Sigma: A Secure Federated Network Gaming Platform
Keyhan Mohammadi - Reza Ebrahimi Atani
Smart City Standardized Evaluation :Use Case of Mashhad
Dr ُSeyed Mohammadreza Mirsarraf - Dr Alireza Yari - Dr Navid Zohdi - Ali Motevalizadeh
مدل یادگیری عمیق با بازنمایی چند مقیاسی زمان برای پیشبینی آبشار اطلاعاتی در شبکههای اجتماعی
مبینا پناهی - مهدی عمادی
Improving Training Stability in Variational Autoencoders Through the Integration of Score Matching Loss
Amirreza Mokhtari Rad - Pouya Ardehkhani - Hormehr Alborzi
روشی برای بهبود آزمون جهش پیشگویانه با در نظر گرفتن اثر داده های از دست رفته
طه رستمی - دکتر سعید جلیلی طه رستمی - سعید جلیلی -
بیشتر
ثمین همایش، سامانه مدیریت کنفرانس ها و جشنواره ها - نگارش 42.5.2