Location
Hilton Waikoloa Village, Hawaii
Event Website
https://hicss.hawaii.edu/
Start Date
7-1-2025 12:00 AM
End Date
10-1-2025 12:00 AM
Description
10-K reports are a very important source of information in finance. Unfortunately, due to their text length they can hardly be analyzed by state-of-the-art transformer-based methods. In this paper, we aim to address this by combining the fields of efficient attention mechanisms, knowledge distillation (KD), and ensembling. Our five-step approach, DiFiLE, first pre-processes the data and splits it into data chunks based on the report items. Then, for each chunk, we estimate a teacher Longformer model. This is followed by KD and the generation of the corresponding student models. Finally, we aggregate the results from the chunks with ensembling and in particular stacking. We evaluate DiFiLE on the 10-K reports of the DJIA companies. The results show high performance of the teacher model, which is then well mimicked by its distilled version, requiring 30% fewer resources.
Recommended Citation
Hristova, Diana and Satani, Nirav, "DiFiLE: A Knowledge-Distillation Longformer Model for Finance with Ensembling" (2025). Hawaii International Conference on System Sciences 2025 (HICSS-58). 5.
https://aisel.aisnet.org/hicss-58/da/nlp_and_llms/5
DiFiLE: A Knowledge-Distillation Longformer Model for Finance with Ensembling
Hilton Waikoloa Village, Hawaii
10-K reports are a very important source of information in finance. Unfortunately, due to their text length they can hardly be analyzed by state-of-the-art transformer-based methods. In this paper, we aim to address this by combining the fields of efficient attention mechanisms, knowledge distillation (KD), and ensembling. Our five-step approach, DiFiLE, first pre-processes the data and splits it into data chunks based on the report items. Then, for each chunk, we estimate a teacher Longformer model. This is followed by KD and the generation of the corresponding student models. Finally, we aggregate the results from the chunks with ensembling and in particular stacking. We evaluate DiFiLE on the 10-K reports of the DJIA companies. The results show high performance of the teacher model, which is then well mimicked by its distilled version, requiring 30% fewer resources.
https://aisel.aisnet.org/hicss-58/da/nlp_and_llms/5