Abstract

E-commerce shopping advisors operate in product search settings where user needs are often expressed through short queries, incomplete descriptions, and specific attribute constraints. Large language models (LLMs) are increasingly used to address these challenges, yet their role in the shopping pipeline remains under-specified, especially with respect to whether they add more value in retrieval or in reranking. This paper addresses that question through a stage-based system comparison rather than a model-development study. We construct an aligned benchmark by joining Amazon Shopping Queries relevance judgments with Amazon Reviews 2023 Electronics metadata and compare BM25, dense retrieval, LLM reranking, and LLM retrieval followed by reranking, with evidence augmentation examined in a separate extension study. Across 10,961 benchmark queries, dense retrieval provides a modest improvement over BM25. In the matched-query comparison, LLM reranking preserves the same Hit@5 and Recall@5 as dense retrieval but increases NDCG@5 from 0.922099 to 0.960640. Extending the LLM from reranking to retrieval yields only a marginal additional gain, with Recall@5 increasing from 0.696785 to 0.700000 and NDCG@5 from 0.960640 to 0.961210. These results indicate that the main value of the LLM in this pipeline lies in improving the order of retrieved candidates rather than in substantially expanding retrieval coverage. Case analysis further shows that the LLM is particularly useful for product-type disambiguation and fine-grained attribute matching, while failure cases arise when the initial candidate set is weak. The extension study further shows that review-augmented reranking does not outperform metadata-only reranking and slightly reduces NDCG@5. The findings suggest that LLMs are most valuable as fine-grained rerankers in the shopping advisor pipeline, and that evidence augmentation must be selective and query-aligned.

Download

Share

COinS