Retrieval Methods for Persistent AI Agent Memory in Small Collections: A Review of File Hierarchies, Sparse Search, and Vector Search
Author
Mahipal J. Suchar, Ami M. Mehta
Abstract
Persistent AI agents need to retrieve past information from an external memory store. For a small, human-maintained collection, the choice among folders, keyword search and dense embeddings is not settled by results from large retrieval benchmarks. This structured narrative review synthesizes 61 research and survey works on retrieval-augmented generation, indexing, agent memory, hierarchical and graph retrieval, evaluation, and local deployment. It explains where each method helps, identifies differences between organizing files and ranking them, and examines how the studies were evaluated. Dense retrieval can bridge different wording; explicit file paths support inspection; keyword search offers a strong baseline; and hybrid systems combine these properties. However, published outcomes use different data, models, and hardware, so their scores cannot be directly compared for a small memory store. The reviewed work supports a test of source accuracy, unsupported-answer rejection, updates, latency, memory use, and human maintenance under shared conditions. This article presents a literature synthesis and a reproducible comparison design; it reports no new experiment.
Keywords
retrieval-augmented generation; agent memory; file hierarchy; vector retrieval; sparse retrieval; low-resource systems
Full Text:
References
1)Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems, 33, 9459–9474. https://arxiv.org/abs/2005.11401
2)Karpukhin, V., Oğuz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W.-T. (2020). Dense passage retrieval for open-domain question answering. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. https://aclanthology.org/2020.emnlp-main.550/
3)Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using siamese BERT-Networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. https://aclanthology.org/2019.emnlp-main.398/
4)Shi, Z., Mei, K., Jin, M., Su, Y., Zuo, C., Hua, W., Xu, W., Ren, Y., Liu, Z., Du, M., Deng, D., & Zhang, Y. (2024). From commands to prompts: LLM-Based semantic file system for AIOS [Preprint]. arXiv. https://arxiv.org/abs/2410.11843
5)Guo, Z., Xia, L., Yu, Y., Ao, T., & Huang, C. (2024). LightRAG: Simple and fast retrieval-augmented generation [Preprint]. arXiv. https://arxiv.org/abs/2410.05779
6)Guu, K., Lee, K., Tung, Z., Pasupat, P., & Chang, M.-W. (2020). REALM: Retrieval-augmented language model pre-training [Preprint]. arXiv. https://arxiv.org/abs/2002.08909
7)Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., van den Driessche, G., Lespiau, J.-B., Damoc, B., Clark, A., Casas, D. D. L., Guy, A., Menick, J., Ring, R., Hennigan, T., Huang, S., Maggiore, L., Jones, C., Cassirer, A., . . . Sifre, L. (2022). Improving language models by retrieving from trillions of tokens. Proceedings of the International Conference on Machine Learning. https://arxiv.org/abs/2112.04426
8)Khandelwal, U., Levy, O., Jurafsky, D., Zettlemoyer, L., & Lewis, M. (2020). Generalization through memorization: Nearest neighbor language models. International Conference on Learning Representations. https://arxiv.org/abs/1911.00172
9)Izacard, G., & Grave, E. (2021). Leveraging passage retrieval with generative models for open domain question answering. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics. https://aclanthology.org/2021.eacl-main.74/
10)Jiang, Z., Xu, F. F., Gao, L., Sun, Z., Liu, Q., Dwivedi-Yu, J., Yang, Y., Callan, J., & Neubig, G. (2023). Active retrieval augmented generation. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. https://aclanthology.org/2023.emnlp-main.495/
11)Asai, A., Wu, Z., Wang, Y., Sil, A., & Hajishirzi, H. (2024). Self-RAG: Learning to retrieve, generate, and critique through self-reflection. International Conference on Learning Representations. https://openreview.net/forum?id=hSyW5go0v8
12)Yan, S.-Q., Gu, J.-C., Zhu, Y., & Ling, Z.-H. (2024). Corrective retrieval augmented generation [Preprint]. arXiv. https://arxiv.org/abs/2401.15884
13)Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., & Wang, H. (2023). Retrieval-augmented generation for large language models: A survey [Preprint]. arXiv. https://arxiv.org/abs/2312.10997
14)Fan, W., Ding, Y., Ning, L., Wang, S., Li, H., Yin, D., Chua, T.-S., & Li, Q. (2024). A survey on RAG meeting LLMs: Towards retrieval-augmented large language models. Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. https://arxiv.org/abs/2405.06211
15)Wu, S., Xiong, Y., Cui, Y., Wu, H., Chen, C., Yuan, Y., Huang, L., Liu, X., Kuo, T.-W., Guan, N., & Xue, C. J. (2024). Retrieval-augmented generation for natural language processing: A survey [Preprint]. arXiv. https://arxiv.org/abs/2407.13193
16)Gupta, S., Ranjan, R., & Singh, S. N. (2024). A comprehensive survey of retrieval-augmented generation (RAG): Evolution, current landscape and future directions [Preprint]. arXiv. https://arxiv.org/abs/2410.12837
17)Zhao, W. X., Liu, J., Ren, R., & Wen, J.-R. (2022). Dense text retrieval based on pretrained language models: A survey [Preprint]. arXiv. https://arxiv.org/abs/2211.14876
18)Pan, J. J., Wang, J., & Li, G. (2024). Survey of vector database management systems. The VLDB Journal, 33(5), 1591–1615. https://arxiv.org/abs/2310.14021
19)Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems. https://arxiv.org/abs/1706.03762
20)Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics. https://arxiv.org/abs/1810.04805
21)Robertson, S., & Zaragoza, H. (2009). The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 3(4), 333–389. https://doi.org/10.1561/1500000019
22)Formal, T., Piwowarski, B., & Clinchant, S. (2021). SPLADE: Sparse lexical and expansion model for first stage ranking. Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval. https://doi.org/10.1145/3404835.3463098
23)Xiong, L., Xiong, C., Li, Y., Tang, K.-F., Liu, J., Bennett, P., Ahmed, J., & Overwijk, A. (2021). Approximate nearest neighbor negative contrastive learning for dense text retrieval. International Conference on Learning Representations. https://arxiv.org/abs/2007.00808
24)Izacard, G., Caron, M., Hosseini, L., Riedel, S., Bojanowski, P., Joulin, A., & Grave, E. (2022). Unsupervised dense information retrieval with contrastive learning [Preprint]. arXiv. https://arxiv.org/abs/2112.09118
25)Gao, L., Ma, X., Lin, J., & Callan, J. (2023). Precise zero-shot dense retrieval without relevance labels. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. https://aclanthology.org/2023.acl-long.99/
26)Khattab, O., & Zaharia, M. (2020). ColBERT: Efficient and effective passage search via contextualized late interaction over BERT. Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval. https://arxiv.org/abs/2004.12832
27)Santhanam, K., Khattab, O., Saad-Falcon, J., Potts, C., & Zaharia, M. (2022). ColBERTv2: Effective and efficient retrieval via lightweight late interaction. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics. https://arxiv.org/abs/2112.01488
28)Malkov, Y. A., & Yashunin, D. A. (2020). Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence. https://arxiv.org/abs/1603.09320
29)Johnson, J., Douze, M., & Jégou, H. (2021). Billion-scale similarity search with GPUs. IEEE Transactions on Big Data. https://arxiv.org/abs/1702.08734
30)Douze, M., Guzhva, A., Deng, C., Johnson, J., Szilvasy, G., Mazaré, P.-E., Lomeli, M., Hosseini, L., & Jégou, H. (2024). The Faiss library [Preprint]. arXiv. https://arxiv.org/abs/2401.08281
31)Wang, S., Zhuang, S., & Zuccon, G. (2021). BERT-Based dense retrievers require interpolation with BM25 for effective passage retrieval. Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval. https://doi.org/10.1145/3471158.3472233
32)Cormack, G. V., Clarke, C. L. A., & Buettcher, S. (2009). Reciprocal rank fusion outperforms condorcet and individual rank learning methods. Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval. https://doi.org/10.1145/1571941.1572114
33)Thakur, N., Reimers, N., Rücklé, A., Srivastava, A., & Gurevych, I. (2021). BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. NeurIPS Datasets and Benchmarks Track. https://arxiv.org/abs/2104.08663
34)Park, J. S., O'Brien, J. C., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023). Generative agents: Interactive simulacra of human behavior. Proceedings of the ACM Symposium on User Interface Software and Technology. https://doi.org/10.1145/3586183.3606763
35)Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., & Yao, S. (2023). Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems. https://arxiv.org/abs/2303.11366
36)Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., & Anandkumar, A. (2023). Voyager: An open-ended embodied agent with large language models [Preprint]. arXiv. https://arxiv.org/abs/2305.16291
37)Sumers, T. R., Yao, S., Narasimhan, K., & Griffiths, T. L. (2024). Cognitive architectures for language agents. Transactions on Machine Learning Research. https://arxiv.org/abs/2309.02427
38)Wu, Y., Liang, S., Zhang, C., Wang, Y., Zhang, Y., Guo, H., Tang, R., & Liu, Y. (2025). From human memory to AI memory: A survey on memory mechanisms in the era of LLMs [Preprint]. arXiv. https://arxiv.org/abs/2504.15965
39)Packer, C., Wooders, S., Lin, K., Fang, V., Patil, S. G., Stoica, I., & Gonzalez, J. E. (2023). MemGPT: Towards LLMs as operating systems [Preprint]. arXiv. https://arxiv.org/abs/2310.08560
40)Zhong, W., Guo, L., Gao, Q., Ye, H., & Wang, Y. (2024). Enhancing large language models with long-term memory. Proceedings of the AAAI Conference on Artificial Intelligence. https://ojs.aaai.org/index.php/AAAI/article/view/29946
41)Chhikara, P., Khant, D., Aryan, S., Singh, T., & Yadav, D. (2025). Mem0: Building production-ready AI agents with scalable long-term memory [Preprint]. arXiv. https://arxiv.org/abs/2504.19413
42)Rasmussen, P., Paliychuk, P., Beauvais, T., Ryan, J., & Chalef, D. (2025). Zep: A temporal knowledge graph architecture for agent memory [Preprint]. arXiv. https://arxiv.org/abs/2501.13956
43)Xu, W., Liang, Z., Mei, K., Gao, H., Tan, J., & Zhang, Y. (2025). A-MEM: Agentic memory for LLM agents [Preprint]. arXiv. https://arxiv.org/abs/2502.12110
44)Sarthi, P., Abdullah, S., Tuli, A., Khanna, S., Goldie, A., & Manning, C. D. (2024). RAPTOR: Recursive abstractive processing for tree-organized retrieval. International Conference on Learning Representations. https://arxiv.org/abs/2401.18059
45)Edge, D., Trinh, H., Cheng, N., Bradley, J., Chao, A., Mody, A., Truitt, S., Metropolitansky, D., Ness, R. O., & Larson, J. (2024). From local to global: A graph RAG approach to query-focused summarization [Preprint]. arXiv. https://arxiv.org/abs/2404.16130
46)Gutiérrez, B. J., Shu, Y., Gu, Y., Yasunaga, M., & Su, Y. (2024). HippoRAG: Neurobiologically inspired long-term memory for large language models. Advances in Neural Information Processing Systems. https://arxiv.org/abs/2405.14831
47)Fan, T., Wang, J., Ren, X., & Huang, C. (2025). MiniRAG: Towards extremely simple retrieval-augmented generation [Preprint]. arXiv. https://arxiv.org/abs/2501.06713
48)Maharana, A., Lee, D.-H., Tulyakov, S., Bansal, M., Barbieri, F., & Fang, Y. (2024). Evaluating very long-term conversational memory of LLM agents. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. https://aclanthology.org/2024.acl-long.747/
49)Tang, Y., & Yang, Y. (2024). MultiHop-RAG: Benchmarking retrieval-augmented generation for multi-hop queries. Conference on Language Modeling. https://arxiv.org/abs/2401.15391
50)Wu, D., Wang, H., Yu, W., Zhang, Y., Chang, K.-W., & Yu, D. (2024). LongMemEval: Benchmarking chat assistants on long-term interactive memory [Preprint]. arXiv. https://arxiv.org/abs/2410.10813
51)Es, S., James, J., Espinosa-Anke, L., & Schockaert, S. (2024). RAGAS: Automated evaluation of retrieval augmented generation. Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations. https://aclanthology.org/2024.eacl-demo.16/
52)Saad-Falcon, J., Khattab, O., Potts, C., & Zaharia, M. (2024). ARES: An automated evaluation framework for retrieval-augmented generation systems. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics. https://aclanthology.org/2024.naacl-long.20/
53)Chen, J., Lin, H., Han, X., & Sun, L. (2024). Benchmarking large language models in retrieval-augmented generation. Proceedings of the AAAI Conference on Artificial Intelligence. https://ojs.aaai.org/index.php/AAAI/article/view/29728
54)Ru, D., Qiu, L., Hu, X., Zhang, T., Shi, P., Chang, S., Jiayang, C., Wang, C., Sun, S., Li, H., Zhang, Z., Wang, B., Jiang, J., He, T., Wang, Z., Liu, P., Zhang, Y., & Zhang, Z. (2024). RAGChecker: A fine-grained framework for diagnosing retrieval-augmented generation. Advances in Neural Information Processing Systems. https://arxiv.org/abs/2408.08067
55)Yu, H., Gan, A., Zhang, K., Tong, S., Liu, Q., & Liu, Z. (2024). Evaluation of retrieval-augmented generation: A survey [Preprint]. arXiv. https://arxiv.org/abs/2405.07437
56)Salemi, A., & Zamani, H. (2024). Evaluating retrieval quality in retrieval-augmented generation [Preprint]. arXiv. https://arxiv.org/abs/2404.13781
57)Thakur, N., Bonifacio, L., Zhang, X., Ogundepo, O., Kamalloo, E., Alfonso-Hermelo, D., Li, X., Liu, Q., Chen, B., Rezagholizadeh, M., & Lin, J. (2024). “Knowing when you don’t know”: A multilingual relevance assessment dataset for robust retrieval-augmented generation. Findings of the Association for Computational Linguistics: EMNLP 2024. https://aclanthology.org/2024.findings-emnlp.730/
58)Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024). Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics. https://aclanthology.org/2024.tacl-1.9/
59)Li, J., Xu, C., Jia, L., Wang, F., Zhang, C., & Liu, J. (2024). EACO-RAG: Towards distributed tiered LLM deployment using edge-assisted and collaborative RAG with adaptive knowledge update [Preprint]. arXiv. https://arxiv.org/abs/2410.20299
60)Astrino, P. (2026). CUBO: Self-contained retrieval-augmented generation on consumer laptops 10 GB corpora, 16 GB RAM, single-device deployment [Preprint]. arXiv. https://arxiv.org/abs/2602.03731
61)Steffens, J. (2026). Vstash: Local-first hybrid retrieval with adaptive fusion for LLM agents [Preprint]. arXiv. https://arxiv.org/abs/2604.15484