Citation :

Shruti Hardia, 2026. "A Scalable System for Software Repository Analysis and Retrieval" ESP International Journal of Artificial Intelligence & Data Science [IJAIDS]  Volume 2, Issue 2: 95-113.

Abstract :

The rapid growth of modern software ecosystems has re-sulted in massive, globally distributed code repositories, making efficient indexing, retrieval, and structural analysis increasingly challenging. Ex-isting repository mining tools often struggle with scalability, lack deep structural correlation, or are limited to single-language analysis. This work introduces Search SECO, a distributed and language-independent framework for large-scale software repository mining. The system adopts a modular design consisting of a high-throughput crawler for metadata collection, a parallel retriever for repository acquisition, and hybrid parsers that combine srcML and custom ANTLR grammars for method level extraction. A distributed Apache Cassandra backend, along with an op-timized networking layer, enables low-latency client–server communica-tion. Experimental evaluation on diverse open-source datasets demon-strates near-linear scalability while processing millions of methods across thousands of repositories. By linking methods, authors, and version his-tories, Search SECO supports advanced cross-repository analyses such as vulnerability detection, clone identification, and software evolution stud-ies. Overall, the framework provides a scalable, extensible, and accurate solution adaptable to new programming languages and repository plat-forms.

References :

[1] Kim, S., Woo, S., Lee, H. and Oh, H., 2017, May. Vuddy: A scalable approach for vulnerable code clone discovery. In 2017 IEEE Symposium on Security and Privacy (SP) (pp. 595-614). IEEE.

[2] Software Heritage Graph (https://softwareheritage.org): the largest existing public archive of software source code and accompanying development history.

[3] Apache Cassandra (https://cassandra.apache.org/): a free and open-source, distributed, wide-column store, NoSQL database management system designed to handle large amounts of data across many commodity servers, providing high availability with no single point of failure. Documentation can be found at https: //cassandra.apache.org/doc/latest/

[4] srcML: https://www.srcml.org/

[5] ANTLR: https://www.antlr.org/

[6] libcurl (https://curl.se/libcurl/) is a free and easy-to-use client-side URL transfer library, supporting DICT, FILE, FTP, FTPS, GOPHER, GOPHERS, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, MQTT, POP3, POP3S, RTMP, RTMPS, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, TELNET and TFTP.

[7] Loguru: https://github.com/emilk/loguru

[8] Boost: https://www.boost.org/

[9] Svajlenko, J., Islam, J.F., Keivanloo, I., Roy, C.K. and Mia, M.M., 2014, Septem-ber. Towards a big data curated benchmark of inter-project code clones. In 2014 IEEE International Conference on Software Maintenance and Evolution (pp. 476-480). IEEE.

[10] Farhadi, M.R., Fung, B.C., Charland, P. and Debbabi, M., 2014, June. Binclone: Detecting code clones in malware. In 2014 Eighth International Conference on Software Security and Reliability (SERE) (pp. 78-87). IEEE.

[11] Gousios, G., 2013. The GHTorrent dataset and tool suite. In 2013 10th IEEE Working Conference on Mining Software Repositories (MSR) (pp. 233-236). IEEE.

[12] Dyer, R., Nguyen, H.A., Rajan, H. and Nguyen, T.N., 2013. Boa: A language and infrastructure for analyzing ultra-large-scale software repositories. In 2013 35th International Conference on Software Engineering (ICSE) (pp. 422-431). IEEE.

Keywords :

Software repository mining, distributed software systems, source code analysis, repository retrieval, Apache Cassandra, ANTLR parsing, srcML, method-level indexing, code clone detection, and software evolution analysis. These keywords represent the paper's primary focus on developing a scalable, distributed framework for mining, parsing, indexing, and analyzing large-scale software repositories using hybrid parsing techniques and distributed data management to support efficient cross-repository analysis and retrieval.