in-context-learning-vulnerability prediction

Modern software systems demand earlier vulnerability severity detection to protect systems from critical issues such as data leaking or attacker access. Security analysts are in charge of triaging the vulnerability severity by computing a score following security standard methods. Vulnerability severity triaging becomes critical as the number of vulnerabilities increases. In addition, this severity triaging is also relevant to timely detecting vulnerabilities. Large Language Models (LLMs) have been demonstrated to assist security analysts in vulnerability detection and fixing. However, private models (e.g., ChatGPT, Copilot, and Claude) require sending registered code and proprietary data to third-party companies. We present an industry experience report that uses registered and open-data datasets to evaluate the effectiveness of predicting vulnerability severity. We designed an in-context learning solution to predict the severity score using open source LLms such as CodeLlama2 and Mistral. We observed CodeLlama-7B can predict the severity score with an average MSE of ~7.62 in one-shot. Our findings indicate \codellama is a promising LLM to assist security analysts in assessing the severity impact before deployment while protecting data exposure.