PHP memory exhaustion when reading large log files via LogFileModelTraits

[novastate]

Bug: PHP memory exhaustion when reading large log files via LogFileModelTraits

Description

The REST API crashes with a PHP fatal error when attempting to read large log files. The read_uncompressed_log() method in LogFileModelTraits.inc uses PHP's file() function which loads the entire file into memory, causing memory exhaustion on large rotated log files.

Environment

• pfSense Version: 2.8.1-RELEASE (amd64)
• REST API Version: pfSense-pkg-RESTAPI-2.6_8
• PHP Memory Limit: 512 MB

Error

PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted
(tried to allocate 541402264 bytes) in
/usr/local/pkg/RESTAPI/ModelTraits/LogFileModelTraits.inc on line 38

Root Cause

In LogFileModelTraits.inc, the read_uncompressed_log() method:

private function read_uncompressed_log(string $filepath): array {
    $this->check_file_exists($filepath);
    return file($filepath, FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES);  // Line 38
}

This loads the entire file into memory as an array. When log rotation produces large files (500MB+ in my case due to misconfigured log limits), API calls that query logs will crash.

The same issue exists in read_bzip2_log() and read_gzip_log() which decompress entire files into memory.

Steps to Reproduce

1. Have a large rotated log file (e.g., /var/log/system.log.X > 500MB)
2. Make an API call that reads system logs
3. PHP crashes with memory exhaustion

Suggested Fix

Consider one of:

1. Add file size check - Reject files over a threshold with a clear error
2. Stream reading - Use fgets() in a loop with line limit
3. Generator pattern - Yield lines instead of returning full array
4. Pagination - Only read requested line ranges from file

Example size-limited approach:

private function read_uncompressed_log(string $filepath, int $max_bytes = 50000000): array {
    $this->check_file_exists($filepath);

    if (filesize($filepath) > $max_bytes) {
        throw new NotAcceptableError(
            message: "Log file too large to read. Size: " . filesize($filepath) . " bytes.",
            response_id: 'LOG_FILE_TRAITS_FILE_TOO_LARGE',
        );
    }

    return file($filepath, FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES);
}

Workaround

Reduce log file size limits in pfSense:

• Status → System Logs → Settings → Set log file size to 500KB or less
• Or add to config.xml: <logfilesize>512000</logfilesize>

[jaredhendrickson13]

This should be relatively simple to address. Currently, Models that are not based on config values are not paginated until all objects have been loaded into memory. This was intentional as most of these models simply call internal pfSense callables that always return the full set of data, and cannot be paginated beforehand anyway. For logs however, this does demonstrate a legitimate use case for passing pagination parameters all the way down to these callables. Should be able to have that included in v2.7.0.

In the meantime, my suggestions would be:

1. Reduce the log file size (like you mentioned). pfSense in general does not handle large log files well. Better to ship those logs somewhere better equipped to handle log files of that size.
2. If you cannot reduce the log file size, increase the amount of memory allocated to PHP. In the pfSense webConfigurator, go to System > Advanced > Miscellaneous > PHP Settings and increase the memory limit if possible.