Currently, the HTML is retrieved "on demand" when calling a user created route.
This is bad for several reasons:
- heavy load on the server
- long wait times
- no response if the website is unavailable for any given time
To fix this, we should implement 2 different caches:
Source retrieval caching
The HTML source should be cached, and checked if the html source has changed since the last request
`
$headers = get_headers( $remotePath , 1 );
$remote_mod_date = strtotime( $headers['Last-Modified'] );
$local_mod_date = filemtime( $localPath );
if ( $local_mod_date >= $remote_mod_date ) {
// Local version up to date
} else {
// retrieve new source
}
`
Where to cache the source should be configurable.
Caching parsed data
The actual parsed data should be cacheable as well, with a TTL configurable by the user.
If the parsed data is in the cache, and still valid, theres no need to retrieve the HTML source as well, so we can just serve the data from the cache.
Prewarming the cache
Optionally we should have a script that can prewarm the cache. Its up to the host of the project if he wants to run this script (in a cronjob for instance).
Currently, the HTML is retrieved "on demand" when calling a user created route.
This is bad for several reasons:
To fix this, we should implement 2 different caches:
Source retrieval caching
The HTML source should be cached, and checked if the html source has changed since the last request
`
$headers = get_headers( $remotePath , 1 );
$remote_mod_date = strtotime( $headers['Last-Modified'] );
$local_mod_date = filemtime( $localPath );
}
`
Where to cache the source should be configurable.
Caching parsed data
The actual parsed data should be cacheable as well, with a TTL configurable by the user.
If the parsed data is in the cache, and still valid, theres no need to retrieve the HTML source as well, so we can just serve the data from the cache.
Prewarming the cache
Optionally we should have a script that can prewarm the cache. Its up to the host of the project if he wants to run this script (in a cronjob for instance).