[BUG] `rclone` token rotation/refresh seems to have issues

[theFra985]

Prerequisites

• I have read the troubleshooting guide

Rclone Pre-flight Checklist (if applicable)

• This issue is NOT related to rclone (skip if not using rclone)
• I have tested rclone listremotes and rclone lsd remote: on the host and they work
• I have verified the rclone config is mounted into the container
• I have restarted the container after config changes

Bug Description

When using rclone token rotation seems to not be properly applied, causing the underlying repository remote to become "disconnected".

Steps to Reproduce

1. Create a rclone remote that performs frequent token refreshes (I am currently using Jottacloud)
2. Create one repository, a couple volumes and 3/4 backup schedules (I am able to reproduce the issue consistently when they overlap, I have it happen even when not overlapping if you operate the interface while a backup is being performed)
3. Run two operations at the same time

Expected Behavior

I understand rclone is a bit finicky when it comes with token refreshes. This is technically an rclone issue, but they are clear this behavior is intended. I think there should be a solution to avoid this behavior. It could be allowing separate rclone configs or locking the remote/repository operations to avoid concurrent ones. I would expect the token refresh to be performed correctly.

Zerobyte version / commit

v0.32.1

Deployment Method

Docker

Backup/Repository Context

As mentioned the storage backend is Jottacloud with a single repository having multiple jobs

Logs / Error Messages

Not really helpful but the resulting state is:

Failed to create file system for "remotename:/": couldn't get customer info: Get "https://api.jottacloud.com/account/v1/customer": couldn't fetch token: invalid_grant: maybe token expired? - try refreshing with "rclone config reconnect remotename:" {"message_type":"exit_error","code":1,"message":"Fatal: unable to open repository at rclone:remotename:/: error talking HTTP to rclone: exit status 1"} 

[nicotsx]

Hello @theFra985 the rclone remotes are managed outside of Zerobyte, on your host system. Not sure what we could do here if the remote is requiring you to re-authenticate. Could you point me to some docs where they explain this behavior?

[theFra985]

Hi,
Yes. In the rclone Jottacloud documentation page (I only now notice is actually the only rclone remote having this behavior, I was under the assumption this was more common) the behavior is briefly described.
Unlike other remotes, where config updates are performed to update the tokens but the required refresh tokens are long living (and don't get always consumed in the refresh process) with Jotta you need to refresh the token periodically and the previous token gets invalidated.
This will cause issues when using multiple long-running instances of rclone (or one long-running and other brief processes using the same remote) because if any of the parallel processes "decides" to refresh the token it will make all the other processes fail.
It seems like rclone is pretty aggressive in refreshing the token at the start of any operation. This means that if a second process is started in parallel to the first after the first has been running for a while (which can happen with backups) it will refresh the token, causing the backup to fail.
Just to be clear the rclone config is being used exclusively by zerobyte.
Given only Jottacloud manifests this behavior I understand if any effort or change related to working around it is felt as wasted. Perhaps it could be an option to simply document this incompatibility?
A solution specific for my use-case would be to have a shared rclone serve restic instance to handle all zerobyte operations (if I understand correctly the REST upstream is intended to be pointed at an API like that). Would that be a supported configuration for zerobyte?
Thank you for your reply

[theFra985]

Would it be ok to create a PR with an additional "limit to single instance" repository advanced flag (or something like that) which would handle this kind of setup (available only when selecting an rclone upstream)? Or would it be considered out of scope for the project?

[nicotsx]

I understand the problem better thank you. So you essentially want to eliminate all kind of parallelism on this specific repo. This means cleanup and reading as well? (Eg showing cached data only while a backup is in progress)

Would it be ok to create a PR with an additional "limit to single instance" repository advanced flag (or something like that) which would handle this kind of setup (available only when selecting an rclone upstream)? Or would it be considered out of scope for the project?

Why not, we would have to be careful implementing this and not introduce this business logic where it doesn't belong. My first instinct would be to look into an "exclusive writes" option (or exclusive locks only if parallel reads are also a problem) as this would solve other issues of users that want sequential backups.

If you want to create a PR and you have the time to investigate such options a PR is welcome! Read CONTRIBUTING.md and as long as you don't prompt an AI to do it for you (which I can already do on my side) we can have it merged. Otherwise I'll look into it when I have some time

[theFra985]

No problem. I will look into it in the next few days. No issue about AI stuff, I usually just use it to get me annoyed about how bad it implements stuff to then re-make them myself when I don't feel like starting on a task.

[theFra985]

Looking at the current implementation and given the additional argument of allowing sequential backups would it be a good approach to insert a "restic service" wrapping the functions offered by app/server/core/restic.ts? I know this would be an additional wrapper surface but I see no existing layer standing between the action implementations (request handlers, scheduler executor and similar) and the actual restic call.
This service would allow you to specify the action scheduling behavior per-repository. This would entail having an advanced option like "max concurrent operations" and "max concurrent backups" that, when set, will cause this new service to delay the execution or error out.
My idea would be to error out if the requested action is "interactive" (e.g. the user starts a backup using the /run endpoint (maybe adding a popup asking the user if they want to queue it?), the user runs doctor or a snapshot files preview is loaded) while queueing the operation if it's not (backup starting from a schedule). The queue would then be shown in a dedicated view, allowing the user to cancel queued jobs (rejecting the waiting promise) or just look a the running ones.
This page would also show all the currently running jobs grouped by repository for convenience (even for repositories without the limits specified).
Sorry if the explanation is a bit rambly.

[nicotsx]

Thank you for taking the time to analyze it, it seems reasonable to me. I'm still wondering if this should be part of the mutex or not (current locking mechanism). You are probably correct that a new layer might be necessary here since it's a different concern.

I'm in the middle of a refactor about splitting the core backup logic to a separate agent layer and unlock the feature for multiple workers connecting to one Zerobyte instance. I think we should probably wait until I ship this before working on your item. I should be done by end of next week.

Just to give a bit more context on my previous message (since it looks like you know what you are talking about) I'm not anti AI at all, I was just being a bit conservative. Some people are legitimately trying to help but unfortunately it often makes the AI PR author lose their time and my time aswell

[theFra985]

Ok, I will wait for the refactoring to be pushed to come back and tackle the issue. I understand the viewpoint about AI, I also consider it for what it is, a tool. But I know a lot of people see it as something more and tend to abuse it and waste maintainers time on broken or half-baked features so I wanted to be clear about my stance. 👍