[Feature]: Database Backup & Export Pipeline (Async Workers & Streams)

[yash-pouranik]

Description

Developers using urBackend currently have no way to download or backup their project's data. If they want to migrate, analyze their data locally, or create a snapshot, they are stuck. We need an asynchronous pipeline to export collection data to JSON or CSV formats.

Since collections can grow large, doing this synchronously via an HTTP request will cause timeouts and memory crashes. We need a background job system using BullMQ and Mongoose Streams to handle this safely.

Goals

• Add a new endpoint POST /api/projects/:projectId/export in the dashboard API to trigger the export job.
• Implement a background worker using BullMQ that connects to the project's MongoDB instance.
• Safely stream the data out using Model.find().cursor() to avoid running out of memory.
• Convert the stream to JSON/CSV and upload it to the project's Supabase Storage bucket.
• Send an email to the project owner (using the existing Mail API/emailService) with a secure, temporary download link once the export is complete.

Implementation Steps

1. Queue Setup: Create a new dataExportQueue and worker in packages/common/src/queues.
2. Dashboard API: Create the trigger endpoint in apps/dashboard-api and ensure only the project owner can trigger it.
3. Worker Logic:
   • Use cursor() on the Mongoose model to stream documents.
   • Transform and pipe the data to a temporary file or directly to the Storage bucket (using apps/public-api/src/controllers/storage.controller.js logic as a reference).
   • Generate a presigned download URL.
4. Email Notification: Dispatch an email to the user with the download link upon job success, or a failure notification if it errors out.

Skills Required

• Node.js & Express
• Mongoose (Streams and Cursors)
• BullMQ (Background Jobs)
• Supabase Storage

Acceptance Criteria

• Endpoint successfully enqueues an export job.
• Worker successfully streams data without crashing on collections > 100k documents.
• Exported file is accessible via a secure link sent to the project owner's email.

[renganathc]

Hi @yash-pouranik, can I be assigned to this issue?

I’ve already explored the relevant parts of the codebase (queues, storage controller, and email flow) and understand the expected architecture for handling large exports safely using BullMQ + Mongoose cursors.

If there are no objections, I can start implementing this.

[yash-pouranik]

Hello @renganathc
Look When I look into this issue, I thought that
right now in urabckend we have 2 backend servers, first one servers all the requests for Frontend + mail worker as well second one server all the public api requests that uses xapi in header basically api keys per project + Mail queue.
So both the servers have their own purpose, I was thinking that We should have 1 another node server in ./apps/Worker or Consumer,
Who works for all the queues urbackend have ?
So basically a new node server for consuming jobs, so that each server will do their own jobs.
So First we should do is

• Creating a new node server in /apps/Consumer
• then Solve this issue [Feature]: Database Backup & Export Pipeline (Async Workers & Streams) #176 in that server only.

create a PR.
and done

Then after merging I will start work on taking all the jobs from public apis and dashboard api server to new Consumer server.

How this sounds??

I hope you got what you have to do?

Please Reply what u understand and also let me know if there's any Qn.

Also If u have a good plan suggest here or
If you are ok to work on this and going to do ? then please share a detailed aproach.

Thank you.

[renganathc]

I understood your intent @yash-pouranik.
Currently, we have the dashboard and public apis running on 2 different servers with their respective queues also running there. The new architecture would move all these jobs to consumer server. And yeah, its a great idea considering it would significantly reduce the overhead on the other servers, and the consumer server could get the jobs done in the background without any hurry, so it makes sense to move the jobs to a separate one. (Especially with the DB export feature which could be very heavy, queuing it on a separate server is a smart move)

This is the workflow I have in mind:

1. I'll create the consumer server under /apps where I'll setup the worker.
2. Then, I'll define the queue under the existing package/common/src/queues folder.
3. After that, I can add a new export endpoint in dashboard-api. This can allow us to add export jobs to the queue.
4. I'll define the BullMQ worker in the new server.

If I have any clarifications through this, I'll confirm with you before proceeding further.

[renganathc]

Also, as for the export plan itself, I can use mongoose cursor to create a stream and keep pushing the stream into a json file created locally. I'll then upload it to supabase with the existing uploadToStorage logic. Then I'll generate a url and use the email logic to mail the user.

[yash-pouranik]

@coderabbitai
I think this feature should be in pro only?

[yash-pouranik]

@renganathc
please wait I have some thing to share.

[coderabbitai]

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Search for existing plan/tier/subscription/pro gating logic
rg -rn --type=ts -i "isPro\|pro.*plan\|plan.*pro\|subscription\|tier\|featureFlag\|feature_flag" --include="*.ts" -l | head -20

echo "---"
# Search for any plan/tier checks in middleware or guards
rg -rn -i "pro\|tier\|plan\|subscription" --include="*.ts" -C2 | grep -i "guard\|middleware\|check\|limit\|restrict" | head -30

echo "---"
# Look for any existing export-related code
rg -rn -i "export\|exportDb\|dbExport\|download" --include="*.ts" -l | head -20

echo "---"
# Check if there's a pricing or plan model
fd -t f -e ts -e js -i "plan\|pricing\|subscription\|tier" | head -20

Length of output: 1338

---

🏁 Script executed:

#!/bin/bash
# Search for existing plan/tier/subscription/pro gating logic
rg -rn --type=ts -i "isPro|pro.*plan|plan.*pro|subscription|tier|featureFlag" -l | head -20

echo "---"
# Search for any plan/tier checks
rg -rn --type=ts -i "plan|subscription|tier" -l | head -20

echo "---"
# Look for any existing export-related code
rg -rn --type=ts -i "exportDb|dbExport|export.*collection|collection.*export" -l | head -20

echo "---"
# Check if there's a pricing or plan model
fd -t f "plan|pricing|subscription|tier" | head -20

echo "---"
# Look at overall app structure
tree -L 3 apps/ 2>/dev/null || ls -la apps/ 2>/dev/null

Length of output: 1204

---

🏁 Script executed:

#!/bin/bash
cat apps/dashboard-api/src/middlewares/planEnforcement.js
echo "==="
cat packages/common/src/utils/planLimits.js
echo "==="
# Also check if there's a pricing page to understand what's pro vs free
cat apps/web-dashboard/src/pages/Pricing.jsx | head -100

Length of output: 14660

---

@yash-pouranik — Yes, that makes a lot of sense. DB export is a resource-intensive operation (streaming large collections, background BullMQ job, Supabase upload, email delivery), and it fits cleanly alongside other heavy Pro features like BYOM, BYOS, Analytics Pro, and Webhooks.

The existing infrastructure already makes this straightforward to implement. Here's the pattern to follow:

1. Add a flag in packages/common/src/utils/planLimits.js:

free: {
  // ... existing flags
  dbExportEnabled: false,  // add this
},

pro: {
  // ... existing flags
  dbExportEnabled: true,   // add this
}

2. Add a gate middleware in apps/dashboard-api/src/middlewares/planEnforcement.js:

exports.checkDbExportGate = async function(req, res, next) {
    const { resolveEffectivePlan, getPlanLimits, AppError } = require('`@urbackend/common`');
    try {
        if (req.user?.isAdmin || req.user?.email?.toLowerCase() === process.env.ADMIN_EMAIL?.toLowerCase()) return next();

        const effectivePlan = resolveEffectivePlan(req.developer);
        const limits = getPlanLimits({ plan: effectivePlan });

        if (!limits.dbExportEnabled) {
            return next(new AppError(403, 'DB Export is a Pro feature. Please upgrade to export your data.'));
        }
        next();
    } catch (err) {
        next(err);
    }
}

3. Apply it on the export route in dashboard-api:

router.post(
  '/export',
  attachDeveloper,
  checkDbExportGate,
  exportController.requestExport
);

@renganathc — please include this gating as part of your implementation. The checkDbExportGate middleware should be applied to the export endpoint before the job is queued.

[renganathc]

Sure, I'll make sure to add the pro gate middleware before the actual handler 👍