Skip to content

README.md

Latest commit

 

History

History
79 lines (59 loc) · 2.48 KB

README.md

File metadata and controls

79 lines (59 loc) · 2.48 KB
Raw

remotesource

remotesource adapts network-addressable objects to file-like Go sources.

It does not parse the bytes it opens. Format-specific readers, validators, and decoders belong in their own layers; for example, whole-stream decompression can be handled by decodedsource.

Shape

The root package defines small Source and RandomAccessSource interfaces:

type Source interface {
	Open() (io.ReadCloser, error)
}

type RandomAccessSource interface {
	Source
	OpenRange(off, size int64) (io.ReadCloser, error)
	OpenAt(off int64) (io.ReadCloser, error)
	Size() (int64, error)
}

The github.com/saveweb/remotesource/rclone submodule implements those methods using rclone's Go backend API. This keeps the root module dependency-free, while covering common file-like network stores such as HTTP, WebDAV, S3, FTP, SFTP, and SMB through rclone remotes.

Usage

import "github.com/saveweb/remotesource/rclone"

source, err := rclone.New(ctx, "remote:path/to/crawl.warc.gz")
if err != nil {
	return err
}

scanner, err := unwarc.NewScannerFromSource(source, unwarc.ScannerOptions{
	Compression: unwarc.CompressionUnknown,
})

Use normal rclone remote syntax. Configured remotes such as s3:bucket/key work through the user's rclone config; backend literals can be used when supported by rclone. For example:

httpSource, err := rclone.New(ctx,
	":http,url='https://github.com/saveweb/unwarc/raw/refs/heads/main/testdata/corpus/gowarc/':gowarc-solid-gzip.warc.gz",
)
iaSource, err := rclone.New(ctx,
	":internetarchive:XLOGIPFS-crawling/ZENO-20260307125208342-00002-hz1.warc.gz",
)

Each Open, OpenRange, and OpenAt call opens an independent rclone object reader. The remote object must remain stable while collected references are being lazily reopened by consumers such as unwarc.

rclone.New uses its context only to resolve the object. Later reads use context.Background by default; use NewWithOptions to provide a longer-lived open context. The selected backend must honor rclone range reads. HTTP servers that ignore Range requests are not safe random-access sources.

Live rclone backend smokes are skipped by default; run them with REMOTESOURCE_LIVE=1 go test ./... inside rclone.

Tradeoff

Using rclone as a library gives broad backend coverage, but it is intentionally heavier than protocol-specific clients. The rclone adapter lives in its own Go module so callers can opt in to that dependency.

License

CC0-1.0