Skip to content

code cleanup and openapi based route creation #14

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bugmaschine merged 18 commits into from
Jun 1, 2025
Merged

code cleanup and openapi based route creation #14

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
4728c41
small changes & cleanup
bugmaschine May 28, 2025
39e745e
use openapi file to parse, for easy route imports
bugmaschine May 28, 2025
8df6259
fix gin crashing due to the format
bugmaschine May 28, 2025
bea14f1
Add openapi updater
bugmaschine May 28, 2025
f28dd38
Update readme and add development.md
bugmaschine May 28, 2025
592f9cf
improved the openapi parsing, as posts.json was broken
bugmaschine May 28, 2025
7582572
update typos
bugmaschine May 28, 2025
efa28e8
update concurency on s3
bugmaschine May 28, 2025
d83f959
made loading additional openapi definitions easier in the future
bugmaschine May 28, 2025
046747a
reword the readme slightly
bugmaschine May 29, 2025
731a063
Add copyright notice from e621 OpenAPI specification
bugmaschine May 29, 2025
cafeabc
small comment update for clarity
bugmaschine May 29, 2025
6eb0d90
base64 is not encryption
bugmaschine Jun 1, 2025
d2c7bfe
add go test on push
bugmaschine Jun 1, 2025
7905d60
try fixing go test
bugmaschine Jun 1, 2025
ef8b4dc
trying to fix the testing again
bugmaschine Jun 1, 2025
f091c3f
fix formatting
bugmaschine Jun 1, 2025
6c5ed01
Potential fix for code scanning alert no. 3: Workflow does not contai…
bugmaschine Jun 1, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions .github/workflows/go-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
name: Upload Go test results
# https://docs.github.com/en/actions/use-cases-and-examples/building-and-testing/building-and-testing-go
on: [push]
permissions:
contents: read

jobs:
build:

runs-on: ubuntu-latest
strategy:
matrix:
go-version: [ '1.24.3' ]

steps:
- uses: actions/checkout@v4
- name: Setup Go
uses: actions/setup-go@v5
with:
go-version: ${{ matrix.go-version }}
- name: Install dependencies
run: go get .
working-directory: ./src
- name: Test with Go
run: go test -json > TestResults-${{ matrix.go-version }}.json
working-directory: ./src
- name: Upload Go test results
uses: actions/upload-artifact@v4
with:
name: Go-results-${{ matrix.go-version }}
path: src/TestResults-${{ matrix.go-version }}.json
Comment thread Fixed
31 changes: 12 additions & 19 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# e6-cache

**e6-cache** is a locally hosted caching proxy for e621 (or anything that's api compatible), designed to passively archive and cache content you browse.
**e6-cache** is a locally hosted caching proxy for e621 (or anything that's api compatible), designed to passively archive and cache content as you browse.

## Why?

Expand All @@ -9,7 +9,7 @@ Because:
* You don’t remember that *one* post with the perfect lighting and suspiciously specific tags
* **You want your own archive. Your own CDN. Your own sin-server.**
* You have too much storage space
* Your Internet connection is too slow (the only legitimate use case)
* Your Internet connection is too slow
* You worry about e621 going down, or posts being removed.

## What it does
Expand All @@ -24,7 +24,7 @@ Because:
* **Transparent Proxy**: Redirects all requests through e6-cache, then to your chosen instance.
* **Passive Caching**: Automatically caches every post you view.
* **Local Storage**: Stores metadata in a local PostgreSQL database and media files in your own S3-compatible bucket.
* **Customizable**: Easily configurable to work with any e621-compatible instance.
* **Fast**: Streams the images / videos directly to your client.
* **Self-Hosted**: Runs on your own server, giving you full control over your data.
* **Authentication**: Supports authentication for secure access, even when exposed to the world.

Expand Down Expand Up @@ -64,20 +64,20 @@ After the container is running, you can access the API at `http://localhost:8080

For most users i recommend using [e1547](https://github.com/clragon/e1547) as it has built-in support for custom instances.

### e1547 Setup
### e1547
> [!IMPORTANT]
> You need to have an public URL for the API to work with e1547, as it requires https.

https://github.com/user-attachments/assets/d2304e64-0c08-4065-bd55-aaa24d13727e

### The Wolf's Stash Setup
### The Wolf's Stash
For The Wolf's Stash, it just reports the host not being supported.

### Other Clients
Feel free to open a PR to add documentation for other clients.


## Speed Comparison (Speed depends on your internet, and database speed)
## Speed Comparison

### Image 1:
- **Normal e621 image load:** 2.859s
Expand All @@ -94,25 +94,18 @@ Feel free to open a PR to add documentation for other clients.

* Proxy Mode (it act's like a proxy and redirects all e621 requests to e6-cache)
* Firefox Extension (to make it easier to use by replacing all e621 links with e6-cache links)
* Offline Mode (to use the cache without internet connection, like a local e621)
* Offline API Mode (to use the cache without internet connection, like a local e621)
* Website (basically a mirror of e621, but with the cache enabled)

## Architecture

* Every API request you make goes through `e6-cache`
* We pull the data from the API and store it in our database. And then we serve it to you
* If there are images, we create special proxy links
* If the user or a client accesses the proxy link, we check if the image is already cached, and if not, we download it and store it in the S3 bucket

## Ethical & Legal Notice

This project **does not** scrape, spider, or hammer the API. It only caches what *you* manually request.
You, as the operator, are fully responsible for your usage. This just hands you a shovel. What you dig up is on you.

## Contributing

Feel free to open an issue or a pull request. I don't have any specific guidelines for contributing, just be nice and respectful.

## Why?

Idk. I just wanted to play with Go, Docker and wanted to make something (relatively) useful. It's also my first published project on GitHub.

## Ethical & Legal Notice

This project **does not** scrape, spider, or hammer the API. It only caches what *you* manually request.
You, as the operator, are fully responsible for your usage. This just hands you a shovel. What you dig up is on you.
5 changes: 5 additions & 0 deletions THIRD_PARTY.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# Third Party Libraries / Assets

This project wouldn't be possible without the following third party libraries and assets:

- e621 OpenAPI Specification from [https://github.com/DonovanDMC/E621OpenAPI](https://github.com/DonovanDMC/E621OpenAPI) (License: MIT)
23 changes: 23 additions & 0 deletions development.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@

## Development
`e6-cache` works by forwarding the users request, which means that it should never require any client changes, and should work on anything that abides by the offical api.

## Request Forwarding Process
The core concept of the implementation looks like this:

1. Receive an api request
2. Forward the request to the target e6-based service (While checking for the Proxy Auth for example)
3. Capture the response
4. Modify the response and save it in the DB (URIs dont change in the DB)
5. Return the modified response to the client

## File Proxying Process
File Proxying works like this:

1. Check the Signature and decode the base64 encoded url
2. Check in S3 if the file exists
3. If not, then request it and save it while forwarding it to the client. If it exist than stream it to the client from S3.

## OpenAPI Updates
The `update_openapi.sh` script:
- Updates the openai.yaml file from another repo
2 changes: 1 addition & 1 deletion docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ services:
# Proxy settings
PROXY_URL: http://localhost:8080 # Set this to the Server IP / URL, as otherwise the proxy will not work.
E6_BASE: https://e621.net
PROXY_AUTH: "" # Leave empty to disable proxy auth append like this to your username "Username:YourPassword"
PROXY_AUTH: "" # Leave empty to disable proxy auth. If you want to use it, append like this to your username "Username:YourProxyPassword"
ports:
- "8080:8080" # Point this to an Reverse Proxy and set the Proxy Url acordingly.

Expand Down
2 changes: 1 addition & 1 deletion src/.env.example
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,4 +17,4 @@ S3_REGION=us-east-1
# Proxy settings
PROXY_URL=http://localhost:8080
E6_BASE=https://e621.net
PROXY_AUTH="" # Leave empty to disable proxy auth append like this to your username "Username:YourPassword"
PROXY_AUTH="" # Leave empty to disable proxy auth. If you want to use it, append like this to your username "Username:YourProxyPassword"
21 changes: 10 additions & 11 deletions src/db.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@ import (
"context"
"database/sql"
"fmt"
"log"
"strings"
"time"

Expand Down Expand Up @@ -80,23 +79,23 @@ func (d *DB) CreatePost(ctx context.Context, p *Post) error {
)

if err != nil {
logging.Error("Error inserting post: ", err)
logging.Error("Error inserting post: %v", err)
}
return err
}

func (d *DB) CheckAndInsertPost(ctx context.Context, p *Post) error {
logging.Info("Checking if post exists: ", p.ID)
logging.Info("Checking if post exists: %v", p.ID)
const existsQuery = `SELECT 1 FROM posts WHERE id = $1`
row := d.db.QueryRowContext(ctx, existsQuery, p.ID)
var dummy int
err := row.Scan(&dummy)
switch {
case err == sql.ErrNoRows:
logging.Info("Post does not exist, inserting: ", p.ID)
logging.Error("Post does not exist, inserting: %v", p.ID)
return d.CreatePost(ctx, p)
case err != nil:
log.Println("Error checking post existence: ", err)
logging.Error("Error checking post existence: %v", err)
return err
default:
// Row exists, nothing to do
Expand Down Expand Up @@ -143,7 +142,7 @@ func (d *DB) SaveComments(comments []Comment) error {
c.UpdaterName,
)
if err != nil {
logging.Error(err.Error())
logging.Error("%v", err.Error())
return err
}
}
Expand Down Expand Up @@ -227,7 +226,7 @@ func (d *DB) UpdatePost(ctx context.Context, p *Post) error {
p.ApproverID, p.UploaderID, p.Description, p.CommentCount, p.IsFavorited,
)
if err != nil {
logging.Error("Error updating post: ", err)
logging.Error("Error updating post: %v", err)
}
return err
}
Expand All @@ -236,7 +235,7 @@ func (d *DB) UpdatePost(ctx context.Context, p *Post) error {
func (d *DB) DeletePost(ctx context.Context, id int64) error {
_, err := d.db.ExecContext(ctx, `DELETE FROM posts WHERE id = $1`, id)
if err != nil {
logging.Error("Error deleting post: ", err)
logging.Error("Error deleting post: %v", err)
}
return err
}
Expand Down Expand Up @@ -271,20 +270,20 @@ func (d *DB) UpdatePool(ctx context.Context, p *Pool) error {
p.Description, p.IsActive, p.Category, p.PostCount,
)
if err != nil {
logging.Error("error upserting pool: ", err)
logging.Error("error upserting pool: %v", err)
return err
}

_, err = tx.ExecContext(ctx, `DELETE FROM pool_posts WHERE pool_id = $1`, p.ID)
if err != nil {
logging.Error("error clearing pool_posts: ", err)
logging.Error("error clearing pool_posts: %v", err)
return err
}

for _, postID := range p.PostIDs {
_, err = tx.ExecContext(ctx, `INSERT INTO pool_posts (pool_id, post_id) VALUES ($1, $2)`, p.ID, postID)
if err != nil {
logging.Error("error inserting pool_post: ", err)
logging.Error("error inserting pool_post: %v", err)
return err
}
}
Expand Down
9 changes: 9 additions & 0 deletions src/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ require (
github.com/aws/aws-sdk-go-v2/credentials v1.17.67
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.17.76
github.com/aws/aws-sdk-go-v2/service/s3 v1.79.4
github.com/getkin/kin-openapi v0.132.0
github.com/gin-gonic/gin v1.10.1
github.com/joho/godotenv v1.5.1
github.com/lib/pq v1.10.9
Expand All @@ -35,17 +36,25 @@ require (
github.com/cloudwego/iasm v0.2.0 // indirect
github.com/gabriel-vasile/mimetype v1.4.3 // indirect
github.com/gin-contrib/sse v0.1.0 // indirect
github.com/go-openapi/jsonpointer v0.21.0 // indirect
github.com/go-openapi/swag v0.23.0 // indirect
github.com/go-playground/locales v0.14.1 // indirect
github.com/go-playground/universal-translator v0.18.1 // indirect
github.com/go-playground/validator/v10 v10.20.0 // indirect
github.com/goccy/go-json v0.10.2 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/klauspost/cpuid/v2 v2.2.7 // indirect
github.com/leodido/go-urn v1.4.0 // indirect
github.com/mailru/easyjson v0.7.7 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826 // indirect
github.com/oasdiff/yaml v0.0.0-20250309154309-f31be36b4037 // indirect
github.com/oasdiff/yaml3 v0.0.0-20250309153720-d2182401db90 // indirect
github.com/pelletier/go-toml/v2 v2.2.2 // indirect
github.com/perimeterx/marshmallow v1.1.5 // indirect
github.com/twitchyliquid64/golang-asm v0.15.1 // indirect
github.com/ugorji/go/codec v1.2.12 // indirect
golang.org/x/arch v0.8.0 // indirect
Expand Down
18 changes: 18 additions & 0 deletions src/go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,10 +51,16 @@ github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/gabriel-vasile/mimetype v1.4.3 h1:in2uUcidCuFcDKtdcBxlR0rJ1+fsokWf+uqxgUFjbI0=
github.com/gabriel-vasile/mimetype v1.4.3/go.mod h1:d8uq/6HKRL6CGdk+aubisF/M5GcPfT7nKyLpA0lbSSk=
github.com/getkin/kin-openapi v0.132.0 h1:3ISeLMsQzcb5v26yeJrBcdTCEQTag36ZjaGk7MIRUwk=
github.com/getkin/kin-openapi v0.132.0/go.mod h1:3OlG51PCYNsPByuiMB0t4fjnNlIDnaEDsjiKUV8nL58=
github.com/gin-contrib/sse v0.1.0 h1:Y/yl/+YNO8GZSjAhjMsSuLt29uWRFHdHYUb5lYOV9qE=
github.com/gin-contrib/sse v0.1.0/go.mod h1:RHrZQHXnP2xjPF+u1gW/2HnVO7nvIa9PG3Gm+fLHvGI=
github.com/gin-gonic/gin v1.10.1 h1:T0ujvqyCSqRopADpgPgiTT63DUQVSfojyME59Ei63pQ=
github.com/gin-gonic/gin v1.10.1/go.mod h1:4PMNQiOhvDRa013RKVbsiNwoyezlm2rm0uX/T7kzp5Y=
github.com/go-openapi/jsonpointer v0.21.0 h1:YgdVicSA9vH5RiHs9TZW5oyafXZFc6+2Vc1rr/O9oNQ=
github.com/go-openapi/jsonpointer v0.21.0/go.mod h1:IUyH9l/+uyhIYQ/PXVA41Rexl+kOkAPDdXEYns6fzUY=
github.com/go-openapi/swag v0.23.0 h1:vsEVJDUo2hPJ2tu0/Xc+4noaxyEffXNIs3cOULZ+GrE=
github.com/go-openapi/swag v0.23.0/go.mod h1:esZ8ITTYEsH1V2trKHjAN8Ai7xHb8RV+YSZ577vPjgQ=
github.com/go-playground/assert/v2 v2.2.0 h1:JvknZsQTYeFEAhQwI4qEt9cyV5ONwRHC+lYKSsYSR8s=
github.com/go-playground/assert/v2 v2.2.0/go.mod h1:VDjEfimB/XKnb+ZQfWdccd7VUvScMdVu0Titje2rxJ4=
github.com/go-playground/locales v0.14.1 h1:EWaQ/wswjilfKLTECiXz7Rh+3BjFhfDFKv/oXslEjJA=
Expand All @@ -70,6 +76,8 @@ github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/joho/godotenv v1.5.1 h1:7eLL/+HRGLY0ldzfGMeQkb7vMd0as4CfYvUVzLqw0N0=
github.com/joho/godotenv v1.5.1/go.mod h1:f4LDr5Voq0i2e/R5DDNOoa2zzDfwtkZa6DnEwAbqwq4=
github.com/josharian/intern v1.0.0 h1:vlS4z54oSdjm0bgjRigI+G1HpF+tI+9rE5LLzOg8HmY=
github.com/josharian/intern v1.0.0/go.mod h1:5DoeVV0s6jJacbCEi61lwdGj/aVlrQvzHFFd8Hwg//Y=
github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM=
github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
github.com/klauspost/cpuid/v2 v2.0.9/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
Expand All @@ -80,15 +88,25 @@ github.com/leodido/go-urn v1.4.0 h1:WT9HwE9SGECu3lg4d/dIA+jxlljEa1/ffXKmRjqdmIQ=
github.com/leodido/go-urn v1.4.0/go.mod h1:bvxc+MVxLKB4z00jd1z+Dvzr47oO32F/QSNjSBOlFxI=
github.com/lib/pq v1.10.9 h1:YXG7RB+JIjhP29X+OtkiDnYaXQwpS4JEWq7dtCCRUEw=
github.com/lib/pq v1.10.9/go.mod h1:AlVN5x4E4T544tWzH6hKfbfQvm3HdbOxrmggDNAPY9o=
github.com/mailru/easyjson v0.7.7 h1:UGYAvKxe3sBsEDzO8ZeWOSlIQfWFlxbzLZe7hwFURr0=
github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc=
github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M=
github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=
github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826 h1:RWengNIwukTxcDr9M+97sNutRR1RKhG96O6jWumTTnw=
github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826/go.mod h1:TaXosZuwdSHYgviHp1DAtfrULt5eUgsSMsZf+YrPgl8=
github.com/oasdiff/yaml v0.0.0-20250309154309-f31be36b4037 h1:G7ERwszslrBzRxj//JalHPu/3yz+De2J+4aLtSRlHiY=
github.com/oasdiff/yaml v0.0.0-20250309154309-f31be36b4037/go.mod h1:2bpvgLBZEtENV5scfDFEtB/5+1M4hkQhDQrccEJ/qGw=
github.com/oasdiff/yaml3 v0.0.0-20250309153720-d2182401db90 h1:bQx3WeLcUWy+RletIKwUIt4x3t8n2SxavmoclizMb8c=
github.com/oasdiff/yaml3 v0.0.0-20250309153720-d2182401db90/go.mod h1:y5+oSEHCPT/DGrS++Wc/479ERge0zTFxaF8PbGKcg2o=
github.com/pelletier/go-toml/v2 v2.2.2 h1:aYUidT7k73Pcl9nb2gScu7NSrKCSHIDE89b3+6Wq+LM=
github.com/pelletier/go-toml/v2 v2.2.2/go.mod h1:1t835xjRzz80PqgE6HHgN2JOsmgYu/h4qDAS4n929Rs=
github.com/perimeterx/marshmallow v1.1.5 h1:a2LALqQ1BlHM8PZblsDdidgv1mWi1DgC2UmX50IvK2s=
github.com/perimeterx/marshmallow v1.1.5/go.mod h1:dsXbUu8CRzfYP5a87xpp0xq9S3u0Vchtcl8we9tYaXw=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
Expand Down
Loading
Loading