Reverse Proxy does not work

[mind-overflow]

Context

Hello,
I have tried setting up a self-hosted instance of syncing-server.
I followed your instructions, which are pretty straightforward:

• clone the repo
• run the setup script
• start the server

Since this instance is in a remote server, I have no way to test it by visiting http://localhost:3000/.
What I did instead, was to create an Apache2 reverse proxy. I tried it both with and without SSL, with no difference.
This is the setup:

<VirtualHost *:443>
        ServerName sync.example.org

        <Location />
                ProxyPass http://127.0.0.1:3000/
                ProxyPassReverse http://127.0.0.1:3000/
        </Location>

        Include /etc/letsencrypt/options-ssl-apache.conf
        SSLCertificateFile /etc/letsencrypt/live/example.org/fullchain.pem
        SSLCertificateKeyFile /etc/letsencrypt/live/example.org/privkey.pem

</VirtualHost>

The issue

However, when I try to visit sync.example.org, I get a 404 error:

Not Found

If I type ./server.sh logs, this is what pops up:

api-gateway_1               | {"level":"debug","message":"Calling legacy syncing server on: /"}
api-gateway_1               | {"message":"Calling [GET] http://syncing-server-js:3000/,\n        headers: {\"host\":\"127.0.0.1:3000\",\"cache-control\":\"max-age=0\",\"dnt\":\"1\",\"upgrade-insecure-requests\":\"1\",\"user-agent\":\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.72 Safari/537.36\",\"accept\":\"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\",\"sec-gpc\":\"1\",\"sec-fetch-site\":\"none\",\"sec-fetch-mode\":\"navigate\",\"sec-fetch-user\":\"?1\",\"sec-fetch-dest\":\"document\",\"accept-encoding\":\"gzip, deflate, br\",\"accept-language\":\"it-IT,it;q=0.9,en-US;q=0.8,en;q=0.7\",\"cookie\":\"mybbuser=1_QI0AGpsc7iCVepZ3phhwgo5MeKaEqPjQqvk1SDhOkpFHe7TwPr; sid=fc5d81d754395a54426f893616a4a6ab; PHPSESSID=pq8loagorqi77fgb58a5grpqio; rememberme=1%3Ab7f5e7e4997a79aa99eccb6a9103407923a8ec9b20b5480b497e8d9d50e8fb33%3A93dafe779d1e92d1fe736380253a40e33c0cdc1f7b6fa73e2dca1958d30f5d0f\",\"x-forwarded-for\":\"87.2.42.126\",\"x-forwarded-host\":\"sync.example.org\",\"x-forwarded-server\":\"sync.example.org\",\"connection\":\"Keep-Alive\"},\n        query: {},\n        payload: {}","level":"debug"}
syncing-server-js_1         | {"meta":{"req":{"url":"/","headers":{"host":"syncing-server-js:3000","accept-encoding":"gzip, deflate, br","cache-control":"max-age=0","dnt":"1","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.72 Safari/537.36","accept":"application/json","sec-gpc":"1","sec-fetch-site":"none","sec-fetch-mode":"navigate","sec-fetch-user":"?1","sec-fetch-dest":"document","accept-language":"it-IT,it;q=0.9,en-US;q=0.8,en;q=0.7","cookie":"mybbuser=1_QI0AGpsc7iCVepZ3phhwgo5MeKaEqPjQqvk1SDhOkpFHe7TwPr; sid=fc5d81d754395a54426f893616a4a6ab; PHPSESSID=pq8loagorqi77fgb58a5grpqio; rememberme=1%3Ab7f5e7e4997a79aa99eccb6a9103407923a8ec9b20b5480b497e8d9d50e8fb33%3A93dafe779d1e92d1fe736380253a40e33c0cdc1f7b6fa73e2dca1958d30f5d0f","x-forwarded-for":"87.2.42.126","x-forwarded-host":"sync.example.org","x-forwarded-server":"sync.example.org","connection":"Keep-Alive","content-type":"application/json","content-length":"2"},"method":"GET","httpVersion":"1.1","originalUrl":"/","query":{}},"res":{"statusCode":404},"responseTime":0},"level":"info","message":"HTTP GET /"}
api-gateway_1               | {"meta":{"req":{"url":"/","headers":{"cache-control":"max-age=0","dnt":"1","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.72 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","sec-gpc":"1","sec-fetch-site":"none","sec-fetch-mode":"navigate","sec-fetch-user":"?1","sec-fetch-dest":"document","accept-encoding":"gzip, deflate, br","accept-language":"it-IT,it;q=0.9,en-US;q=0.8,en;q=0.7","cookie":"mybbuser=1_QI0AGpsc7iCVepZ3phhwgo5MeKaEqPjQqvk1SDhOkpFHe7TwPr; sid=fc5d81d754395a54426f893616a4a6ab; PHPSESSID=pq8loagorqi77fgb58a5grpqio; rememberme=1%3Ab7f5e7e4997a79aa99eccb6a9103407923a8ec9b20b5480b497e8d9d50e8fb33%3A93dafe779d1e92d1fe736380253a40e33c0cdc1f7b6fa73e2dca1958d30f5d0f","x-forwarded-for":"87.2.42.126","x-forwarded-host":"sync.example.org","x-forwarded-server":"sync.example.org","connection":"Keep-Alive"},"method":"GET","httpVersion":"1.1","originalUrl":"/","query":{}},"res":{"statusCode":404},"responseTime":8},"level":"info","message":"HTTP GET /"}
api-gateway_1               | Error: Not Found
api-gateway_1               |     at Request.callback (/var/www/node_modules/superagent/lib/node/index.js:883:15)
api-gateway_1               |     at IncomingMessage.<anonymous> (/var/www/node_modules/superagent/lib/node/index.js:1127:20)
api-gateway_1               |     at IncomingMessage.emit (events.js:327:22)
api-gateway_1               |     at endReadableNT (_stream_readable.js:1327:12)
api-gateway_1               |     at processTicksAndRejections (internal/process/task_queues.js:80:21)

Temporary fix

I found a temporary fix, that however is not how it's supposed to work.
If I modify my Apache configuration and use the docker container's IP instead of 127.0.0.1, it works.

Find the docker container ID

docker ps -a  | grep sync

Result:

77bbca3e17a0        standardnotes/auth                 "./wait-for.sh synci…"   8 minutes ago       Up 7 minutes                                                                                                                                                           syncing-server_auth_1
3024c4a0ee11        standardnotes/syncing-server-js    "./wait-for.sh synci…"   8 minutes ago       Up 7 minutes                                                                                                                                                           syncing-server_syncing-server-js-worker_1
9fb6d0263ed8        standardnotes/api-gateway          "./wait-for.sh synci…"   8 minutes ago       Up 7 minutes              0.0.0.0:3000->3000/tcp                                                                                                                       syncing-server_api-gateway_1
090590aa093c        standardnotes/syncing-server-js    "./wait-for.sh synci…"   8 minutes ago       Up 7 minutes                                                                                                                                                           syncing-server_syncing-server-js_1
06f874d2b544        redis:6.0-alpine                   "docker-entrypoint.s…"   8 minutes ago       Up 7 minutes              0.0.0.0:32851->6379/tcp                                                                                                                      syncing-server_cache_1
cf9e18522cbe        syncing-server_syncing-server      "./wait-for.sh db 33…"   8 minutes ago       Up 7 minutes                                                                                                                                                           syncing-server_syncing-server_1
9bbba9543007        mysql:5.6                          "docker-entrypoint.s…"   8 minutes ago       Up 7 minutes              0.0.0.0:32850->3306/tcp                                                                                                                      syncing-server_db_1

Find the docker container IP

docker inspect cf9e18522cbe

Result:

<...>
"syncing_server": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "syncing-server",
                        "cf9e18522cbe"
                    ],
                    "NetworkID": "<...>",
                    "EndpointID": "<...>",
                    "Gateway": "172.30.0.1",
                    "IPAddress": "172.30.0.8",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "<...>",
                    "DriverOpts": null
                }
<...>

So, the IP address is 172.30.0.8.

Point Apache2 to the container IP

<VirtualHost *:443>
        ServerName sync.example.org

        <Location />
                ProxyPass http://172.30.0.8:3000/
                ProxyPassReverse http://172.30.0.8:3000/
        </Location>

        Include /etc/letsencrypt/options-ssl-apache.conf
        SSLCertificateFile /etc/letsencrypt/live/example.org/fullchain.pem
        SSLCertificateKeyFile /etc/letsencrypt/live/example.org/privkey.pem

</VirtualHost>

Conclusion

By applying the temporary fix, when I visit the page, I get the expected result:

{"message":"Hi! You're not supposed to be here."}

However, the fix is clearly temporary, as the IP docker assigns to its containers is not static. Besides, this is not how it's supposed to work anyway, so I wouldn't feel comfortable leaving it like that anyway.

Any suggestion on how to fix this? If this is not an issue related to syncing-server itself.
I'm always available to provide more logs and info if needed.
Thanks!

PS: .env and docker-compose.yml are left untouched.

PPS: My server is on Ubuntu Server 20.04, running Apache 2.4.41. Any firewall has been disabled for those tests.

[karolsojko]

Hmm not sure why the use of Apache in here. Maybe change the env files to run the docker setup not at port 3000 but 80 so that you can acess your remote host directly?

[mind-overflow]

@karolsojko The use of Apache is mandatory, because it is the webserver for that machine. It is handling multiple websites and subdomains, while syncing-server is running in a docker container. So, Apache is running as a reverse proxy, tunneling traffic to the docker container, and thus, running the container directly on port 80/443 is not possible as it would conflict with Apache and block access to all the other services.

In theory, as pointed out by this blog post, making a nginx reverse proxy to syncing-server is straightforward and only requires a few lines of config. This is obviously not the case with Apache, as the standard proxy configuration does not work.

Edit: here's the nginx config, to avoid opening the linked url:

server {
    listen 80;
    listen [::]:80;
    server_name notes.example.org;

    location / {
        proxy_pass http://127.0.0.1:3000; # syncing-server address
    }
}

[karolsojko]

We have a new documentation suite out. Please give a try and re-check if it helps with your issue: https://docs.standardnotes.org/self-hosting/getting-started/