fscrawler REST API not reachable after starting with docker-compose

[koopmac]

Describe the bug

I have successfully connected FSCrawler with elasticsearch in docker-compose and indexed a pdf document (I have queried the 'non-rest' job index with a successful result). However I'm unable to reach the REST client after successfully running it. I'm trying to create a pipeline with elasticsearch to OCR pdfs upon upload but can't seem to get the REST client working properly.

Docker-compose settings

version: '3.6'

services:
  postgres:
    image: "postgres:12.1"
    env_file:
      - '.env'
    ports:
      - '127.0.0.1:5432:5432'
    restart: "${DOCKER_RESTART_POLICY:-unless-stopped}"
    stop_grace_period: "${DOCKER_STOP_GRACE_PERIOD:-3s}"
    volumes:
      - postgres:/var/lib/postgresql/data
    networks: 
      - esnet

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.8.0
    # build: ./es
    container_name: elasticsearch
    env_file:
      - ".env"
    depends_on:
      - "postgres"
    volumes:
      - esdata:/usr/share/elasticsearch/data
    environment:
      - node.name=elasticsearch
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - discovery.type=single-node
      - network.host=0.0.0.0
      - network.publish_host=0.0.0.0
      - http.cors.enabled=true
      - http.cors.allow-origin=*
      - http.host=0.0.0.0
      - transport.host=0.0.0.0
    ulimits:
      memlock:
        soft: -1
        hard: -1
    ports:
      - 9200:9200
      - 9300:9300
    networks:
      - esnet

  fscrawler:
    # I have taken this docker image and updated to 2.7 snapshot: toto1310/fscrawler
    build:
      context: ${PWD}
      dockerfile: Dockerfile-toto
    container_name: fscrawler
    depends_on:
      - elasticsearch
    restart: always
    volumes:
      - ${PWD}/config:/root/.fscrawler
      - ${PWD}/data:/tmp/es
    networks: 
      - esnet
    environment:
      - FS_URL=/tmp/es
      - ELASTICSEARCH_URL=http://elasticsearch:9200
      - ELASTICSEARCH_INDEX=fscrawler_rest
    command: fscrawler fscrawler_rest --loop 0 --rest debug

  web:
    build: 
      context: "."
      args:
        - "FLASK_ENV=${FLASK_ENV:-production}"
    depends_on:
      - "postgres"
      - "elasticsearch"
      # - fscrawler
    env_file:
      - ".env"
    ports:
      - "${DOCKER_WEB_PORT:-127.0.0.1:8000}:8000"
    restart: "${DOCKER_RESTART_POLICY:-unless-stopped}"
    stop_grace_period: "${DOCKER_STOP_GRACE_PERIOD:-3s}"
    volumes:
      # - "${DOCKER_WEB_VOLUME:-./public:/app/public}"
      - "${DOCKER_WEB_VOLUME:-.:/prosebit}"
    networks:
      - esnet

volumes:
  postgres:
  esdata:
    driver: local

networks:
  esnet:

Job Settings

name: "fscrawler_rest"
fs:
  url: "/tmp/es"
  ocr:
    language: "eng"
    enabled: true
    # path: "/path/to/tesseract/executable"
    # data_path: "/path/to/tesseract/tessdata"
elasticsearch:
  nodes:
  - url: "http://elasticsearch:9200"
  pipeline: fscrawler
rest :
  # url: "http://localhost:8080/fscrawler"
  # enable_cors: true

Link to dockerfile here (I have updated for 2.7): toto1310

Logs

fscrawler        | 07:18:14,343 INFO  [f.p.e.c.f.c.BootstrapChecks] Memory [Free/Total=Percent]: HEAP [39.8mb/697mb=5.71%], RAM [284mb/3gb=9.08%], Swap [1010.2mb/1gb=98.66%].
fscrawler        | 07:18:14,880 WARN  [f.p.e.c.f.c.FsCrawlerCli] `rest` settings are not defined. Falling back to default: [http://127.0.0.1:8080/fscrawler].
fscrawler        | 07:18:15,620 INFO  [f.p.e.c.f.c.v.ElasticsearchClientV7] Elasticsearch Client for version 7.x connected to a node running version 7.8.0
fscrawler        | 07:18:15,888 INFO  [f.p.e.c.f.FsCrawlerImpl] Starting FS crawler
fscrawler        | 07:18:17,300 WARN  [o.g.j.i.i.Providers] A provider fr.pilato.elasticsearch.crawler.fs.rest.ServerStatusApi registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider fr.pilato.elasticsearch.crawler.fs.rest.ServerStatusApi will be ignored.
fscrawler        | 07:18:17,301 WARN  [o.g.j.i.i.Providers] A provider fr.pilato.elasticsearch.crawler.fs.rest.UploadApi registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider fr.pilato.elasticsearch.crawler.fs.rest.UploadApi will be ignored.
fscrawler        | 07:18:17,838 INFO  [f.p.e.c.f.r.RestServer] FS crawler Rest service started on [http://127.0.0.1:8080/fscrawler]
fscrawler        | 07:18:38,633 INFO  [f.p.e.c.f.c.BootstrapChecks] Memory [Free/Total=Percent]: HEAP [39.6mb/697mb=5.69%], RAM [316.4mb/3gb=10.12%], Swap [1010mb/1gb=98.63%].
fscrawler        | 07:18:39,608 WARN  [f.p.e.c.f.c.FsCrawlerCli] `rest` settings are not defined. Falling back to default: [http://127.0.0.1:8080/fscrawler].
fscrawler        | 07:18:40,954 INFO  [f.p.e.c.f.c.v.ElasticsearchClientV7] Elasticsearch Client for version 7.x connected to a node running version 7.8.0
fscrawler        | 07:18:41,257 INFO  [f.p.e.c.f.FsCrawlerImpl] Starting FS crawler
fscrawler        | 07:18:43,404 WARN  [o.g.j.i.i.Providers] A provider fr.pilato.elasticsearch.crawler.fs.rest.ServerStatusApi registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider fr.pilato.elasticsearch.crawler.fs.rest.ServerStatusApi will be ignored.
fscrawler        | 07:18:43,404 WARN  [o.g.j.i.i.Providers] A provider fr.pilato.elasticsearch.crawler.fs.rest.UploadApi registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider fr.pilato.elasticsearch.crawler.fs.rest.UploadApi will be ignored.
fscrawler        | 07:18:44,293 INFO  [f.p.e.c.f.r.RestServer] FS crawler Rest service started on [http://127.0.0.1:8080/fscrawler]

curl -XGET "127.0.0.1:8080/fscrawler" or curl -XGET "127.0.0.1:8080/fscrawler/" or localhost
returns: curl: (7) Failed to connect to 127.0.0.1 port 8080: Connection refused as does the browser

Expected behavior

A GET request from 127.0.0.1:8080/fscrawler/ should return a valid json response from FSCrawler.

Versions:

• docker-compose version 1.21.2, build a133471
• file version 3.6

[dadoonet]

I wonder if you are not missing this in the FSCrawler part:

ports:
  - 8080:8080

[koopmac]

I forgot to add that I have tried that but it returns ‘ERR_EMPTY_RESPONSE’

[dadoonet]

You need to have that for sure. I guess the error message comes from the fact you are seeing some warnings when starting.

I'm wondering if this is caused by the fact it tries by default to use localhost:8080 to bind the rest service. May be you should configure it to fscrawler:8080 instead?

[koopmac]

Yep you're absolutely right. I've been stuck all day on that. Would you like me to add a section in the docs about connecting to the REST API with docker? Love the project and it seems it's the least I can do.

[dadoonet]

That'd be lovely!
May be modify a bit what has been done with #982?

[koopmac]

No worries, just updated and opened a pull request at #986