Intermittent error "Could not get stream throughput" 404; api response "I do not have a metric called [org.graylog2.plugin.streams.Stream.5891ca6995186c0001613209.incomingMessages]."

[vincebowdrentribal]

Issue: The API call to graylog which gets a stream throughput is sometimes failing, being given a 404 response with the message that there is no such metric. This is unpredictable, sometimes failing for a given stream but sometimes succeeding

Details of setup

• running both graylog and cli-dashboard in docker containers, on different hosts
• graylog version is 2.1.3-1 (graylog2/server:latest from https://hub.docker.com/r/graylog2/server/ as of 2017-02-02)
• graylog's api is listening on port 12901
• there is an nginx reverse proxy in front of it, listening on port 12900 (this is for the convenience of seeing the api requests, for debugging)
• graylog-cli-dashboard is version 1.0.0 (npm install as of 2017-02-02)
• graylog-cli-dashboard is running on node v6.9.5 (most recent node v6 build available in docker as of 2017-02-02)
• graylog-cli-dashboard is running in a docker container, with the image built using an npm install as of 2017-02-02 with the following Dockerfile:

FROM node:6
RUN npm install graylog-cli-dashboard -g
ENTRYPOINT ["graylog-dashboard"]

• graylog-cli-dashboard is invoked with the following docker command:

docker run \
--rm \
-it \
myprivatedockerrepository:graylog-cli \
--server-url http://myprivatedevbox:12900 \
--stream-title "si-w2" \
--username user \
--password password

• the name of the repository and the ip address of the dev box have been changed, otherwise this is verbatim; the stream title is correct, and accessible by the configured user account
• note that I have deliberately configured graylog so it can be accessed on port 12900, to avoid confusion with the advice given in issue Could not get stream throughput #25.

Expected Results

• graylog-cli runs when started
• all api requests connect and get a 200 response
• the stream's messages are shown
• I can switch streams in the UI, with the same successful results

Actual results

1. (sometimes) graylog-cli runs, and makes a number of successful api calls, before getting a 404 on the call which is trying to get the stream throughput (GET /system/metrics/...) and crashes out.
2. (sometimes) graylog-cli runs successfully. When I pick a different stream from the UI by double-clicking, it gets a 404 on the call which is trying to get the stream throughput (GET /system/metrics/...) and crashes out.

Error and callstack shown at the command-line:

Error: Could not get stream throughput. Bad status code: 404 Not Found [object Object]
    at makeRequest.then.catch (/usr/local/lib/node_modules/graylog-cli-dashboard/lib/graylog-api.js:77:13)
    at tryCatcher (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/promise.js:510:31)
    at Promise._settlePromise (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/promise.js:567:18)
    at Promise._settlePromise0 (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/promise.js:612:10)
    at Promise._settlePromises (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/promise.js:687:18)
    at Async._drainQueue (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/async.js:133:16)
    at Async._drainQueues (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/async.js:143:10)
    at Immediate.Async.drainQueues (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/async.js:17:14)
    at runCallback (timers.js:649:20)
    at tryOnImmediate (timers.js:622:5)
    at processImmediate [as _immediateCallback] (timers.js:594:5)

nginx log showing the api calls made by cli-dashboard:

"GET /streams HTTP/1.1" 200 8182 "-" "-"
"GET /system/metrics/org.graylog2.plugin.streams.Stream.5891bf7595186c0001612671.incomingMessages.1-sec-rate HTTP/1.1" 404 152 "-" "-"
"GET /streams/5891bf7595186c0001612671/alerts/check HTTP/1.1" 200 77 "-" "-"
"GET /search/universal/relative?query=%2A&range=86400&fields=timestamp%2Cmessage&filter=streams%3A5891bf7595186c0001612671&limit=50 HTTP/1.1" 499 0 "-" "-"
"GET /system/throughput HTTP/1.1" 499 0 "-" "-"

Note:

1. The first call (GET /streams) gets a 200 response; this confirms that the server-url configuration is correct and that graylog-cli-dashboard can connect to the graylog api
2. The second call (GET /system/metrics/... ) fails with a 404
3. The third call (GET /streams/.../alerts/check) succeeds with a 200
4. The fourth call (GET search/...) receives a 499 response
5. The fifth call (GET /system/throughput) also receives a 499 response

Double-checking in the API browser

On the graylog server (http://myprivatedevbox:12900), it is possible to load the api-browser page to examine and try out api calls. The metrics call can be retried here, again getting a 404 response:

• Request URL: http://10.250.130.125:12900/system/metrics/org.graylog2.plugin.streams.Stream.5891bf7595186c0001612671.incomingMessages.1-sec-rate
• Response Code: 404
• Response Body:

{
  "type": "ApiError",
  "message": "I do not have a metric called [org.graylog2.plugin.streams.Stream.5891bf7595186c0001612671.incomingMessages.1-sec-rate]."
}

Further notes

• The problem is semi-intermittent. Most often, if I run graylog-cli-dashboard for the same stream as before, it will fail as before (or succeed as before). No obvious pattern about when and why it changes.

• see further screenshots (031694-031696) showing an example of when it starts successfully, but then fails on switching to a different stream, getting the same 404 response and crashing out.

• 031694_screenshot

• 031695_screenshot

• 031696_screenshot

[aarvee11]

i too have the same issue

Error: Could not get streams. Error: Bad status code: 404 Not Found [object Object]
at request.getAsync.then (/usr/local/lib/node_modules/graylog-cli-dashboard/lib/graylog-api.js:98:13)
at tryCatcher (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/util.js:16:23)
at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/promise.js:510:31)
at Promise._settlePromise (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/promise.js:567:18)
at Promise._settlePromise0 (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/promise.js:612:10)
at Promise._settlePromises (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/promise.js:691:18)
at Promise._fulfill (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/promise.js:636:18)
at Request._callback (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/nodeback.js:42:21)
at Request.self.callback (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/request/request.js:186:22)
at emitTwo (events.js:106:13)
at Request.emit (events.js:191:7)
at Request. (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/request/request.js:1081:10)
at emitOne (events.js:96:13)
at Request.emit (events.js:188:7)
at IncomingMessage. (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/request/request.js:1001:12)
at Object.onceWrapper (events.js:290:19)
at makeRequest.then.catch (/usr/local/lib/node_modules/graylog-cli-dashboard/lib/graylog-api.js:65:13)
at tryCatcher (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/util.js:16:23)
at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/promise.js:510:31)
at Promise._settlePromise (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/promise.js:567:18)
at Promise._settlePromise0 (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/promise.js:612:10)
at Promise._settlePromises (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/promise.js:687:18)
at Async._drainQueue (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/async.js:133:16)
at Async._drainQueues (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/async.js:143:10)
at Immediate.Async.drainQueues (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/async.js:17:14)
at runCallback (timers.js:651:20)
at tryOnImmediate (timers.js:624:5)
at processImmediate [as _immediateCallback] (timers.js:596:5)

[henry911]

I'm having a similar error. Though the API definitely working outside of graylog-cli, from the same box graylog-cli is installed. Below is an example of the graylog-cli call and result and corresponding curl call and result.

Setup

Graylog Version: 2.1.3+040d371
Node.js Version: 6.10.0
NPM Versoin: 3.10.10

Call being made through graylog-cli

graylog-dashboard --stream-title "exceptions" --server-url "http://server:9000/api/"

Results/Error from graylog-cli

Error: Could not get streams. Error: Bad status code: 404 Not Found [object Object]
    at request.getAsync.then (/usr/lib/node_modules/graylog-cli-dashboard/lib/graylog-api.js:98:13)
    at tryCatcher (/usr/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/usr/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/promise.js:510:31)
    at Promise._settlePromise (/usr/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/promise.js:567:18)
    at Promise._settlePromise0 (/usr/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/promise.js:612:10)
    at Promise._settlePromises (/usr/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/promise.js:691:18)
    at Promise._fulfill (/usr/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/promise.js:636:18)
    at Request._callback (/usr/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/nodeback.js:42:21)
    at Request.self.callback (/usr/lib/node_modules/graylog-cli-dashboard/node_modules/request/request.js:186:22)
    at emitTwo (events.js:106:13)
    at Request.emit (events.js:191:7)
    at Request.<anonymous> (/usr/lib/node_modules/graylog-cli-dashboard/node_modules/request/request.js:1081:10)
    at emitOne (events.js:96:13)
    at Request.emit (events.js:188:7)
    at IncomingMessage.<anonymous> (/usr/lib/node_modules/graylog-cli-dashboard/node_modules/request/request.js:1001:12)
    at IncomingMessage.g (events.js:291:16)
    at makeRequest.then.catch (/usr/lib/node_modules/graylog-cli-dashboard/lib/graylog-api.js:65:13)
    at tryCatcher (/usr/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/usr/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/promise.js:510:31)
    at Promise._settlePromise (/usr/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/promise.js:567:18)
    at Promise._settlePromise0 (/usr/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/promise.js:612:10)
    at Promise._settlePromises (/usr/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/promise.js:687:18)
    at Async._drainQueue (/usr/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/async.js:133:16)
    at Async._drainQueues (/usr/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/async.js:143:10)
    at Immediate.Async.drainQueues (/usr/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/async.js:17:14)
    at runCallback (timers.js:651:20)
    at tryOnImmediate (timers.js:624:5)
    at processImmediate [as _immediateCallback] (timers.js:596:5)

Call being made through CURL

curl -v -s -L --user "username:password" "http://server:9000/api/streams"

Results from CURL

* Hostname was NOT found in DNS cache
*   Trying xxx.xxx.xxx.xxx...
* Connected to server (xxx.xxx.xxx.xxx) port 9000 (#0)
* Server auth using Basic with user 'username'
> GET /api/streams HTTP/1.1
> Authorization: Basic ZGFzaGJUvYXJkOmRib2FyZDkxMQ=
> User-Agent: curl/7.35.0
> Host: server:9000
> Accept: */*
> 
< HTTP/1.1 200 OK
< X-Graylog-Node-ID: 10061e19-d96a-4212-f41c-60e22e0bc546
< X-Runtime-Microseconds: 102373
< Content-Type: application/json
< Date: Wed, 22 Feb 2017 20:29:28 GMT
< Content-Length: 967
< 
* Connection #0 to host server left intact
{
	"total":2,
	"streams":[{
		"id":"58efae86e51a5f3cab75135e",
		"creator_user_id":"admin",
		"outputs":[],
		"matching_type":"AND",
		"description":"exceptions",
		"created_at":"2017-02-21T21:17:58.215Z",
		"disabled":false,
		"rules":[{
			"field":"facility",
			"stream_id":"58efae86e51a5f3cab75135e",
			"description":"",
			"id":"58a761aee51a5f3cab6f83c5",
			"type":1,
			"inverted":false,
			"value":"exceptions"
		}],
		"alert_conditions":[],
		"alert_receivers":{
			"emails":[],
			"users":[]
		},
		"title":"exceptions",
		"content_pack":null
	},{
		"id":"58a7611ae51a5f3cab6f8344",
		"creator_user_id":"admin",
		"outputs":[],
		"matching_type":"AND",
		"description":"default.",
		"created_at":"2017-02-17T20:55:22.983Z",
		"disabled":false,
		"rules":[{
			"field":"Source",
			"stream_id":"58a7633ae51a5f3cab6f8344",
			"description":"",
			"id":"58a761aee51a5f3cab6f83c5",
			"type":5,
			"inverted":false,
			"value":""
		}],
		"alert_conditions":[],
		"alert_receivers":{
			"emails":[],
			"users":[]
		},
		"title":"other_stream",
		"content_pack":null
	}]
}

[henry911]

I decide to output what the serverURL was getting set as.

{
    _: [],
    'stream-title': 'exceptions',
    streamTitle: 'exceptions',
    'server-url': 'http://server:9000/api/',
    serverUrl: 'http://server:9000/api/',
    'poll-interval': 1000,
    pollInterval: 1000,
    'cred-file-path': '/root/.graylog_dashboard',
    credFilePath: '/root/.graylog_dashboard',
    '$0': 'graylog-dashboard',
    serverURL: 'http://server:9000/api/:12900/',
    username: 'username',
    password: 'password'
}

graylog-dashboard.js lines 88-94

    // Make sure we have a protocol (default: https)
    if (config.serverURL.slice(0, 4) !== 'http') config.serverURL = 'https://' + config.serverURL;
    // Make sure we have a port (default REST API port is 12900)
    if (!/:\d+$/.test(config.serverURL)) config.serverURL += ':12900';
    // Make sure config.serverURL has a trailing slash. (computers.)
    if (config.serverURL[config.serverURL.length - 1] !== '/') config.serverURL += '/';
    return config;

Since serverURL: 'http://server:9000/api/:12900/', looks horribly incorrect, I decided to comment out the lines of code making the changes to the URL.

With that being said, if you comment out lines 89, 91, 93 (for testing).

    // Make sure we have a protocol (default: https)
    //if (config.serverURL.slice(0, 4) !== 'http') config.serverURL = 'https://' + config.serverURL;
    // Make sure we have a port (default REST API port is 12900)
    //if (!/:\d+$/.test(config.serverURL)) config.serverURL += ':12900';
    // Make sure config.serverURL has a trailing slash. (computers.)
    //if (config.serverURL[config.serverURL.length - 1] !== '/') config.serverURL += '/';
    return config;

Which results in getting a different error message:

Error: Could not get stream throughput. Bad status code: 404 Not Found [object Object]
    at makeRequest.then.catch (/usr/lib/node_modules/graylog-cli-dashboard/lib/graylog-api.js:77:13)
    at tryCatcher (/usr/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/usr/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/promise.js:510:31)
    at Promise._settlePromise (/usr/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/promise.js:567:18)
    at Promise._settlePromise0 (/usr/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/promise.js:612:10)
    at Promise._settlePromises (/usr/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/promise.js:687:18)
    at Async._drainQueue (/usr/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/async.js:133:16)
    at Async._drainQueues (/usr/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/async.js:143:10)
    at Immediate.Async.drainQueues (/usr/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/async.js:17:14)
    at runCallback (timers.js:651:20)
    at tryOnImmediate (timers.js:624:5)
    at processImmediate [as _immediateCallback] (timers.js:596:5)

Which I took a look at the URL it was creating to get the metric for said stream and apparently it does not exist. Which I'm comparing this to getting all the metrics and all the metric names.

const url = `${options.serverURL}system/metrics/org.graylog2.plugin.streams.Stream.${options.streamID}.incomingMessages.1-sec-rate`

In the metrics output, the closest thing I see to this is:

"org.graylog2.plugin.streams.StreamRule.stream_id.executionTime",
"org.graylog2.plugin.streams.StreamRule.stream_id.executionTime",

Is there additional setup needed on that stream itself, that I missed, which is required to be configured for the stream to be used with the graylog-cli ?

[STRML]

Hi - I plan to take a look at this soon. Would be happy to accept a PR that refactors this logic and does some better error logging.

[tholu]

Similar problem here with a newly configured stream in latest version 1.1.1 with Graylog 2.3.0:

Error: Could not get stream throughput. Bad status code: 404 Not Found {"type":"ApiError","message":"I do not have a metric called [org.graylog2.plugin.streams.Stream.598ae2472ab79c000141ef7c.incomingMessages.1-sec-rate]."}
    at makeRequest.then.catch (/usr/local/lib/node_modules/graylog-cli-dashboard/lib/graylog-api.js:87:13)
    at tryCatcher (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/promise.js:512:31)
    at Promise._settlePromise (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/promise.js:569:18)
    at Promise._settlePromise0 (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/promise.js:689:18)
    at Async._drainQueue (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/async.js:133:16)
    at Async._drainQueues (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/async.js:143:10)
    at Immediate.Async.drainQueues (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/async.js:17:14)
    at runCallback (timers.js:781:20)
    at tryOnImmediate (timers.js:743:5)
    at processImmediate [as _immediateCallback] (timers.js:714:5)

[runger1101001]

+1 from me. Basically I'm guessing this occurs in environments where the rate can't be computed because the number of messages is low, but that's just a guess.

Please fix as the tool can't be used as is :-(

graylog-dashboard
Error: Could not get stream throughput. Bad status code: 404 Not Found {"type":"ApiError","message":"I do not have a metric called [org.graylog2.plugin.streams.Stream.5a008e503bcd87b21e595f93.incomingMessages.1-sec-rate]."}
at makeRequest.then.catch (/usr/local/lib/node_modules/graylog-cli-dashboard/lib/graylog-api.js:87:13)
at tryCatcher (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/util.js:16:23)
at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/promise.js:512:31)
at Promise._settlePromise (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/promise.js:569:18)
at Promise._settlePromise0 (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/promise.js:614:10)
at Promise._settlePromises (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/promise.js:689:18)
at Async._drainQueue (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/async.js:133:16)
at Async._drainQueues (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/async.js:143:10)
at Immediate.Async.drainQueues (/usr/local/lib/node_modules/graylog-cli-dashboard/node_modules/bluebird/js/release/async.js:17:14)
at runCallback (timers.js:637:20)
at tryOnImmediate (timers.js:610:5)
at processImmediate [as _immediateCallback] (timers.js:582:5)

[diranged]

I ran into this issue .. but worse than this, the code doesn't seem to work if you are hitting a dedicated webui endpoint, and have multiple graylog data ingestion nodes. The nodes themselves get the metric that its looking for, but the webui endpoint doesn't expose it. I think unfortunately that fixing this would require a pretty hefty refactor.. :/