Message list doesn't update, but throughput graphs update

[jjo93sa]

I've tried running the latest version of cli-dashboard in a Docker container using various versions of node, currently current-buster-slim, which is node 12.10. (I've tried other node in Docker versions back to 6.17 with the same results.)

The dashboard displays just fine, and the graphs seem to update in realtime. However, the message list seems to be stuck on the first few events in the stream, and never updates. For example, it is now 21:25 UTC, but the last message displayed in the widget is from 15:58 UTC. If I check the stream in Graylog, the last message I see is 21:23.

Does anyone have any ideas? My working hypothesis is that this is something to do with changes to the Graylog API.

[jjo93sa]

I notice also that the newest message in the list is at the bottom, which isn't the same as the screen shot in the README.md (where the newest message seems to be inserted at the top of the list)

[jjo93sa]

Further updates (for anyone coming along behind me). I've found the following:

1. The version of this tool in the npm repository (sorry of this is incorrect terminology) seems to be different than the latest version in this repo, which I found when capturing the API calls and seeing that the ordering requested was ascending, despite all the files in this repository indicating a descending order;
2. The npm version does not seem to work for me (at least not in Docker),per earlier comment the graphs plot but the log messages are not updated, and the log messages that do appear are in the wrong order;
3. Installing the latest version of this repository resolves some of the problems (e.g. installation from my fork using command npm install github:jjo93sa/cli-dashboard -g), namely that log messages update, but the last message in the list (at the bottom) is several messages behind real time. I suspect this is because of the way the messages are being logged in the handler.js function updateMessagesList using this call messageList.setItems(items);
4. There seems to be quite a lot of interplay between the size of the messageList buffer and the number of events retrieved from the Graylog API via the batch size. Making the event count < buffer results in the most up-to-date message being shown at the bottom of the log list, albeit not at the bottom of the pane. I think this and point (3) are both due to the setItems(items) call, which bypasses the Log widget scroll to end step;
5. I've tried adding the scroll to end step from the blessed-contrib Log widget: scrollTo(this.logLines.length) but whilst it causes the display to update, it flickers too much for my liking;
6. My best result has been to modify handlers.js thus:

    updateMessagesList(messages) {
        const messageList = ui.getWidget('messageList');

        const items = messages
        .sort((message1, message2) => new Date(message1.timestamp) - new Date(message2.timestamp))
        .map((message) => `${moment(message.timestamp).format()} - ${message.message}`);

        let old_items = messageList.logLines;
        let difference = items.filter(x => !old_items.includes(x));
        difference.forEach(item => messageList.log(item));
    },

Which I think (a) orders the message list from the Graylog API, (b) gets the current message list from the messageList Log widget (c) takes the difference between those two arrays and (d) iteratively adds the new log messages to the display. However, I don't know if (b) is working and I don't know JS. Even so, it seems necessary to change the following parameter in Graylog-dashboard.js:

  .describe('batch', 'events per request')
  .default('batch','30')

such that the batch size is smaller than the buffer in the Log widget (set to 50). Without this step, occasionally old log messages appear again in the window.

My intention is to push a version of my files to my forked repository here. I will also publish the Dockerfile here

[jonmelia]

Nice work on the above, it now seems to be logging correctly for me..

[jjo93sa]

@jonmelia - that's great. Could you please comment on which version you're running successfully?

[jonmelia]

@jonmelia - that's great. Could you please comment on which version you're running successfully?

I'm using your fork jjo93sa. Because we get so many messages it's difficult to see how far behind real-time they are, but for our purposes this isn't much of an issue as long as we can see them rotating in near realtime. The graylog-labs version just kept freezing and wouldn't update some streams