Periodic missing satellite sectors

[corepuncher]

LDM 6.14.5 on Debian

Issue: Despite a strong signal, on some days we get missing satellite sectors, especially goes channel 02 / VIS.
Other channels like 09 and 14 always arrive.
Seems between 12-18Z are the most common times for this to occur.

For example, this file may be missing: TIRE02_KNES_031416_PAI.nc

Other places online do not show the missing sector.

Any ideas what may be happening?

Does LDM have any sort of retry mechanism, or, perhaps a way to prioritize data types, assuming it could be a processing issue?

Have you ever heard of this issue?

Here is our pqact.goes:

# CONUS VIS
NOTHER  ^(TIRE02) (....) (......) (...)
        PIPE    -close  /home/ldm/scripts/goes/goes_stitch.py /mnt/data/goes16/incoming/02/\1_\2_\3_\4.nc

[sebenste]

Try upgrading to 6.15.0. It fixed that issue for me. Using Ubuntu 22.04.

[corepuncher]

Try upgrading to 6.15.0. It fixed that issue for me. Using Ubuntu 22.04.

Excellent! Thank you Mr. Sebenste. Hopefully it goes smoothly, I"m always afraid something will get overwritten.

[corepuncher]

Try upgrading to 6.15.0. It fixed that issue for me. Using Ubuntu 22.04.

Well shoot, that didn't work. Still getting a missing VIS sector about once per day.

I do wonder if maybe my product queue and the registry values are not correct. I noticed that doing:

ldmadmin add metrics

Sometimes the last value is large like 11,000 seconds (well above 3600), but sometimes, it drops to like 50, which is well below the 3600 seconds in registry. I upped the cache to 2 GB from the default 500 mb. Strangely, it seems to use disk space, whereas I thought it was going to be stored in RAM.

20240708.022040 2.31 2.68 2.54 0 0 475 3566 353602992 35 1 63 0 1938010112 266407936 37650432 984707072 1790
20240708.022516 8.88 4.81 3.35 0 0 42 3565 347372648 62 9 29 1 2011152384 832516096 39239680 983117824 13111
20240708.024318 2.16 0.78 0.28 0 0 185 14626 1100868304 4 4 93 0 3186950144 6914895872 0 1022357504 12575
20240708.025543 1.56 1.76 1.19 0 0 180 14625 1453676600 8 4 87 0 3338756096 5446045696 0 1022357504 11241
20240708.030230 1.39 1.61 1.33 0 0 214 14623 1279613112 5 3 91 0 3325075456 5086826496 0 1022357504 11073
20240708.032556 5.43 4.83 3.62 0 0 243 14626 1163054192 67 8 24 0 3640803328 314605568 4755456 1017602048 10720
20240708.043836 2.13 2.95 3.78 0 0 194 14623 1478714984 10 5 84 0 3503296512 598880256 34553856 987803648 12345
20240709.012915 4.17 5.07 4.31 0 0 209 14619 1149290944 15 5 80 0 3524358144 603381760 52912128 969445376 11952
20240711.172248 3.88 3.95 3.23 0 0 169 14615 873140952 10 23 0 0 3652431872 249581568 47022080 975335424 66
20240711.172533 3.88 4.02 3.38 0 0 186 14613 1437809024 61 10 28 1 3760893952 242573312 47022080 975335424 14019
20240713.001427 3.87 2.78 2.42 0 0 246 14602 1209811440 7 52 0 0 3666370560 446566400 49774592 972582912 40
20240713.002240 4.24 4.40 3.41 0 0 229 14626 1069907320 53 7 40 1 3587522560 315830272 31948800 990408704 9111
20240713.032145 5.21 4.61 3.65 0 0 134 14621 868003848 48 5 48 0 3596095488 386326528 42000384 980357120 8042

The 500 MB queue just seems really low IMO, at least for 2024. I wonder if there is a place to find updated/recommended configurations for LDM nowadays?

[sebenste]

Wait, you only had 500 MB? You need 20 GB for that feed alone...

[corepuncher]

That's good to know Mr. Sebenste, thank! I assumed the values were "good" since they came straight from Unidata, and (surprisingly?) things seemed to run "fairly well" even with only that much.

So I'll up the 2048M
To I guess 20480M

Any other major things that should be upped from defaults? I have a feeling this may fix the missing sectors. Those satellite files are large, and somtimes the "last" one comes in so late, the next image time is already processing.

<?xml version="1.0"?>
<registry>
  <delete-info-files>0</delete-info-files>
  <hostname>LDM1.site.net</hostname>
  <insertion-check-interval>300</insertion-check-interval>
  <reconciliation-mode>do nothing</reconciliation-mode>
  <check-time>
    <enabled>1</enabled>
    <limit>10</limit>
    <warn-if-disabled>1</warn-if-disabled>
    <ntpdate>
      <command>ntpdate</command>
      <servers>us.pool.ntp.org</servers>
      <timeout>5</timeout>
    </ntpdate>
  </check-time>
  <log>
    <count>7</count>
    <file>/home/ldm/var/logs/ldmd.log</file>
    <rotate>1</rotate>
  </log>
  <metrics>
    <count>4</count>
    <file>/home/ldm/var/logs/metrics.txt</file>
    <files>/home/ldm/var/logs/metrics.txt*</files>
    <netstat-command>/usr/bin/netstat -A inet -t -n</netstat-command>
    <top-command>/usr/bin/top -b -n 1</top-command>
  </metrics>
  <pqact>
    <config-path>/home/ldm/etc/pqact.conf</config-path>
    <datadir-path>/home/ldm/data</datadir-path>
  </pqact>
  <pqsurf>
    <config-path>/home/ldm/etc/pqsurf.conf</config-path>
    <datadir-path>/home/ldm/data</datadir-path>
  </pqsurf>
  <queue>
    <path>/home/ldm/var/queues/ldm.pq</path>
    <size>2048M</size>
    <slots>default</slots>
  </queue>
  <scour>
    <config-path>/home/ldm/etc/scour.conf</config-path>
    <exclude-path>/home/ldm/etc/scour_excludes.conf</exclude-path>
  </scour>
  <server>
    <config-path>/home/ldm/etc/ldmd.conf</config-path>
    <ip-addr>0.0.0.0</ip-addr>
    <max-clients>256</max-clients>
    <max-latency>3600</max-latency>
    <port>388</port>
    <time-offset>3600</time-offset>
    <enable-anti-DOS>TRUE</enable-anti-DOS>
  </server>
  <surf-queue>
    <path>/home/ldm/var/queues/pqsurf.pq</path>
    <size>2M</size>
  </surf-queue>
  <oess-pathname>/home/ldm/etc/OESS-account.yaml</oess-pathname>
  <fmtp-retx-timeout>300</fmtp-retx-timeout>
</registry>

[sebenste]

That should do it! You should be good to go.