six hour video -> too many open files

[gerritholl]

When attempting to produce a six hour video by passing start/end time, using ABI, GLM, and caching, the maximum number of open files (1024 on my workstation) is exceeded.

Source:

from datetime import datetime
from sattools import log
from sattools import vis
from sattools import io
log.setup_main_handler()
vis.show_video_abi_glm_times(
        datetime(2020, 7, 18, 6, 0),
        datetime(2020, 7, 18, 12, 0),
        io.plotdir(create=True))

Result:

Traceback (most recent call last):
  File "show-remote-glm.py", line 15, in <module>
  File "/home/gholl/checkouts/sattools/src/sattools/vis.py", line 196, in show_video_abi_glm_times
  File "/home/gholl/checkouts/sattools/src/sattools/vis.py", line 168, in show_video_abi_glm
  File "/home/gholl/checkouts/sattools/src/sattools/scutil.py", line 29, in get_resampled_multiscene
  File "/home/gholl/checkouts/sattools/src/sattools/scutil.py", line 41, in _get_all_areas_from_multiscene
  File "/home/gholl/checkouts/satpy/satpy/multiscene.py", line 221, in scenes
  File "/home/gholl/checkouts/satpy/satpy/multiscene.py", line 123, in __iter__
  File "/home/gholl/checkouts/satpy/satpy/multiscene.py", line 113, in _create_cached_iter
  File "/home/gholl/checkouts/satpy/satpy/multiscene.py", line 262, in _call_scene_func
  File "/home/gholl/checkouts/satpy/satpy/multiscene.py", line 123, in __iter__
  File "/home/gholl/checkouts/satpy/satpy/multiscene.py", line 113, in _create_cached_iter
  File "/home/gholl/checkouts/satpy/satpy/multiscene.py", line 200, in <genexpr>
  File "/home/gholl/checkouts/satpy/satpy/scene.py", line 166, in __init__
  File "/home/gholl/checkouts/satpy/satpy/scene.py", line 215, in _create_reader_instances
  File "/home/gholl/checkouts/satpy/satpy/readers/__init__.py", line 513, in load_readers
  File "/home/gholl/checkouts/satpy/satpy/readers/yaml_reader.py", line 612, in create_filehandlers
  File "/home/gholl/checkouts/satpy/satpy/readers/yaml_reader.py", line 600, in _new_filehandlers_for_filetype
  File "/home/gholl/checkouts/satpy/satpy/readers/yaml_reader.py", line 568, in filter_fh_by_metadata
  File "/home/gholl/checkouts/satpy/satpy/readers/yaml_reader.py", line 502, in _new_filehandler_instances
  File "/home/gholl/checkouts/satpy/satpy/readers/abi_base.py", line 52, in __init__
  File "/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/fsspec/implementations/cached.py", line 376, in <lambda>
  File "/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/fsspec/spec.py", line 897, in open
  File "/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/fsspec/implementations/cached.py", line 376, in <lambda>
  File "/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/fsspec/implementations/cached.py", line 324, in _open
  File "/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/fsspec/caching.py", line 54, in __init__
  File "/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/fsspec/caching.py", line 76, in _makefile
OSError: [Errno 24] Too many open files

[gerritholl]

I seem to have resolved this in a recent commit to pytroll/satpy#1321, although now I'm getting 'ValueError's for operation on closed file instead, despite explicitly closing and reopening including the phantom object...

[gerritholl]

See the comments at pytroll/satpy#1321 (comment) for thoughts on why this is happening.