Skip to content

Timeouts with MKL artifact download #4

@ablaom

Description

@ablaom

Been trying to update my HelloJulia.jl Pluto course to Julia 1.8 but running into problems. I can create a sys image fine. But when I call PrecompilePlutoCourse.start() it triggers a MKL artifact download (ultimately needed by Makie) which seems to timeout and then mysteriously reports that MKL is downloaded, but pluto never launches:

 Downloading artifact: MKL
ERROR: TaskFailedException===========================>    ]  89.9 %
    Downloading [====================================>    ]  90.0 %
    nested task error:     Downloading [=====================================>  Timed out waiting to read host:port string from worker.   ]  90.4 %
    Stacktrace:
     [1] worker_from_id(pg::Distributed.ProcessGroup, i::Int64)
       @ Distributed /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Distributed/src/cluster.jl:1092
     [2] worker_from_id
       @ /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Distributed/src/cluster.jl:1089 [inlined]
     [3] #remote_do#170
       @ /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Distributed/src/remotecall.jl:557 [inlined]
     [4] remote_do
       @ /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Distributed/src/remotecall.jl:557 [inlined]
     [5] kill
       @ /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Distributed/src/managers.jl:687 [inlined]
     [6] create_worker(manager::Distributed.LocalManager, wconfig::Distributed.WorkerConfig)
       @ Distributed /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Distributed/src/cluster.jl:603
     [7] setup_launched_worker(manager::Distributed.LocalManager, wconfig::Distributed.WorkerConfig, launched_q::Vector{Int64})
       @ Distributed /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Distributed/src/cluster.jl:544
     [8] (::Distributed.var"#45#48"{Distributed.LocalManager, Vector{Int64}, Distributed.WorkerConfig})()
       @ Distributed ./task.jl:484
    
    caused by: Timed out waiting to read host:port string from worker.
    Stacktrace:
     [1] read_worker_host_port(io::Base.PipeEndpoint)
       @ Distributed /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Distributed/src/cluster.jl:344
     [2] connect(manager::Distributed.LocalManager, pid::Int64, config::Distributed.WorkerConfig)
       @ Distributed /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Distributed/src/managers.jl:536
     [3] create_worker(manager::Distributed.LocalManager, wconfig::Distributed.WorkerConfig)
       @ Distributed /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Distributed/src/cluster.jl:599
     [4] setup_launched_worker(manager::Distributed.LocalManager, wconfig::Distributed.WorkerConfig, launched_q::Vector{Int64})
       @ Distributed /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Distributed/src/cluster.jl:544
     [5] (::Distributed.var"#45#48"{Distributed.LocalManager, Vector{Int64}, Distributed.WorkerConfig})()
       @ Distributed ./task.jl:484
Stacktrace:
 [1] sync_end(c::Channel{Any})
   @ Base ./task.jl:436
 [2] macro expansion
   @ ./task.jl:455 [inlined]
 [3] addprocs_locked(manager::Distributed.LocalManager; kwargs::    Downloading Base.Pairs{Symbol, Vector{String}, Tuple{Symbol}, NamedTuple{(:exeflags,), Tuple{Vector{String}}}})
   @ Distributed /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Distributed/src/cluster.jl:490
 [4] addprocs(manager::Distributed.LocalManager; kwargs::Base.Pairs{Symbol, Vector{String}, Tuple{Symbol}, NamedTuple{(:exeflags,), Tuple{Vector{String}}}})
   @ Distributed /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Distributed/src/cluster.jl:450
 [5] #addprocs#263
   @ /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Distributed/src/managers.jl:455 [inlined]
 [6] start(; sysimage::String)
   @ PrecompilePlutoCourse ~/HelloJulia/temporary_julia_depot/packages/PrecompilePlutoCourse/reLu6/src/PrecompilePlutoCourse.jl:35
 [7] start()
   @ PrecompilePlutoCourse ~/HelloJulia/temporary_julia_depot/packages/PrecompilePlutoCourse/reLu6/src/PrecompilePlutoCourse.jl:30
 [8] top-level scope
   @ REPL[6]:1
  Downloaded artifact: MKL

I try calling PrecompilePlutoCourse.start() again, and no artifact download occurs, but I get a new error:

Starting a Pluto notebook in your browser.
Please use `HelloJulia.stop()` to interrupt the Pluto notebook server.

julia>       From worker 3:	[ Info: Loading...
      From worker 3:	┌ Info: 
      From worker 3:	└ Opening http://localhost:1234/open?secret=LY6KePM8&path=%2FUsers%2Fanthony%2FHelloJulia%2Fnotebooks%2Fpluto_index.jl in your default browser... ~ have fun!
      From worker 3:	┌ Info: 
      From worker 3:	│ Press Ctrl+C in this terminal to stop Pluto
      From worker 3:	└ 
      From worker 3:	
      From worker 3:	signal (11): Segmentation fault: 11
      From worker 3:	in expression starting at none:1
      From worker 3:	ijl_subtype_env at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
      From worker 3:	YY.send_notebook_changesNOT.YY.345 at /Users/anthony/GoogleDrive/Julia/HelloJulia/precompile/hellojulia.so (unknown line)
      From worker 3:	response_update_notebook at /Users/anthony/GoogleDrive/Julia/HelloJulia/precompile/hellojulia.so (unknown line)
      From worker 3:	jfptr_response_update_notebook_154845 at /Users/anthony/GoogleDrive/Julia/HelloJulia/precompile/hellojulia.so (unknown line)
      From worker 3:	ijl_apply_generic at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
      From worker 3:	process_ws_message at /Users/anthony/HelloJulia/temporary_julia_depot/packages/Pluto/U4wvO/src/webserver/WebServer.jl:399
      From worker 3:	ijl_apply_generic at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
      From worker 3:	#397 at /Users/anthony/HelloJulia/temporary_julia_depot/packages/Pluto/U4wvO/src/webserver/WebServer.jl:190
      From worker 3:	#upgrade#13 at /Users/anthony/HelloJulia/temporary_julia_depot/packages/HTTP/VAOuj/src/WebSockets.jl:439
      From worker 3:	upgrade at /Users/anthony/HelloJulia/temporary_julia_depot/packages/HTTP/VAOuj/src/WebSockets.jl:419 [inlined]
      From worker 3:	#396 at /Users/anthony/HelloJulia/temporary_julia_depot/packages/Pluto/U4wvO/src/webserver/WebServer.jl:171
      From worker 3:	unknown function (ip: 0x10d4d9c02)
      From worker 3:	ijl_apply_generic at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
      From worker 3:	jl_f__call_latest at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
      From worker 3:	#invokelatest#2 at ./essentials.jl:729 [inlined]
      From worker 3:	invokelatest at ./essentials.jl:726 [inlined]
      From worker 3:	handle_connection at /Users/anthony/HelloJulia/temporary_julia_depot/packages/HTTP/VAOuj/src/Servers.jl:442
      From worker 3:	#16 at ./task.jl:484
      From worker 3:	unknown function (ip: 0x10d4c7dff)
      From worker 3:	ijl_apply_generic at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
      From worker 3:	start_task at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
      From worker 3:	Allocations: 18516703 (Pool: 18507923; Big: 8780); GC: 23
      From worker 3:	┌ Warning: temp cleanup
      From worker 3:	│   exception =
      From worker 3:	│    task switch not allowed from inside gc finalizer
      From worker 3:	│    Stacktrace:
      From worker 3:	│     [1] try_yieldto(undo::typeof(Base.ensure_rescheduled))
      From worker 3:	│       @ Base ~/GoogleDrive/Julia/HelloJulia/precompile/hellojulia.so:-1
      From worker 3:	│     [2] Channel{Tuple{String, Vector{String}, Vector{String}}}(func::Base.Filesystem.var"#28#31"{String}, size::Int64; taskref::Nothing, spawn::Bool)
      From worker 3:	│       @ Base ~/GoogleDrive/Julia/HelloJulia/precompile/hellojulia.so:-1
      From worker 3:	│     [3] prepare_for_deletion(path::String)
      From worker 3:	│       @ Base.Filesystem ~/GoogleDrive/Julia/HelloJulia/precompile/hellojulia.so:-1
      From worker 3:	│     [4] temp_cleanup_purge(; force::Bool)
      From worker 3:	│       @ Base.Filesystem ~/GoogleDrive/Julia/HelloJulia/precompile/hellojulia.so:-1
      From worker 3:	│     [5] (::Base.var"#886#887")()
      From worker 3:	│       @ Base ~/GoogleDrive/Julia/HelloJulia/precompile/hellojulia.so:-1
      From worker 3:	│     [6] _atexit()
      From worker 3:	│       @ Base ~/GoogleDrive/Julia/HelloJulia/precompile/hellojulia.so:-1
      From worker 3:	└ @ Base.Filesystem file.jl:551
      From worker 3:	task switch not allowed from inside gc finalizer
      From worker 3:	atexit hook threw an error: ErrorException("task switch not allowed from inside gc finalizer")
      From worker 3:	ijl_error at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
      From worker 3:	ijl_switch at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
      From worker 3:	try_yieldto at /Users/anthony/GoogleDrive/Julia/HelloJulia/precompile/hellojulia.so (unknown line)
      From worker 3:	wait at /Users/anthony/GoogleDrive/Julia/HelloJulia/precompile/hellojulia.so (unknown line)
      From worker 3:	uv_write at /Users/anthony/GoogleDrive/Julia/HelloJulia/precompile/hellojulia.so (unknown line)
      From worker 3:	unsafe_write at /Users/anthony/GoogleDrive/Julia/HelloJulia/precompile/hellojulia.so (unknown line)
      From worker 3:	print at /Users/anthony/GoogleDrive/Julia/HelloJulia/precompile/hellojulia.so (unknown line)
      From worker 3:	jfptr_print_69719 at /Users/anthony/GoogleDrive/Julia/HelloJulia/precompile/hellojulia.so (unknown line)
      From worker 3:	ijl_apply_generic at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
      From worker 3:	showerror at ./errorshow.jl:144
      From worker 3:	unknown function (ip: 0x10d51f2ed)
      From worker 3:	ijl_apply_generic at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
      From worker 3:	_atexit at /Users/anthony/GoogleDrive/Julia/HelloJulia/precompile/hellojulia.so (unknown line)
      From worker 3:	jfptr__atexit_73562 at /Users/anthony/GoogleDrive/Julia/HelloJulia/precompile/hellojulia.so (unknown line)
      From worker 3:	ijl_apply_generic at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
      From worker 3:	ijl_atexit_hook at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
      From worker 3:	ijl_exit at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
      From worker 3:	jl_exit_thread0_cb at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)Worker 3 terminated.

Unhandled Task ERROR: EOFError: read end of file
Stacktrace:
 [1] (::Base.var"#wait_locked#680")(s::Sockets.TCPSocket, buf::IOBuffer, nb::Int64)
   @ Base ./stream.jl:945
 [2] unsafe_read(s::Sockets.TCPSocket, p::Ptr{UInt8}, nb::UInt64)
   @ Base ./stream.jl:953
 [3] unsafe_read
   @ ./io.jl:759 [inlined]
 [4] unsafe_read(s::Sockets.TCPSocket, p::Base.RefValue{NTuple{4, Int64}}, n::Int64)
   @ Base ./io.jl:758
 [5] read!
   @ ./io.jl:760 [inlined]
 [6] deserialize_hdr_raw
   @ /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Distributed/src/messages.jl:167 [inlined]
 [7] message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:172
 [8] process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool)
   @ Distributed /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:133
 [9] (::Distributed.var"#103#104"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})()
   @ Distributed ./task.jl:484

I've tried this (from a fresh Julia depot) 3 times with the same result.

I'm guessing these errors are unrelated to my use of PrecompilePlutoCourse and am sorry for posting here, but I wondered if you had seen anything like this.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions