Looks like #{...} can often only accept immediate values. For example:
$ sbcl
* (require :reader)
NIL
* (require :hash-set)
NIL
* (reader:enable-reader-syntax 'get-val 'hash-table 'not 'string 'describe 'array 'set 'run-program)
#<READTABLE {100261DBA3}>
* (defvar x #{(cons 1 2)})
debugger invoked on a TYPE-ERROR @536416CB in thread
#<THREAD "main thread" RUNNING {1004DC81F3}>:
The value
2
is not of type
LIST
;; ...
* (defvar s #{:a 1 :b 2})
S