3.1 - Structured Streaming - "books" table not registered

[Lewandowski-commits]

Running cell 4 of notebook 3.1 - Structured Streaming fails, because the table is not being registered within the setup notebook /Includes/Copy-Datasets or within this notebook.

Running the following code first is solves that, but it needs integrating into the setup notebook ideally

spark.sql( f""" CREATE EXTERNAL TABLE IF NOT EXISTS books USING PARQUET OPTIONS ( header "true", inferSchema "true", recursiveFileLookup "true" ) LOCATION '{dataset_bookstore}/orders-streaming' """ )

[chlorenz]

Hi there,

unfortunately this will not work with Unity Catalog, which is now the standard for the Associate exam (see also branch unity-catalog). To create the books table you first need to find the external location as in Notebook 1.4.,

dbutils.widgets.text("external_location", '<URL>/external_storage')
external_location = dbutils.widgets.get("external_location")

copy the files to the external location,

dbutils.fs.cp(f"{dataset_bookstore}/books-streaming", f"{external_location}/books-streaming", recurse=True)

and finally create the table:

%sql
CREATE TABLE IF NOT EXISTS books 
USING JSON
OPTIONS (
  path '${external_location}/books-streaming',
  header "true", 
  inferSchema "true", 
  recursiveFileLookup "true"
); 

However, since there are many more minor and major things that are broken/out-of-date, with the new UI and Unity Catalog I feel it would be best for @derar-alhussein to iterate on the course as a whole.