Support for markdown cells

[wangshaonan]

The current Chatify only works on code. I think it would be more useful to make Chatify work on notes and even videos. It doesn't have to be embedded in Jupyter Notebook, a chatbot like the automatic customer service we saw on various applications including shopping websites would also be helpful. In that way, whatever question we have, we can directly open the chatbot and ask questions.

[jeremymanning]

Thanks @wangshaonan! Copying some comments from slack for our records:

the %%explain syntax is an iPython “magic” command. those work by “hijacking” the contents of code cells and passing them to the relevant functions (specified wherever the magic command is defined/registered). the way chatify works is that the contents of the code cell gets sent to the language model, along with a pre-defined prompt that depends on what you select in the chatify menus.

unfortunately there’s no built-in way of running magic commands on markdown cells, which is where those notes are defined. it’d be nice to support markdown cells, though! and support for other media, like videos, images, etc., would be neat too.

i do suspect that it is possible to support those things, but i think it’d require (a) some hacking to change how those markdown cells are rendered and (b) some thinking about how the implementation would work. for example:

• how would chatify get triggered from markdown cells? e.g., suppose the markdown cell contains the %%explain command on top. does that mean it should be treated as a call to chatify? if so, how should the notebook render that cell and/or respond?
• should there be an “escape sequence”? e.g., you can include a comment in front of the %%explain magic command in a code cell to prevent it from executing that magic command. and since %%explain isn’t valid Python (except as a magic command), there’s no loss of function when its used to invoke chatify. (it just uses the “magic command” infrastructure that is already in place.). but in markdown cells, it’s perfectly “valid” to include %%explain at the top of a cell. How could you distinguish when the user intends to call chatify, versus if they just want the text “%%explain” to appear at the top of the cell, and get rendered as markdown? (this is probably why iPython notebooks don’t support magic commands in markdown cells by default.)

one could also imagine creating a new magic command that works like %%explain, but instead of sending the text of the current code cell (where the magic command is called), it could instead read in the current notebook and returns the content of one or more other cells, potentially including markdown cells. how should the syntax for that command work? maybe something like: if you follow an %%explain command with a non-negative integer, it sends the contents of that cell instead of the current cell (where the current cell is the default if no cell ID is specified)?

videos are an interesting case too. videos are displayed via iPython widgets, which are called in code cells. technically you can already “explain” those cells using chatify. however, chatify interprets those cells as code rather than trying to guess at what the code does on an arbitrarily deep level (e.g., by following links, actually running the code, etc.). one could imagine hard-coding in some tweaks, e.g. if the only code a cell contains is a call to an ipython widget displaying a video, maybe there could be an additional tab that comes up that enables you do do something related to the content of that video (e.g., answering questions about the video’s transcript, etc.). we’d need to then figure out what to do if multiple videos are embedded in a single code cell (e.g., do you then get multiple extra “video” tabs?). and maybe it’d be good to support other types of media too, like interactive demos, embedded web pages, audio, animations, etc.

anyways, cool idea!