Adding highlights as custom elements

[BasilPH]

First of all, thank you very much for working on this and open-sourcing it! I am working on a tool for podcasters to edit their show transcripts. A core feature is that they should be able to create highlights. A highlight is basically a start- and end-timestamp that can cross over multiple speakers.

Browsing the code, I think I would need to add some logic to TimedTextElement that checks if a given word is in the interval of a highlight and changes its styling accordingly. Does this mean rewriting TimedTextElement, or is there any way I can hook the editor more smartly? Happy for any other input you might have.

[pietrop]

Hi @BasilPH ,
Yes, I think there's a few things to do to enable a feature like that. But I think you'd have to tweak TimedTextElement in the current setup.

For selection, some of the logic in slateJs hovering-toolbar example might be useful.

Fo displaying the selection, yes, probably something with ranges of selection, and then adding some markup like underlying highlight the text. That can be done at word level or paragraph level depending on how you do the selection.

In autoEdit.io I deliberately separate the correction stage from the markup/hilighting / selection stage (see the user manual, or download the app for more info) to keep the slateJs editor as simple as possible. And then have a separate view that allows to do selection, and display text that has already been selected/tagged/annotated with labels.

Hope this helps. Let me know how you get on. Would be interesting to see what approach you take.

[pietrop]

oh, also see this comment #21 (comment) might be relevant for what you are trying to do

[BasilPH]

Thank you for your response @pietrop. I didn't have as much time as I hoped to really dig into this, I'll give a more detailed updated once I'm further along. So far though:

• The hovering toolbar looks really interesting. I'm planning on using it.
• I think your approach of splitting the editing and then highlighting makes sense. I think we'll start with the same separation because i) it makes the code easier and ii) highlighting and editing might be a bit fiddly for users, especially on mobile. For example, if you tap to highlight you might not want your keyboard to pop up. I'll definitely want to test with users though.
• Our first prototype uses React and Redux to highlight the current word being spoken:
  
  It does work well for transcripts that are about 1.5 hours long, but I fear that it might break down with larger transcripts or once we add more features. It's pretty efficient, because each word has an ID, and so we only re-render that specific word and the one before it when we jump to the next one.

[pietrop]

Hi @BasilPH
One distinction I did in the past, was single click and double click. Eg single click to edit, and double click to jump to that point. (but it's an "hidden interaction" so you'd have to let your users know in some other way, via text or onboarding info etc...)

Would you be up for sharing a PR with the hilighting? would love to see your approach, and test it out on longer transcript (eg 5 hours from the storybook demo) to see how it performs.

[BasilPH]

I'm definitely up for sharing the PR if something usable comes out of my experiment. I'm splitting the editing and highlighting, as you also do in autoEdit.io. I'm still using SlateTranscriptEditor for the highlighting view, but with isEditable=false: This allows me to reuse the logic and styling you already implemented.

I'm thinking of Highlights as simple time ranges that have a start and end. They can span across multiple speakers, so we have to model them at the word level.

The approach for "Custom formatting" in the Slate.js documentation could be the way to go. If you look at the last code box at the very end of the linked page, you see that the leaf node decides if it should render itself bold or not.

  // Define a leaf rendering function that is memoized with `useCallback`.
  const renderLeaf = useCallback(props => {
    return <Leaf {...props} />
  }, [])


const Leaf = props => {
  return (
    <span
      {...props.attributes}
      style={{ fontWeight: props.leaf.bold ? 'bold' : 'normal' }}
    >
      {props.children}
    </span>
  )
}

If the leaf is bold or not is set with Transforms, but I assume we could also pass this information in the initial data or just compute it by checking if the leaf timestamp intersects with a highlight start and end timestamp.

 Transforms.setNodes(
                editor,
                { bold: true },
                { match: n => Text.isText(n), split: true }
              )

I've tried to implement this in the SlateTranscriptEditor as following:

  const TimedTextElement = (props) => {
  ...
  return (...
       {/* Unchanged until here */}
       {/* Moved this up from your original `renderLeaf` function */}
        <Grid item xs={12} sm={12} md={12} lg={textLg} xl={textXl} className={'p-b-1 mx-auto'}>
          <span
            onDoubleClick={handleTimedTextClick}
            className={classNames('timecode', 'text')}
            data-start={props.children.props.node.start}
            data-previous-timings={props.children.props.node.previousTimings}
            {...props.attributes}
          >
            {props.children}
          </span>
        </Grid>
      </Grid>
    );
  };

  const DefaultElement = (props) => {
    return <p {...props.attributes}>{props.children}</p>;
  };

  const renderElement = (props) => {
    switch (props.element.type) {
      case 'timedText':
        return <TimedTextElement {...props} />;
      default:
        return <DefaultElement {...props} />;
    }
  };

  const HIGHLIGHTS = [{ start: 1, end: 20 }]; // Static dummy data with one single highlight

  const Leaf = ({ attributes, children, leaf }) => {
    // Check if the leaf intersects with a highlight
    const start = children.props.parent.start;
    const isInQuote = HIGHLIGHTS.filter((quote) => quote.start < start && quote.end > start).length > 0;

    return (
      //The "inQuote" class could have a different color background to make the quote stand out.
      <span className={classNames({ inQuote: isInQuote })} {...attributes}>
        {children}
      </span>
    );
  };

  const renderLeaf = (props) => {
    return <Leaf {...props} />;
  };

  /*Notes:
  - I've removed `useCallback` for the moment to make debugging easier
  */

I hoped TimedTextElement would render the rows in the grid, and then hand over the rendering of the leaves (i.e. words) to renderLeaf. What is happening instead, is that renderLeaf receives only TimedTextElement objects and nothing with a smaller granularity. It's as if the TimeTextElement was the smallest granularity supported. Do you have an idea why this is the case? I wonder if I don't have to change the data model we pass to Slate to support one more level of nesting.

[allisonking]

Hi @BasilPH , any chance you've tried out decorations for this use case? Like what is done in this slate example: https://github.com/ianstormtaylor/slate/blob/main/site/examples/search-highlighting.tsx

From what I understand, you write a decorate function which returns a list of ranges where your criteria (i.e. words that are part of a highlight) is met. Then renderLeaf looks for the intersection between what it is rendering and the range that your decorate function returned.

I used a decorate function to highlight words as they are spoken, though I'm not sure about its performance. I think for just a list of highlights it may work okay though. Also curious how you gave each word an id when you implemented highlight words as they're spoken 😄