Skip to content

Pipeline vfs raw parsing and DuckDB appends #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
fanyang89 wants to merge 1 commit into from
Closed

Pipeline vfs raw parsing and DuckDB appends #3

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 39 additions & 3 deletions cmd_vfs_raw.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -300,9 +300,45 @@ var vfsRawCmd = &cli.Command{
r = f
}

return jsonParseThenAppend(r, func(e *vfsEvent) error {
return appender.AppendRow(e.Timestamp, e.Probe, e.Tid, e.ReturnValue,
e.Path, e.Inode, e.Offset, e.Length)
events := make(chan *vfsEvent, 1024)
writeErrCh := make(chan error, 1)
done := make(chan struct{})

go func() {
defer close(done)
for e := range events {
if err := appender.AppendRow(e.Timestamp, e.Probe, e.Tid, e.ReturnValue,
e.Path, e.Inode, e.Offset, e.Length); err != nil {
writeErrCh <- err
return
}
}
}()

parseErr := jsonParseThenAppend(r, func(e *vfsEvent) error {
select {
case events <- e:
return nil
case err := <-writeErrCh:
Comment on lines +318 to +322
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot Feb 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Stop parsing immediately when writer errors

The select can still choose events <- e even when writeErrCh is already readable, because Go selects randomly among ready cases. If the writer goroutine fails, the parser may enqueue up to the remaining buffer capacity (1024) of additional events before it eventually notices the error, and those events will be dropped because the writer has exited. That creates silent data loss in the exact scenario the error reporting is meant to handle. Consider prioritizing the error path (e.g., non-blocking check of writeErrCh before enqueue) so parsing halts as soon as the writer fails.

Useful? React with 👍 / 👎.

if err != nil {
return err
}
return errors.New("writer stopped")
}
})

close(events)
<-done

if parseErr != nil {
return parseErr
}

select {
case err := <-writeErrCh:
return err
default:
return nil
}
},
}
Loading