Skip to content

Avoid frequent HDFS writes in documents similarity #426

@marekhorst

Description

@marekhorst

After inspecting several PIG scripts it seems there are plenty of redundant HDFS writes, followed by instant reads which could be optimized.

Apparently this was used for debugging purposes so it will be reasonable to handle debug flag when possible and skip redundant writes/read when flag is not set.

This could significantly improve overall performance.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions