Thinking Cobra with Scratchpad and CoT#32
Open
AndrasFerenczy wants to merge 42 commits intoOpenHelix-Team:mainfrom
Open
Thinking Cobra with Scratchpad and CoT#32AndrasFerenczy wants to merge 42 commits intoOpenHelix-Team:mainfrom
AndrasFerenczy wants to merge 42 commits intoOpenHelix-Team:mainfrom
Conversation
…nd save BLEU scores with timestamps. Add new entries to .gitignore for output files.
Scratchpad Reasoning + Benchmark
…ctionality for clearing GPU memory and saving BLEU scores has been integrated into the main workflow.
Update notebook: Clear GPU RAM and save image of results
This commit introduces a comprehensive set of files and scripts for fine-tuning the Cobra VLM on the LLaVA-CoT-100k dataset. Key additions include: - **Dataset Preparation**: A script to download, validate, and prepare the dataset for training. - **Custom Dataset Loader**: A new loader that supports JSONL format and integrates with existing training infrastructure. - **Fine-Tuning Script**: A dedicated script for fine-tuning the model using the prepared dataset. - **Documentation**: Detailed guides and summaries for setup and usage. These changes enhance the model's reasoning capabilities by leveraging structured reasoning annotations in the dataset.
X Merge branch 'main' of https://github.com/AndrasFerenczy/thinking_cobra
This commit introduces the foundational structure for the Cobra Evaluation System, including: - **Main Module**: The entry point for running evaluations with command-line argument parsing. - **Configuration Management**: A dedicated module for handling CLI arguments and settings. - **Registry System**: A registry for managing generators and metrics, allowing for extensibility. - **Generators**: Implementations for baseline, scratchpad, and external generation methods. - **Metrics**: Initial implementations for BLEU and BERTScore metrics. - **Utilities**: Functions for GPU management, JSON I/O, and visualization of results. These changes establish a modular framework for evaluating visual language models with various inference strategies and metrics, enhancing the system's extensibility and usability.
This commit introduces significant improvements to the evaluation process, including: - **Method Comparison**: Added functionality to run and compare results from multiple methods (baseline and scratchpad) within the same evaluation session. - **Visualization Enhancements**: Implemented a new comparison visualization that displays results side-by-side for easier analysis of generated captions and metrics. - **BERTScore Metric Updates**: Enhanced the BERTScore metric to store per-sample scores, allowing for detailed performance analysis. - **Code Refactoring**: Cleaned up the main evaluation logic for better readability and maintainability. These changes improve the usability and analytical capabilities of the Cobra Evaluation System, facilitating more comprehensive evaluations of visual language models.
This commit introduces several improvements to the evaluation process, including: - **Dynamic Output Directories**: Results are now saved in timestamped directories for better organization, allowing users to easily manage multiple runs. - **Comparison Statistics**: Added functionality to compute and save comparison statistics between baseline and scratchpad methods, including win rates and metric differences. - **Visualization Updates**: Enhanced the comparison visualization to include detailed metrics and reasoning traces, improving the clarity of results. These changes enhance the usability and analytical capabilities of the Cobra Evaluation System, facilitating more comprehensive evaluations of visual language models.
This commit modifies the .gitignore file to ensure that shell scripts are ignored and removes the output.png file, which is no longer needed. These changes help streamline the project by keeping unnecessary files out of version control.
This commit modifies the .gitignore file to include the __pycache__ directory, ensuring that Python bytecode files are not tracked in version control. This helps maintain a cleaner project structure by excluding unnecessary files.
This commit introduces several new files and enhancements to the Cobra Evaluation System, including: - **New Scripts**: Added `analyze_significance.py`, `compare_scratchpad_passes.py`, and `visualize_scratchpad_passes.py` for analyzing and visualizing scratchpad performance across multiple passes. - **Checkpointing Guide**: Introduced `CHECKPOINTING_GUIDE.md` to document the new automatic checkpointing feature for long-running evaluations. - **Improved Documentation**: Added `SCRATCHPAD_COMPARE_MODE.md`, `SCRATCHPAD_DEGRADATION_ANALYSIS.md`, and `SCRATCHPAD_IMPROVEMENTS.md` to provide insights into scratchpad methods and their performance. - **New Data Files**: Included various JSON and PNG files for results and visualizations from recent evaluations. These changes enhance the analytical capabilities and usability of the evaluation system, facilitating better understanding and comparison of different methods in visual language models.
This commit introduces support for external model API clients, allowing users to run evaluations using models such as GPT-5, Gemini, Claude, and Llama. Key changes include: - **New Inference Methods**: Added options for external models in the evaluation workflow. - **API Key Management**: Introduced command-line arguments for specifying API keys and model configurations. - **Conditional Model Loading**: Updated the main evaluation logic to skip local model loading when using external models. - **Checkpointing Improvements**: Enhanced checkpointing functionality to support overwriting the latest checkpoint file. These updates significantly expand the evaluation options and flexibility of the Cobra Evaluation System, facilitating integration with various external AI models.
…ith 10,100,1000,10000 examples
…ing scripts as well as my install requirements script.
…r_it MMStar and accuracy eval added (& benchmarked for 1000 images on both COCO and MMStar)
… into feature/qlora-finetune
Feature/qlora finetune
Collaborator
|
Hi @AndrasFerenczy, @philosophercode and @MatteoPerona , Thank you so much for submitting this PR to this project! The effort you put into integrating LoRA adapters into the training pipeline and experimenting on reasoning datasets demonstrates a promising direction for extending the model’s capabilities. However, after carefully reviewing your contribution, I’ve decided not to merge this feature directly into the main branch at this time. The main reason is that our priority is to keep the code and experiments in the main repository as clear and simple as possible, ensuring we can reliably reproduce the core results from the paper. That said, I genuinely believe your work is highly valuable and deserves visibility. Rather than merging this into the current repository, I recommend you maintain these excellent improvements as a standalone fork or a new, independent repository. Once again, thank you sincerely for your insightful contribution. I look forward to seeing your work grow into an exciting project of its own! All the best, |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.