Scanning multiple cards at same time, Ala binder page?

[kapilt]

cool project, was just curious about scanning in my collection with it, but i have quite a few in binders, and level of effort / considerations on scanning a binder page at a time (9 cards). TIA

[HanClinto]

Hi @kapilt -- thank you for the feedback

You should be able to scan directly out of a binder, but the neural network that identifies where in an image a card is located is currently limited to only try and pick out the single card that is the primary focus of the image. So while you can scan out of a binder, you'll (currently) still need to pass over the cards one-at-a-time.

You can see an example of scanning out of a binder here:
https://www.youtube.com/watch?v=80NDsxKQALs&t=122s

You'll notice that the identifier gets a bit confused when there is more than one card kinda' straddling the view, but when one makes its way more towards the center of the camera view, then it settles down and is (usually) able to pick out the card.

Is that good enough for your purposes, or would you really prefer to have a full "9 cards at a time"?

I've developed this sort of functionality for some of my previous scanners (see this video here of ScryGlass identifying 16 cards at once):

https://www.youtube.com/watch?v=-C_8czfRDrQ

While it was a cool "party trick" -- what I feel like I saw in practice was that most users didn't use this functionality, and it added a fair bit of complexity in the code for very little return.

That said, I can think of many instances where a scanner would benefit from being able to identify many cards at once (such as live-stream coverage and wanting to identify many cards on the field at a single time), so I'm very willing to reconsider this problem.

In the meantime, what would help me is if you would be willing to provide me a dataset of the sorts of multi-card images that you would expect to be able to identify (bonus points if you include the Scryfall IDs of each card in each image, but if that's too much work I can annotate them myself) -- the more images the better (uploading a video is the best way to provide a large number of images all at once). I.E., record a video of you flipping through your binder, and hold your phone in a manner that you would expect to be able to scan the cards -- but instead of running my web scanner, just record a video. Upload that video to YouTube, send a link to the video here, and I'll see what I can do about adding such functionality in my backlog.

[HanClinto]

Generated by Copilot: Cross-posting the TODO action items here so @kapilt and other volunteers can see the current next step.

I added a TODO section for this feature: multi-card and binder-page scanning. Current behavior: the scanner can be used over a binder, but Cornelius currently tries to locate the single primary card in view. Multi-card detection would require broader detector work and additional UX decisions, so the next important piece is dataset collection before implementation.

Action items:

• Next step: collect a realistic multi-card dataset. @kapilt (or other volunteers) has the next responsibility here: provide sample binder-page captures that show the desired scanning workflow, preferably as a phone video flipping through binder pages. Other volunteers can help by contributing similar videos or still images from binders, play fields, and other multi-card layouts.
• For each image or video segment, collect card IDs when practical. Scryfall IDs are preferred for MTG cards, but unlabeled media is still useful and can be annotated later.
• Review the collected dataset and decide whether the first supported mode should be binder pages, play-field/live-stream recognition, or a smaller multi-card detector path.
• Prototype multi-card detection after enough representative data exists, then evaluate complexity, performance, and scanner UI changes before adding it to the product backlog.

I also tagged this issue with volunteers wanted to make it clear that dataset collection is the current volunteer-friendly path forward.

[eheiden]

Hi @HanClinto! I left a few comments on your youtube channel video for CollectorVision (twice!) and both times the comment got deleted - possibly because I had linked one my livestreams to show you what my setup / needs were.

I livestream a card game (Netrunner) on my channel. Here's a slightly outdated demo of my current card recognition solution, and a previous home grown project trying to do something similar to your solution with python and much less success.

I also need multi-card support - in addition to some more features, which I could probably build around if I use CollectorVision at the core.

Here's an example of what a typical livestream for us looks like: https://youtu.be/J8qLRFgQ_Mg?t=3532

The ultimate goal for me is to have a local program running that I can run that has a region defined over each play mat, and whenever a new card is played to the play mat it is detected, pushed to a queue which is sent over to OBS (streaming software) to be displayed on that player's side of the screen for ~8-10 seconds, and then the next card played or moved on that player's side of the screen gets moved.

I have been building around your code for the past few days and I currently have something that does this, but it has a lot of limitations - if cards overlap or touch each other the program won't recognize them as cards to send to CollectorVision. If one of the players is holding a card (a thumb is over a card anywhere) it won't work. It has a bunch of other issues.

I'm downloading image files from https://netrunnerdb.com/ - the "standard format" card pool has ~650 unique cards in it which CollectorVision seems to handle very well. I also have about 100 community alt-arts I've added to the catalogue as well, which allows us to play with our Alt Arts.

I'd be happy to share my code with you to get your thoughts on it. I'm using a fair amount of help from ChatGPT in the process but I'm trying to keep it from making absolute spaghetti code. The python script takes a fair chunk of my CPU right now when I'm running it which I'm slightly concerned might be too resource intensive for my PC when I'm also livestreaming. I have yet to do a test on it.

Other features my particular problem space needs to handle:

• recognizing and ignoring facedown cards (cards in Netrunner are often played faced down).
• dealing with cards overlapping
• dealing with any orientation (not just vertical or upside down) - if this is too strenuous than any of the 4 cardinal rotations would be acceptable as 45 degree rotations are rare (but cards do get "tapped" 45 degrees sometimes.
• Tracking cards that have been recognized/displayed already and haven't moved position to be ignored (not take up CPU cycles constantly rescanning and re-rotating).
• when we move our hands over cards I want the program not to re-recognize them and send them to the stream feed over and over again if they haven't moved.

I've got a pretty good solution to the OBS connection side which is obviously not the point of this post, but having native support for multiple cards and potentially some generic tools for tracking them and storing metadata with each track would be incredibly helpful!

If you'd like to talk more about this project my email is neonstaticpod@gmail.com and I'd be happy to continue the conversation there or over a discord chat some time. I'm really hoping to make something stable and good that my community can use in general.

Thanks!
Eric

[HanClinto]

Hi @eheiden -- apologies for my late reply.

I'm guessing you're correct about the YouTube comments getting deleted for spam links -- I never saw them come across my feed, and certainly didn't delete them myself.

Stream overlays is a really good use-case for multi-card scanning. In those cases, we don't care as much about edition-vs-edition accuracy (does it really matter if we identify a Sol Ring as Commander 2017 vs. Commander 2018?) -- but we do need to identify (and track) multiple cards across multiple frames.

If one is dealing with a limited corpus of cards (750-ish cards), then I almost wonder if you would be able to use SIFT / RootSIFT descriptors and FindHomography() to match them -- this would probably be pushing at the limits of what you can do in real time, but this is what I use to help me annotate my datasets -- it's a fantastic technique so long as you generally know which cards you're likely to find in your images.

That said, by default this technique is intended to only find a single instance of a card in an image at a time. I'm sure there's a way to cluster the Lowe ratio groups so that local maxima don't find against each other (but instead can find clusters of descriptors and match those clusters to separate cards in an image), but I haven't messed around with that enough to ever get a working solution.

So as I see it, there are two or three ways that one could do multi-card matching for stream overlays like you're talking about:

• Skip CollectorVision entirely, and use traditional SIFT / RootSIFT descriptors to match multiple cards in an image. This option scales very poorly (linearly) as you increase the number of cards in a dataset, and I personally find it topping out around 300-500 cards or so. 750 cards might be pushing it, but you may be able to find a way to make this work.

• The current corner detector (Cornelius) only outputs a single set of card corners at a time. We could build a new corner detector that outputs corners for multiple cards at a time (similar to something like YOLO). Would need an annotated dataset, probably created with something like the above, and come up with a novel architecture. Not sure how I feel about this one.

• Could possibly page Cornelius over the image in a grid search sort of thing to try and find multiple regions with cards of interest. Not sure I like this one.

• Start by detecting aabb's (axis-aligned bounding-boxes) for the various cards in an image (similar to what YOLO outputs), and then either crop those again to align the corners (essentially to get rid of the diagonals), or else get a version of the embedding model (Milo) that is more comfortable with cards being diagonally aligned in the bounding box. This is probably the most viable option, and I'll mentally queue it up to run some experiments with this next time I get on a set of runs to train embedding models.

I suspect the last option is something closest to what you're doing (given that your system works better when you align the card to be "squared-up"), but I think with a little help from the embedding model, you could get something that works better for your needs.