Error after 10 Channel Queries

[usu-access]

We have identified an issue with larger YouTube videos (around 500-ish channels). As an example, the issue can be seen with either of these channels:

• Jon M. Huntsman School of Business (Channel ID: UCHvDm56-nrkZEOUWEHrpYuw).
• Utah State University Extension (Channel ID: UC3UzpQYVg7K7FfLXCi19Gdw).

For the Huntsman channel for example, there are actually 639 videos in the channel, but YTCA returns 606, 113 of which are duplicates. To troubleshoot I went through the Channel Queries for that channel (14) and noticed that the 11th query returns the following:

{
"kind": "youtube#searchListResponse",
"etag": "AXfSerh4fAv4w-TxjTjN-PEIcSw",
"prevPageToken": "CPQDEAE",
"regionCode": "US",
"pageInfo": {
"totalResults": 665,
"resultsPerPage": 0
},
"items": []
}

For some reason the "nextPageToken" is missing so no videos are returned and then with Query #12 it starts over with the same videos queried in Query #1 leading to the duplicates.

The issue can be duplicated in other channels. We will try to look into it to fix it if we can, but if anyone has any tips on where we can start looking or ideas on what might be causing the issue, we would appreciate any help. Thanks so much for this incredible tool - it is a tremendous resource!

[usu-access]

It looks like this is a limitation of the YouTube API - more information in this StackOverflow discussion - How can get all results from Youtube API (search API) response.

The workaround in that response also works with YTCA - you can use the YTCA "Published after" and "Published before" fields to break up channels with more than 500 videos in them. As long as each period has less than 500 videos it should work without any problems.

[usu-access]

Unfortunately, the "Published after" and "Published before" solution doesn't seem to work - it still goes through all 500+ videos. @terrill can you tell me if the API call is using the publishedAfter and publishedBefore parameters or does the code filter out those dates once all results are returned?

[usu-access]

So we finally have a little bit more clarity and a bit of a workaround on this. It looks like the "Published after" and "Published before" dates in the tool do not limit what information is requested from the API. However, there are parameters that can be added to the channel query to pull entries only from a specific time period which gets around the 500 video limit and allows you to pull videos in batches.

To do so, you just need to add something like:

&publishedBefore=2019-01-01T00%3A00%3A00%2B00%3A00
or
&publishedAfter=2016-01-01T00%3A00%3A00%2B00%3A00

to the channel query with the date in RFC 3339/Atom format.

Our temporary workaround is to add something like the below to the search YouTube query:

• $request .= '&publishedBefore=2019-01-01T00%3A00%3A00%2B00%3A00&publishedAfter=2016-01-01T00%3A00%3A00%2B00%3A00';
• Or $request .= '&publishedAfter=2019-01-01T00%3A00%3A00%2B00%3A00';

Right after:
$request .= '&channelId='.$id;

This will return just the videos for that time period so you can keep the query under 500 videos and get a more accurate report. We hope to build this into the tool using the dates collected from the UI eventually, but this works for those larger channels for now.