Problem with Google crawling

[KevinHansen23]

Hi @Rayn93,

we have the following problem: Google crawls faulty links, all of which originate from FactFinder templates:

We have already added the following rules to robots.txt:

Disallow: */{{
Disallow: /factfinder/result?
Disallow: /fact-finder/proxy

Nevertheless, the URLs are still being crawled. With the exception of record.deeplink and attributes.deeplink, all URLs originate from Suggest, as far as we can tell. The behaviour is described here in the FAQ (How to avoid crawlers indexing unresolved URLs which results in a 404?). However, the data-redirect attributes in the suggest are not parsed.

Things get even stranger with record.Deeplink and attributes.Deeplink. The former only appears in an Slider template (record-list-slider.html.twig), which we do not use at all, and the latter does not even appear anywhere in the source code.

Can you please add the data-redirect functionality to the Suggest and check where the two deeplink attributes come from?

Kind regards
Kevin

[Rayn93]

Hi @KevinHansen23

We have already confirmed the problem with data-redirect in suggest component. We will try to add this functionality to suggest.

When it comes to record.deepplink, I have already located and I will fix it soon. It is a bit worse with attributes.deepplink but I will also try to look for where such a link can come from.

[KevinHansen23]

Hi @Rayn93,

Thank you for your, as always, quick feedback. It would be great if you could also integrate the fixes into the SDK version for Shopware 6.6.

Kind regards
Kevin

[Rayn93]

Hi @KevinHansen23

I fixed a problem with data-binding for record-list-slider.html.twig. It is already released in v6.5: https://github.com/FACT-Finder-Web-Components/shopware6-plugin/releases/tag/v6.5.0

The data-redirect doesn't work in ff-suggest. Instead you should use deferred attribute resolving: https://web-components.fact-finder.de/api/4.x/DeferredAttributeResolving

[KevinHansen23]

Hi @Rayn93,

fixed a problem with data-binding for record-list-slider.html.twig. It is already released in v6.5: https://github.com/FACT-Finder-Web-Components/shopware6-plugin/releases/tag/v6.5.0

That is excellent. Do you have any idea where the attributes.deeplink could come from?

The data-redirect doesn't work in ff-suggest. Instead you should use deferred attribute resolving: https://web-components.fact-finder.de/api/4.x/DeferredAttributeResolving

Thank you, we will try that out

Kind regards
Kevin

[Rayn93]

@KevinHansen23

Unfortunately I couldn't find the source of the probleme for attributes.deeplink in the SDK. I spoke with WebComponents developer and he said it came from the ff-suggest component. Did you implement any changes on your site for this component?

[KevinHansen23]

Hi @Rayn93,

yes, we overrode suggest.html.twig because the standard integration did not work. Here is an example of our category container:

<div data-container="category" class="text-truncate">
    <h3 class="h5 container-caption pb-2">{{ 'ff.suggest.Categories'|trans|striptags }}</h3>
    <ff-suggest-item class="p-1" type="category">
        <a href="{{ '{{attributes.Url}}' }}" class="d-flex justify-content-between">
            <span class="ff-category">{{ '{{{name}}}' }}</span>
            <span class="badge text-bg-primary text-white" title="{{ '{{$dec hitCount}}' }} {{ "ff.suggest.categoryHitCount"|trans|sw_sanitize }}">{{ '{{$dec hitCount}}' }}</span>
        </a>
    </ff-suggest-item>
</div>

However, there is no {{attributes.deeplink}} in the entire file.

Kind regards
Kevin

[KevinHansen23]

Hi @Rayn93

we have updated to version 6.5.0 and switched to data-defer-href in suggest.html.twig, but unfortunately without success. Google continues to have loads of 404 errors:

This is how the current integration of the categories in suggest.html.twig looks like:

<div data-container="category" class="text-truncate">
    <h3 class="h5 container-caption pb-2">{{ 'ff.suggest.Categories'|trans|striptags }}</h3>
    <ff-suggest-item class="p-1" type="category" data-ffw-defer="href">
        <a data-defer-href="{{ '{{attributes.Url}}' }}" class="d-flex justify-content-between">
            <span class="ff-category">{{ '{{{name}}}' }}</span>
            <span class="badge text-bg-primary text-white" title="{{ '{{$dec hitCount}}' }} {{ "ff.suggest.categoryHitCount"|trans|sw_sanitize }}">{{ '{{$dec hitCount}}' }}</span>
        </a>
    </ff-suggest-item>
</div>

What also seems very strange is that the URL contains the product ID, everything after that is cut off, and then the unresolved variable appears.

Kind regards
Kevin

[Rayn93]

Hi @KevinHansen23
Unfortunately, I have no idea what else could be wrong. The issue is more related to the WebC library, so when the lead WebC developer returns from vacation, I'll ask him to check it again.

[KevinHansen23]

Hi @Rayn93,

thanks for the update. We will wait for feedback from the WebComponents team.

[Rayn93]

Hi @KevinHansen23 Thanks for your patience.
@omikron-fp from Web Components developer team has reviewed your banemo.de website (home page, search result page, and category page) and everything looks correct there.

Is Google crawling still an issue? Perhaps it's due to caching and Google still accessing the page before the changes?

[KevinHansen23]

Hi @Rayn93,

sorry for the late reply. Currently, it looks as though the errors are slowly decreasing. However, new 404 errors are still appearing.

We will continue to monitor the situation over the next few weeks and then provide another update.

[KevinHansen23]

Hello @Rayn93

I think we've finally found the problem now. When we disable JavaScript in Firefox, the Suggest-Code contains an unrendered handelbars.js template. The variables it contains correspond 100% to the strange links that Google is trying to index:

Kind regards
Kevin

[Rayn93]

Hi @KevinHansen23
Thanks for the information. It's great that you were able to determine the cause.
Unfortunately, we can't really do anything about it because WebComponents is a JS library and requires JavaScript to function properly.

[KevinHansen23]

Hy @Rayn93,

I didn't mean to suggest making webComponents executable without JavaScript.
The question is, why is the template with the variables stored in the DOM in this form in the first place?

After the template has been parsed by the webComponents, it looks like this:

So why are the variables and links first written into the template (which causes the problem) only to be removed again immediately?
If the cleaned-up version were in the DOM, there would be no problem at all.

Kind regards
Kevin

[Rayn93]

@KevinHansen23

I don't really understand what you mean. I just checked your site, and the data-ffw-defer attribute in ff-suggest-item correctly adds the href attribute:

This is an example from the screenshot you sent this morning.

Where exactly do you have a case where you add a link or variable via data binding and it gets deleted?