Google indexes Template, Helpcenter, and Notion Landing Pages on my Domain

[marcklingen]

Hi @stephenou,

thanks again for your ongoing development and support of this project! While #19 #18 aim to fix this issue, I just would like to reiterate why it would be super important to exclude all pages that are not whitelisted with a 404 or no-index header.

Yesterday, I saw that Google indexed 374 pages on my domain labeled as Indexed, not submitted in sitemap. Here's a screenshot with examples:

While I am not sure about the legal implications, it would be for sure nice to only have own pages in the index.

I am happy to contribute to the solution, let me know how I can help best.

[stephenou]

Hey @marcklingen! Yep, I agree we should fix this. The solution I came up with is to 301-redirect unlisted pages back to the homepage.

See 7f273ed for what changes you need to make in your script.

Do you mind helping me test it out before I announce it widely? Thanks!

[marcklingen]

Thanks @stephenou, the redirect works for me. I'd suggest to also add X-Robots-Tag: noindex to the header of the response.

[marcklingen]

Just found an exception to the rule in the screenshot. While this solution solves most of the problem, it does not address pages which do not have the characteristic page id, e.g.: /tools-and-craft/01-andy-hertzfeld, /pricing

[ThallyssonKlein]

@marcklingen How did you manage to index the domain on Google? Mine does not appear ...

[marcklingen]

@ThallyssonKlein If you domain is not automatically indexed, you can add the /sitemap.xml in the Search Console.

[ThallyssonKlein]

@marcklingen Where does this sitemap.xml file come from?

[marcklingen]

@ThallyssonKlein The sitemap is generated by the worker, you can find the line here:

fruitionsite/worker.js

Line 91 in 7f273ed

if (url.pathname === "/sitemap.xml") {

[lasharor]

Had the same problem. Changed the code as per 7f273ed. Pages indexed by google now all go back to the homepage.

[marcklingen]

@lasharor That does not work for pages with a nice slug such as /pricing

[ThallyssonKlein]

@marcklingen How long can indexing take?

[marcklingen]

@ThallyssonKlein Usually it takes a couple of days but less than a week.

[ThallyssonKlein]

@marcklingen I have been notified that the pages are not being read correctly

[stephenou]

Just found an exception to the rule in the screenshot. While this solution solves most of the problem, it does not address pages which do not have the characteristic page id, e.g.: /tools-and-craft/01-andy-hertzfeld, /pricing

Hey Marc, yeah unfortunately that's true. One solution is to define a denylist of URLs that Notion uses for marketing, but it's hard to keep it updated when Notion adds new pages. Another solution is to define an allowlist of URLs that your site can visit, but it's also hard to keep it updated when you add new pages.

[vlafriday]

Hi @stephenou,

thanks again for your ongoing development and support of this project! While #19 #18 aim to fix this issue, I just would like to reiterate why it would be super important to exclude all pages that are not whitelisted with a 404 or no-index header.

Yesterday, I saw that Google indexed 374 pages on my domain labeled as Indexed, not submitted in sitemap. Here's a screenshot with examples:

While I am not sure about the legal implications, it would be for sure nice to only have own pages in the index.

I am happy to contribute to the solution, let me know how I can help best.

Hi, @marcklingen, I know it's been many years, but maybe you know the solution? Now I get the same problem with indexing. Although sitemap.xml has been added to Google Console.

I see your website (https://marcklingen.com/) it works without problems. How did you do that?
Can you share and help?

@stephenou