robots.txt Plugin

[kwaa]

Description

I didn't add the comment in the Rule type for now due to the complexity.

Related Issues

closed #543

Check List

• Have you read the
  CODE OF CONDUCT
• Have you read the document
  CONTRIBUTING
  • One pull request per feature. If you want to do more than one thing,
    send multiple pull request.
  • Write tests.
  • Run deno fmt to fix the code format before commit.
  • Document any change in the CHANGELOG.md.

[oscarotero]

The plugin looks great, thank you!
I just have a couple of questions:

• Maybe the plugin name should be robots_txt, instead of simply robots, because I think is clearer about what the plugins does. What do you think?
• I like the simplicity of the configuration, with an array of allow and disallow. If I only want to allow Google and Bing, the configuration is:

  site.use(robots({
    allow: ["Googlebot", "Bingbot"],
  }));

  But this doesn't prevent other bots to scan the site because I'm not explicitly saying that they shall not do it. I don't know if this is the expected behavior. I guess the proper way to do it is:

  site.use(robots({
   allow: ["Googlebot", "Bingbot"],
   disallow: ["*"],
  }));

  Is that correct?

[kwaa]

• Maybe the plugin name should be robots_txt, instead of simply robots, because I think is clearer about what the plugins does. What do you think?

I don't think there is any problem with the file name, but the variable name will be robotsTxt(), which feels a bit ugly.

• I like the simplicity of the configuration, with an array of allow and disallow. If I only want to allow Google and Bing, the configuration is:

  site.use(robots({
    allow: ["Googlebot", "Bingbot"],
  }));

  But this doesn't prevent other bots to scan the site because I'm not explicitly saying that they shall not do it. I don't know if this is the expected behavior. I guess the proper way to do it is:

  site.use(robots({
   allow: ["Googlebot", "Bingbot"],
   disallow: ["*"],
  }));

  Is that correct?

Yes.

[oscarotero]

Thank you!