Skip to content

add crawler strict judge option #8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kkyouhei wants to merge 1 commit into
base: master
Choose a base branch
from
Open

add crawler strict judge option #8

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion lib/first_click_free.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ class << self

require 'yaml'

attr_accessor :test_mode, :permitted_paths, :free_clicks
attr_accessor :test_mode, :permitted_paths, :free_clicks, :crawler_bot_strict_if

def root
File.expand_path(File.join(File.dirname(__FILE__), '..'))
Expand All @@ -30,6 +30,10 @@ def free_clicks
def test_mode
@test_mode || false
end

def crawler_bot_strict_if
@crawler_bot_strict_if || false
end
end

class Railtie < Rails::Railtie
Expand Down
2 changes: 1 addition & 1 deletion lib/first_click_free/concerns/controller.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ def user_for_first_click_free
# if this is the first click recorded for this session.
def record_or_reject_first_click_free!
# Always allow requests from Googlebot
return true if googlebot?
return true if googlebot? FirstClickFree.crawler_bot_strict_if

# Always allow requests from authenticated users
return true if user_for_first_click_free
Expand Down
5 changes: 3 additions & 2 deletions lib/first_click_free/helpers/google.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,9 @@ module Google
#
# Returns true if a Googlebot has been identified, false if not.
def googlebot?(use_dns_lookup = true)
user_agent = request.user_agent || ""
(FirstClickFree.test_mode && params.key?(:googlebot)) ||\
(request.user_agent == "Googlebot" && (use_dns_lookup ? verify_googlebot_domain : true))
(user_agent.include?("Googlebot") && (use_dns_lookup ? verify_googlebot_domain : true))
end

private
Expand All @@ -38,4 +39,4 @@ def verify_googlebot_domain

end
end
end
end