Setup default settings when we run a spider

[botzill]

I was wondering how easy would be to configure default settings to show when we run a spider?

Thx.

[botzill]

Yes, for sure not a bug :), just asking;.

[my8100]

Do you mean customizing the default value of the textarea below?

[botzill]

I mean

Here, set there some custom one which we can define in settings.py, like make it configurable which one to show here. It's harder to setup in that textarea but here would be nice. Plus, some dropdown as well to be possible.

Thx.

[my8100]

For now, you can manually modify the code below like this.
I would consider making it configurable in the configuration file.
Thanks for your suggestion.

 self.kwargs.setdefault('USER_AGENT', 'Chrome')  # Chrome|iPhone|iPad|Android 
 self.kwargs.setdefault('ROBOTSTXT_OBEY', 'False') 
 self.kwargs.setdefault('COOKIES_ENABLED', 'False') 
 self.kwargs.setdefault('CONCURRENT_REQUESTS', '16') 
 self.kwargs.setdefault('DOWNLOAD_DELAY', '0') 
 _additional = "-d setting=CLOSESPIDER_TIMEOUT=60\r\n-d setting=CLOSESPIDER_PAGECOUNT=10\r\n-d arg1=val1" 
 self.kwargs.setdefault('additional', _additional) 

scrapydweb/scrapydweb/operations/schedule.py

Lines 175 to 181 in 01d0a4f

	self.kwargs.setdefault('USER_AGENT', '') # Chrome|iPhone|iPad|Android
	self.kwargs.setdefault('ROBOTSTXT_OBEY', '')
	self.kwargs.setdefault('COOKIES_ENABLED', '')
	self.kwargs.setdefault('CONCURRENT_REQUESTS', '')
	self.kwargs.setdefault('DOWNLOAD_DELAY', '')
	_additional = "-d setting=CLOSESPIDER_TIMEOUT=60\r\n-d setting=CLOSESPIDER_PAGECOUNT=10\r\n-d arg1=val1"
	self.kwargs.setdefault('additional', _additional)

[botzill]

I see, thx @my8100! Would be really useful to have this configurable.

[my8100]

@botzill

1. pip install -U git+https://github.com/my8100/scrapydweb.git
2. Update the existing config file with the options below:
   

   scrapydweb/scrapydweb/default_settings.py

   Lines 121 to 151 in 30b39b7

   	############################## Run Spider #####################################
   	# The default is False, set it to True to automatically
   	# expand the 'settings & arguments' section in the Run Spider page.
   	SCHEDULE_EXPAND_SETTINGS_ARGUMENTS = False
   	# The default is 'Mozilla/5.0', set it a non-empty string to customize the default value of `custom`
   	# in the drop-down list of `USER_AGENT`.
   	SCHEDULE_CUSTOM_USER_AGENT = 'Mozilla/5.0'
   	# The default is None, set it to any value of ['custom', 'Chrome', 'iPhone', 'iPad', 'Android']
   	# to customize the default value of `USER_AGENT`.
   	SCHEDULE_USER_AGENT = None
   	# The default is None, set it to True or False to customize the default value of `ROBOTSTXT_OBEY`.
   	SCHEDULE_ROBOTSTXT_OBEY = None
   	# The default is None, set it to True or False to customize the default value of `COOKIES_ENABLED`.
   	SCHEDULE_COOKIES_ENABLED = None
   	# The default is None, set it to a non-negative integer to customize the default value of `CONCURRENT_REQUESTS`.
   	SCHEDULE_CONCURRENT_REQUESTS = None
   	# The default is None, set it to a non-negative number to customize the default value of `DOWNLOAD_DELAY`.
   	SCHEDULE_DOWNLOAD_DELAY = None
   	# The default is "-d setting=CLOSESPIDER_TIMEOUT=60\r\n-d setting=CLOSESPIDER_PAGECOUNT=10\r\n-d arg1=val1",
   	# set it to '' or any non-empty string to customize the default value of `additional`.
   	# Use '\r\n' as the line separator.
   	SCHEDULE_ADDITIONAL = "-d setting=CLOSESPIDER_TIMEOUT=60\r\n-d setting=CLOSESPIDER_PAGECOUNT=10\r\n-d arg1=val1"

[botzill]

Thx @my8100, very useful!

[botzill]

So, if I want to define a custom one can I do SCHEDULE_MY_CUSTOM_ONE? Or add in SCHEDULE_ADDITIONAL ?

[my8100]

Customize the SCHEDULE_ADDITIONAL option.

[botzill]

OK, thx.

[botzill]

Do you think is possible to add smth like SCHEDULE_MY_CUSTOM_ONE but the same we have SCHEDULE_CONCURRENT_REQUESTS and other?

[my8100]

Which setting options do you use frequently?

[botzill]

I have some custom ones defined, settings that are not part of scrapy. Thx.

…

On Sun, 23 Jun 2019 at 11:38, LxL ***@***.***> wrote: Which setting options do you use frequently? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#55?email_source=notifications&email_token=ABDYBWFCJKR2BMOVTSB2IRLP34Y6ZA5CNFSM4HZTKZP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYKZQVY#issuecomment-504731735>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABDYBWBXTDQ4AP6NLJCRSKTP34Y6ZANCNFSM4HZTKZPQ> .

-- Chirica Gheorghe, Co-Founder - https://www.crawless.com

[my8100]

I think the SCHEDULE_ADDITIONAL option is good enough for most cases.

[botzill]

Yes sure it is, I thought we could add inputs like that for easier setup for nontechnical users.

…

On Sun, 23 Jun 2019 at 12:41, LxL ***@***.***> wrote: I think the SCHEDULE_ADDITIONAL option is good enough for most cases. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#55?email_source=notifications&email_token=ABDYBWEXH56CSJISQBTCI2TP35ANRA5CNFSM4HZTKZP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYK2PPA#issuecomment-504735676>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABDYBWBZ5K4QGMBEE55YVGTP35ANRANCNFSM4HZTKZPQ> .

-- Chirica Gheorghe, Co-Founder - https://www.crawless.com