Skip to content
This repository was archived by the owner on Jul 19, 2018. It is now read-only.

HCF experiments #27

Open
kmike wants to merge 14 commits into
base: master
Choose a base branch
from
Open

HCF experiments #27

Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
285a69d
WIP experiment: job scheduling is removed, more granular batches proc…
kmike Jun 6, 2013
6f2f410
limit a number of maximum concurrent batches; use HCF for start_requests
kmike Jun 7, 2013
40df7f4
use the same slot for input and output by default
kmike Jun 7, 2013
32b4a4a
Rename slot_callback to hs_slot_for_request
kmike Jun 7, 2013
a634251
More experiments:
kmike Jun 8, 2013
c0d007a
rename hs_make_request to hs_deserialize_request and change its signa…
kmike Jun 8, 2013
c3f7225
hs_ vs hcf_
kmike Jun 8, 2013
b1f4dcd
Fixed an issue with untracked requests: all started batches are delet…
kmike Jun 10, 2013
5644ebb
delete all started batches if spider finished normally
kmike Jun 10, 2013
e47f377
set default HCF_MAX_CONCURRENT_BATCHES to 5 (for more frequent checkp…
kmike Jun 10, 2013
5abfa2b
minor tweaks
kmike Jun 10, 2013
4512d4c
less verbose output
kmike Jun 10, 2013
95bac7e
more stats; less verbose log
kmike Jun 11, 2013
b6e2ca4
remove pointless self
kmike Jul 2, 2013
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading