[issue147] smart's fetcher.py needs keep-alive and serialization support

[Gustavo Niemeyer at Labix Tracker]

Gustavo Niemeyer <gustavo at niemeyer.net> added the comment:

> W/o keep-alive smart's concurrent downloading kills a web server, as
> each package/metadata file opens up a new concurrent httpd process on
> the other side.

Opening new connections for each package, while not ideal, shouldn't
kill a web server, should it?

> A typical smart session on a rather often updated system shows up to
> 15 httpd processes simultaneously serving this one IP.

When using URLLIB, Smart is currently limited to 5 active connections.
If you want to make tests, or even patch your local Smart, you can easily
change the constant in fetcher.py (MAXACTIVE). If you belive that 5 is
really a bad idea, let's talk. I'm open to reducing this limit.

> There are keep-alive solutions for urllib2, for instance urlgrabber's
> keepalive.py that can be used as a handler for urllib2. This looks
> easy enough for me to try patching up smart with it, if it's
> considered useful.

There's already a urllib2 fetcher, but it's commented out because it's
not thread safe. I'm not sure if we can do the same thing in urllib.

Have you tested Smart with pycurl?  It should reuse connections
automatically, like you're suggesting.

> But most probably keep-alive is not enough as smart deliberately fires
> up package retrievals in parallel, and keep-alive is only of help for
> reusing connections. If a connection is still in use, you end up
> creating a new one.  Therefore a serialization procedure is neccessary
> when the packages come from the same host (or at least the same
> channel).

Smart does limit the number of active connections already. Improving
that limit shouldn't be hard. OTOH, since you say that you have 15
open connections, there must be something else wrong.

----------
nosy: +niemeyer
status: unread -> chatting

_______________________________________
Labix issue tracker <tracker at labix.org>
<http://tracker.labix.org/issue147>
_______________________________________