[issue147] smart's fetcher.py needs keep-alive and serialization support

[Axel Thimm at Labix Tracker]

New submission from Axel Thimm <Axel.Thimm at ATrpms.net>:

W/o keep-alive smart's concurrent downloading kills a web server, as each
package/metadata file opens up a new concurrent httpd process on the other side.

A typical smart session on a rather often updated system shows up to 15 httpd
processes simultaneously serving this one IP.

There are keep-alive solutions for urllib2, for instance urlgrabber's
keepalive.py that can be used as a handler for urllib2. This looks easy enough
for me to try patching up smart with it, if it's considered useful.

But most probably keep-alive is not enough as smart deliberately fires up
package retrievals in parallel, and keep-alive is only of help for reusing
connections. If a connection is still in use, you end up creating a new one.
Therefore a serialization procedure is neccessary when the packages come from
the same host (or at least the same channel).

----------
messages: 480
nosy: thimm
priority: bug
project: smart
status: unread
title: smart's fetcher.py needs keep-alive and serialization support

_______________________________________
Labix issue tracker <tracker at labix.org>
<http://tracker.labix.org/issue147>
_______________________________________