Quote:
Thanks, I probably should have read the full man page (I just skimmed the readme). I'll try that when I resume the downloading. I'm not sure whether it will work or not, as technically, I'm not connecting to multiple files, but one php file that I'm passing a different parameter to each call. It's definitely worth a shot though.
I couldn't resist following up on this... mostly because of your mention of curl. But also because I had a vacation cut short because my spouse wrecked our 2008 Mustang (but that's another story).
Anyway I used a series of commands to get the usernames and birthday's of the all the Lurker Lounge accounts. I was worried about RAM usage (probably needlessly) so I didn't use just one command to pull down all the profiles.
Code:
curl -s http://www.lurkerlounge.com/forums/index.php?showuser=[1-5000] | egrep -A1 "Birthday|<div id=\"profilename" | ./parseAge > out1
curl -s http://www.lurkerlounge.com/forums/index.php?showuser=[5001-10000] | egrep -A1 "Birthday|<div id=\"profilename" | ./parseAge > out2
curl -s http://www.lurkerlounge.com/forums/index.php?showuser=[10001-15000] | egrep -A1 "Birthday|<div id=\"profilename" | ./parseAge > out3
curl -s http://www.lurkerlounge.com/forums/index.php?showuser=[15001-20000] | egrep -A1 "Birthday|<div id=\"profilename" | ./parseAge > out4
curl -s http://www.lurkerlounge.com/forums/index.php?showuser=[20001-25000] | egrep -A1 "Birthday|<div id=\"profilename" | ./parseAge > out5
curl -s http://www.lurkerlounge.com/forums/index.php?showuser=[25001-30000] | egrep -A1 "Birthday|<div id=\"profilename" | ./parseAge > out6
curl -s http://www.lurkerlounge.com/forums/index.php?showuser=[30001-35000] | egrep -A1 "Birthday|<div id=\"profilename" | ./parseAge > out7
curl -s http://www.lurkerlounge.com/forums/index.php?showuser=[35001-39465] | egrep -A1 "Birthday|<div id=\"profilename" | ./parseAge > out8The "parseAge" command is an awk script to parse out the username and the birthdate. It's rather ugly but I'll be happy to post it if anyone requests it.
It still ran pretty slow. All together it took about 2.5 to 3 hours to run. I PM'd the results to Pete, so he can work his numerical analysis.
I did notice that, even though the user numbers start with one and are incremented for subsequent users, not all numbers represent valid users. For example, there is no user associated with 10000 (i.e. there is no profile for this URL: http://www.lurkerlounge.com/forums/index...user=10000 )
Plus, new users are being added all the time (obviously), so my list is accurate as of around 4/9/2009 7:30 am EST.