Yesterday the site was very slow again. Restarting the web server solved the problem for some minutes – then it went down again. While watching the logs with tail -f I noticed a IP addess which requested a lot of different video files at the same time. I denied access to that address and the server came back up to it’s original speed in moments.
IF YOU WANT TO DOWNLOAD A LOT OF VIDEOS, PLEASE, DO IT IN A SEQUENCE AND NOT PARALLEL AT THE SAME TIME!
I have lifted the ban on that IP (59.92.1xx.x – it’s in India and on a big pipe) but will keep an eye on the server performance and have the axe ready.
I’ll ask Marcin to limit the amount of connections for a single IP, then the problem should be gone.
UPDATE: Blocking the bot worked. And if you want to have a look at the log…. Whow!
Ops,
sorry some weeks ago I’ve made the same thing, probably I’ve blocked this site for some hours
Rolf said:
‘…but will keep an eye on the server performance and have the axe ready’
Cheers
> I have lifted the ban on that IP (59.92.1xx.x – it’s in India and on a big pipe) but will keep an eye on the server
Could be a proxy server or someone mirroring the site for local distribution. So I wouldn’t use the axe yet but maybe the scissors and just disallow access to the video files. The torrents are as good and seem to be working atm.
Can you not remove older Download episodes from the server and put them a FTP or use BitTorrent or something like that?
These problems should be solvable by limiting the number of connections per IP or by giving a separate demon the job to serve video files.
I don’t want to make the access to the old files more difficult than a click. Torrents are fine, but a lot of people don’t know how to use them.
maybe you could set up torrents containing 10 or more episode each chunk, so that those who want to grab em all have a convienient, non site killing way for doing that
I need a “CTO” on this project, T= Torrent….
… the Axe had to fall
I am now researching how to block only MP4 or the client software in .htaccess.
UPDATE: I added
RewriteCond %{HTTP_USER_AGENT} ^FDM
RewriteRule ^(.*)$ http://look.at.site/you/are/burning/the/server/
to the .htaccess. Seems to work and the error messages must be noticeable.
Marcin set up the server to allow only 4 parallel connections to the download dir, so I can lift the ban again.
Let’s see if there are any more outages. I hope not.
It got worse. I had to block an IP again and the machine is swapping heavily.
UPDATE: Swap is off, server refuses connection when full.
Hi Rolf!
I think the problem was because of my downloads. I set the FDM scheduler at night (low traffic hours).
AFAIK, that method has worked well for all websites: They always limit the number of files per IP or even number of streams per file; and the FDM just follows those rules. (Its log reads ‘Creating a new section > reached the limit of server…”) So I usually do not have to set any limits at all.
But AFAIK, I this is the first time a website had not set the limit, and choked as a result.
My sincere apologies.
I guess just setting the limits at your server will solve the issue.
Guiding them to the torrents also would lighten the load on your server.
(There will be plenty of new genuine GIMP-users who would want to get all past episodes, and mass-downloading is really convenient from their point of view. rather than instructing them, it is best to use server technology to control the traffic.)
I must say I was puzzled by the “download failed” and “access denied” messages; and was thinking of contacting the webmaster. Then I tried DownloadThemall, which somehow seemed to work.
Just by pure accident I came across this thread, so would like to clear up the air.
Thank you for opening up the IP again, and for the hard work you are putting in producing these high-quality episodes!
I love your work (and Heathenx’ too), and I would like to contribute in any way I can. For example, your episodes seem to have gone up to 125 but the TOC covers only till episode 112. Do you need volunteers to complete the index? I could do that.
Sincerely,
Narayan
No need to feel bad about it, the site should be able to handle this. It has been up to a point some weeks ago, but something must have changed either on the server on in the amount of demand.
And of course I am happy about everybody who can help me with this project. For example, there are TOC for all(?) the shows. Done either by me during editing or by Kevin after the release of the show.
But I have not found the time to get these TOC into the collection page.
Kevin has not to be the only one to write a TOC and when you volunteer to keep the collection page up to date I’ll give you an account on the blog to do that.
Yes, I would like to update the TOC regularly.
Please.send the details to my email).
Thanks!