Külföldi torrent oldalak RedActed (PTH) | ReD News

Hi all,

First off, I hope everyone had a safe and enjoyable Christmas and a happy 2020 new year! You probably have noticed that our uptime record has been less than stellar in the past few weeks, and we've always tried to stay transparent to keep the users informed whenever we have updates regarding the site. As such, we're giving detailed information that we feel we owe to our users, and we'd greatly appreciate keeping this information within the bounds of the site, and not sharing this news post across various public platforms (Reddit, blogs, etc.).

The very act of sharing private tracker news on a public forum defeats the entire purpose of even having a private tracker in the first place and will potentially expose the inner workings of our infrastructure to our adversaries, including ones that frequently attack our services (IRC, site, AND tracker).

Earlier this month (December), we have had periods of time where the site has went offline due to memcached crashing. For those that are not familiar with memcached, it's an essential piece of software that caches many of our "expensive" database queries around the site. If memcached is offline, then the site slowly grinds to a halt since the database queries all have to re-run for each page load. Consequently, the database tables will lock up and the tracker will crash, PHP-FPM will crash, everything dies and someone has to manually restart memcached, then the tracker, and you get the point from here...

Solution: We've had to tweak our systemctl settings and various memcached settings to prevent future crashes from happening. Even with the watchdog (service restart upon failure) in place, memcached still needed some adjustments and upgrades since this happened quite a few times and we had trouble tracking down the root cause. Luckily this problem seems to be less frequent due to some other system changes we have made, but our set up is now a few years old so we're a few versions behind on some major pieces of software on our servers.

Another issue was that our Sphinx (search engine) catalogue/index had grown so large that it had outgrown our web server, so we had to implement additional compression around the site. We have to store every .torrent file, all logchecker HTML/.log files, and still have enough for the Sphinx index to rebuild itself from the database every few hours, which was not possible before the compression changes. The gzip compression has yielded an additional 20gb of free space, which is more than enough for an entire torrent search index rebuild, but at the cost of some development time and sub-optimal search performance. We've also experienced slowness around the site, which we've attributed to the search issues stated above.

The "bad gateway" issues were rampant in the past few weeks and this is due to a few key issues. First, we believe we may be suffering from some syn flood attacks and HTTP request spam against our frontends. The set up that has been working fine with our high peer count has been in place for a few years now, but suddenly started failing us in early December. We noticed 100% CPU utilization, but lower requests per second on our tracker. We build our servers quite lean to save money and run our operations on a fraction of what other similarly sized trackers consume, but we still have plenty of CPU resources available for one of the most CPU intensive operations on our site: full SSL termination for most of our tracker announces.

The solution to the bad gateway errors were a few changes - we adjusted quite a few kernel settings, in particular TCP stack sysctl settings; implemented nginx config tweaks and tracker backend changes, mostly to allocate more resources to netfilter/iptables; and rotated servers in and out of our DNS roundrobin to examine how equipped they are to handle load in case of failover. We also noticed that a lot of the HTTP request spamming was directed at URLs other than the tracker that did not exist and would not load, so we knew that it wasn't just one single issue that caused problems here.

The domain registration issue with our tracker domain flacsfor.me was a bit unfortunate timing here, but essentially the renewal was supposed to be automatic using our account funds, but I think auto-renewal may have been disrupted somehow and can't really trace the issue down to any one particular fault. Our DNS TTL is set to 300 seconds, meaning that the tracker IPs can be changed quite frequently without anyone having to flush their DNS cache. Unfortunately, when the domain expired, our provider handed out their own DNS records, which had a much higher TTL and resulted in issues that lingered throughout the rest of the day. We do have calendar reminders for every renewal across our entire infrastructure, but we didn't anticipate issues with our domain renewals since they are usually handled without intervention.

Lastly, the upload error was completely unpredictable and very hard to trace because despite seeing a "5xx" error code while trying to upload, we didn't see anything in our error log files. We were, however, able to narrow down the problem to the proxy servers (frontends) not having enough disk space to handle the file upload buffers that are proxied down to the backend. We increased our logical volume space on both frontends and ensured that nginx had a temporary directory with ample free disk space, since we typically run these servers as light as possible and use a lot of defaults.

Thank you for reading through and I hope everyone has a great new year! As always, if you experience any site issues, feel free to use our IRC (#red-help) or help forum, or you can always open a staff PM if it's more sensitive.

Cheers.

Happy Holidays from RED!​

We hope this holiday season provides you opportunities to reflect on the past, enjoy the present, and look forward to the future. To round out the year we wanted to provide you as always with the gift of music. This time around we present selections from Redacted staff as well as donors who have helped keep our holiday lights glowing warmly (as well as our servers plugged in) with their generosity.

For grabbing these or any other releases you would like (see conditions below), please enjoy 25 freeleech tokens which you should now see added to your account! Important note: As always, remember RED Rec's are not freeleech.