TOS Posting Broken

[3dmk]

Checked out TOS this morning and saw that there have been no new posts in any of the really unhappy threads. Is this really what the end looks like?

https://www.youtube.com/watch?v=JSUIQgEVDM4

I give up trying to embed this amazing video.

[wyo-cat]

We got outta that place just in time.

[CalStateTempe]

[Alieberman]

It ended 3 days ago.

[NYCat]

It really is amazing how quickly and suddenly it became vacant.

[3dmk]

Scorched Earth ftw! I was expecting at least a parting shot from the administration letting us all know just how much we were appreciated, but I believe the forums falling over on themselves and just leaving it there speaks volumes for itself.

Not sure if it is possible to export threads there but if you have threads you really enjoyed and want to retain quotes, better go grab them.

[CalStateTempe]

Not sure if it is possible to export threads there but if you have threads you really enjoyed and want to retain quotes, better go grab them.

Thats my biggest concern at the moment. From what I've reviewed about UGC and media law, we are the owners and the website is the proprietor on the basis that our ability to use the website free is on the agreement that they can hold the content.

Frankly as I see it, that means we can take what we feel like. Since you are computer savvy, do you know of any other issues around this that we might be missing and/or know of an easy program or mechanism other then copy/paste/save PDF to archive here?

I'd say hire 4chan, but that would be the equivalent of asking the Godfather to help with a grievance.

[3dmk]

Being as we are using two commonly used message boards on both sides (vBulletin there, phpBB here) and both are using default configurations it could be simple enough to scrape some content from one and post it on another. Accounts that are caught on TOS could be turned into Shadow accounts here. The big issue is the current performance of TOS. The site is running so ungodly slow that any script I write to scrub content is going to take quite some time and could potentially make the situation worse. Page loads are taking progressively longer and longer to complete. Eventually the site will start to hit timeouts caused by the web server.

It is a very slow and depressing death spiral we are watching.

[CalStateTempe]

Interesting.

As an aside, what in your opinion, technologically speaking is happening? Is it soley the upgrade or it is that plus having no traffic to "keep things running as they should" (sorry for the lack of tech speak).

what causes it to start hitting timeouts and what would those timeout lead to?

[Alieberman]

It is a very slow and depressing death spiral we are watching.

I'm actually not depressed about it one bit anymore.

I'm excited about helping create a fan run site from the ground up. This is going to be so much better.

[wyo-cat]

Checked out TOS this morning and saw that there have been no new posts in any of the really unhappy threads. Is this really what the end looks like?

https://www.youtube.com/watch?v=JSUIQgEVDM4

I give up trying to embed this amazing video.

[youtube]JSUIQgEVDM4[/youtube]

You just need to post the business part of the link "JSUIQgEVDM4" between the [youtube] brackets.

[azgreg]

[wyo-cat]

That is a fitting metaphor, Greg!

[3dmk]

100% pure speculation based on observations to date:

GOAZCats is not a small website or forum. There are years and years of posts by likely hundreds of thousands of accounts. Once a forum hits this point migrations become something that requires more than a update script. Databases need to be prepared for schema changes. These changes require shutting down the entire site, backing up the database, and then running DB update scripts against the DB to determine the impact of the site. This was not done. They ran the update on their production environment and have likely realized that their database server does not have the capacity to handle the new software. Oops. Something else tells me they are using Oracle instead of MySQL or MariaDB, or even Postgres. Oracle is not an easy DB to work with. Once the update was run they likely realized they are in far deeper than they expected. The fact that they are reliant on vBulletin support tells me that they do not have the technical chops in house to handle the management of a system like this.

To see if this is the case I went over the portfolio of WebHouse1616. Aside from a couple, none of their sites are really any more complicated than brochure websites sharing information. They also do not run many sites that do nearly as much traffic as we likely see at GoAZCats. Sites like this require distributed systems to handle core needs, like database clusters with Read Slaves for browsing. Based on site performance when under load, we don't see any of this on their end.

This leads me to believe that the site was already running on lower than needed hardware requirements and that this update has now pushed them well over the edge. What we are seeing now is likely processes on the backend becoming more and more unhappy. It is clear that database writes have been suffering since the update. If you look at the cache updates for last updated and latest posts taking time to update, you can see that from start of the update to now, that update was taking progressively longer and longer to complete. We are already reaching the DB write timeout for new post creations. It generally takes about 120 seconds, but the post action is failing. Page loads likely also perform some type of write logic, albeit much smaller. Eventually those writes will hit their timeouts as well.

The site code (PHP I believe) is absorbing most of this sadness but is likely spamming the filesystem with tons of log output about the sadness of all of these writes. NGINX or Apache writing to the filesystem will eventually take so long to complete that they will give up. This will then lead to timeout and request failures. This is the point in which the website will go away.

Based on the words of the last posting from Admin over there, it looks like they are going to try and restore to a date in May. If this is the case, its likely their last known successful backup of the database. My hope is that they have the site code to go along with it and didn't stomp it. It would clearly remove all of our posting about the new site and such, along with our disdain at their latest business practices. Thing is, that update should only take hours to complete. The fact that they haven't done this yet again leads me to believe that they do not have the technical chops to support the site itself.

Long story short, they are in way over their heads now and any chance at resurrecting their system is going to be very painful and likely cost them money. I have tried reaching out to them to provide some high level guidance since I deal with stuff like this every day, and I would do this at no cost because I love the community, but they have not responded back to any of my posts or PMs.

[3dmk]

You just need to post the business part of the link "JSUIQgEVDM4" between the [youtube] brackets.

Win! Thanks!~

[CalStateTempe]

3dmk:

That was an awesome explanation of issues that are outside the content of my knowledge. Thanks for the time to write it all down.

Damn...circling the drain is right.

[3dmk]

I try. I know basketball, but I understand the inner workings of high availability, high volume web architectures far better.

[Katzenfreund]

.

[Chicat]

Very interesting 3dmk. Thanks for posting!

If only the Chairman MAO had bothered to explain even a tenth of what you just did instead of blaming all the troubles of the world on us durned free-loading bastards...

[3dmk]

Most companies have a difficult time admitting that they made an error from a technical perspective. Most companies don't even call it an error. AWS (http://aws.amazon.com/message/680342/) generally calls them Service Events whenever something major happens. They also do a good job of explaining what went wrong, but this is the business they are in.

WebHouse1616 is not a company that operates at this scale. This is not even something you can fault them for either. They are a smaller web company that does business with local Tucson companies and generally doesn't have to deal with things at the scale they do with GoAZCats. There also isn't anything wrong with admitting that you are unable to do something.

Where you can fault them is where they blame the community for not getting premium as their reason why they cannot afford to host the site. If they do not have the resources to host something this large they should be honest about it. There are a number of others out here who would cover that expense on their own to support this community without the need for things like Premium Services.

[3dmk]

Old site is back, missing all of the content concerning the issues between Management and the Free Board users. Depressing.

Now the real excitement, will people stick to this site, or go back to the old version?

[Chicat]

Old site is back, missing all of the content concerning the issues between Management and the Free Board users. Depressing.

Now the real excitement, will people stick to this site, or go back to the old version?

Unless I have a compelling reason to go back (like a mass exodus from this site back to TOS and MAO) I'm staying here. This is where my friends are.

[ASUHATER!]

Old site is back, missing all of the content concerning the issues between Management and the Free Board users. Depressing.

Now the real excitement, will people stick to this site, or go back to the old version?

anyone that has sense or that wants to help run this site and make it our own will stay.

[CalStateTempe]

Same here.

I am big on past actions predict future ones and known MAO's history and how he bungled this affair; a tiger can't change his stripes.

Also, in its simplicity, this site is actually MORE functional and faster than TOS in its current iteration. I am excited for what can be built here.

[Catintheheat]

I strongly believe that what is there now is the best it will be. Unless MAO brings a sincere mea culpa (doubtful) why would anyone want to go back? Leave the past behind. Going back would be like staying in an abusive relationship because it is familiar.

[BearDown89]

someone by the handle of drallafi has started a new thread in the PT.

Other than that, no new activity.

[3dmk]

Same here.

I am big on past actions predict future ones and known MAO's history and how he bungled this affair; a tiger can't change his stripes.

Also, in its simplicity, this site is actually MORE functional and faster than TOS in its current iteration. I am excited for what can be built here.

Totally agree, I just hope it retails the core group of strong content generators who made going to TOS enjoyable. Ever since the 'event' I have been working on a Site Design Doc for what I think the community would enjoy. Once it is completed I will likely shop it here and with a few other posters I know to see if it could gain momentum.

[CalStateTempe]

awesome.

Can't wait to see it.

[Catintheheat]

Same here.

I am big on past actions predict future ones and known MAO's history and how he bungled this affair; a tiger can't change his stripes.

Also, in its simplicity, this site is actually MORE functional and faster than TOS in its current iteration. I am excited for what can be built here.

Totally agree, I just hope it retails the core group of strong content generators who made going to TOS enjoyable. Ever since the 'event' I have been working on a Site Design Doc for what I think the community would enjoy. Once it is completed I will likely shop it here and with a few other posters I know to see if it could gain momentum.

I was just thinking about this and posted it in another thread. Thank you for your service to the community. Can't wait to see what you've come up with.