info Autoshite technical help. Ask questions about how the site works here.

[Dyslexic Viking]

Everything seems fine to me now.

[danthecapriman]

19 minutes ago, SiC said:

Are people still finding it slow? Seems fine for me at the moment. 

I really need to get a jiffy on to find some time and move this over to the new setup. 

Seems ok at the moment.

[SEATMad]

Yeah all good now

[High Jetter]

52 minutes ago, SiC said:

Are people still finding it slow? Seems fine for me at the moment. 

I really need to get a jiffy on to find some time and move this over to the new setup. 

It was, 6 -7pm. Seems ok now

[SiC]

Self healing fixed are the best 🤣

I think a backup might be kicking off around that time and probably when everyone is getting home. I might need to tweak those timings so it doesn't get hammered too hard if this keeps happening. (Or move to the new server that has more horse power)

[eddyramrod]

We all love* an intermittent fault, don't we?

For some weeks now, when the site feels like it, I've had difficulty getting into a thread.  Not in general, oh no: the one thread that really keeps me here, @Split_Pin's magnificent Shite In Miniature II.  If I click on the thread title on the index page, the site thinks about it for a little while, then tells me "502 Bad Gateway."  I've tried clicking on a page number: same result.  If I come back to it this afternoon it might work fine.  Or it might not, I have no way of knowing until I try.  It might do this for the rest of the week, or tomorrow might be fine.

I haven't noticed any other threads doing the same; some are older and more populous, some younger and sparser, but it seems to be an issue with this particular thread for some reason.

Could somebody have a fiddle please?

[Datsuncog]

'502 Bad Gateway' back on the Shite In Miniature II thread this morning...

Seems that the miniature shite thread (1641 pages) and the eBay Tat Vol 3 thread (a whopping 7144 pages) seem to be the most prone to outages; there's a lot of images on there too, at least some of which is my fault.

Is it maybe just that the threads have become too big?

[Split_Pin]

Yep same here! My thread has FTP.

[eddyramrod]

I'm glad it's not just me!

[eddyramrod]

And to reinforce the intermittency of the issue, I just clicked on the last page number available and got in!

[Datsuncog]

I'm wondering whether creating fresh threads, for 'Shite in Miniature III' and 'eBay Tat Vol 4', might be one solution worth discussing with thread regulars - with the original authors credited, and backlinks to the previous threads?

I hate the thought of locking off longrunning threads, especially ones I've been contributing to for a long time - but can appreciate that sometimes things become very unwieldy as they expand far beyond original expectations. These big threads do tend to run as 'hot topics' from time to time whenever there's a lot of engagement, and this seems to precipitate/coincide with the crashes. 

As long as the older stuff remains accessible to search and view (since I spend rather more time than I care to admit delving through posts from 5 years ago because I know I saw a 1/24 Peugeot 504 in a tat box some time in late 2019 - and perhaps that's also part of the issue?), I'd sooner have a reliable link working for everyone!

If there are other possible fixes then of course I'm happy to go with those before considering this sort of nuclear option; but this just seems to be a recurring issue that's causing frustration as well as taking up people's time on the technical side too.

[SiC]

Definitely the larger threads stress the server more. I can see the spikes when I watch the load as I'm posting on them. It shouldn't be any difference in thread length to performance but I suspect it's just the forum software just being badly written. My suspicion is that a new post will rebuild search caches and that takes time in the long threads to complete. 

It's been the case for literally years for me that a post in News 24 thread takes tens of seconds. Noticed too that it's only posting, editing threads is fine. Have a few people do that and do it on other long threads then all the resources get consumed. No resources left causes those 502 errors. 

Shite in miniature has a heady mix of a lot of photos, long posts and many replies in short order. A busy, active thread that uses a lot of resources. It's great that people enjoy it and exactly the sort of thing we we want on here. However it's just one that is pushing the server past the edge on what it can do in the current setup.

I could allocate more resources to the server but the current setup will mean quite a bit more cost to do that (like ~50% more). Tie that into the server software coming end of life, you might as well rebuild the server and allocate things to take into account how we use the server in 2024 rather than in 2019 when it was last rebuilt. That way we can allocate more processor resource without substantially increasing costs.

These reasons why I went through the effort of rebuilding the server and put it up as autoshitetest.co.uk to see how things work. That setup has an emphasis on making the database storage 3 times quicker than it is now. Plus that's running on more modern hardware that does more work per second. 

As an example, I've found the News24 thread on that new setup to post nearly instantly. Yes much less people are on it, but that thread is slow on here even late at night when few people are on the forum too. 

So while we could start new threads on those that are long running, really the best solution is to migrate across to the new setup. There will be a point where we'll run into this again but by then in a few years, we'll be upgrading to the latest hardware again. Also the version 5 forum software will be out and hopefully improve these things (admittedly I'm not holding out too much hope given this forum software clunkiness at times)!

However that'll involve a few hours downtime to backup everything on here and restore it on the new server. Then catch any new issues from migration that may crop up from being a new setup. I just need to find a bit of time to dedicate to that (I'm short of time atm - hence stuff like my MG Midget not being touched in months). There will be kinks that will need ironing out, so when done, I'll need a little bit of patience to get everything solidly stable again. 

I'll see if there are any short term fixes I can do to improve things for now but the tricks left in the box are getting low. 

[Datsuncog]

Cheers for coming back on this, Si - much appreciated!

As a resolute non-techie, I greatly appreciate the effort that's put into making the forum run smoothly - which in fairness it generally does. Sounds like a server rebuild and migration would be the best solution in the medium-term, but as you say, it's not really a click-yer-fingers-and-it's-done type thing. But if that's what it takes to get the usage spikes under control, then some downtime is a small price to pay once a slot opens up.

Good to know that it's the posting which seems to be causing the issues - I'd somehow thought it was maybe me messing things up by going back into old, old posts on the SiMII thread to retrieve info, or check a photo. But it's a busy old thread, agreed!

[SiC]

23 minutes ago, Datsuncog said:

Sounds like a server rebuild and migration would be the best solution in the medium-term, but as you say, it's not really a click-yer-fingers-and-it's-done type thing.

The server is built and mostly configured (need to change config to autoshite.com from autoshitetest.co.uk) but it's the backup of this server and restore onto the new that needs doing. It'll be a couple hours job due to the amount of data to be transferred (mostly waiting for that to be done), then flicking the switch to move over. 

As the backup is a snapshot at a point in time, the whole site will need to be put in an offline mode until it's done. 

[SiC]

Interesting its doing it right now on Shite In Miniture. First time I've had a chance to look at it when its actually failing. 

Server is maxed out with the database server taking most of the resources. A database backup had just finished and a content backup was working away (not long finished).

Number of file transfers is very high too.

Restarting the database server doesn't fix it and still is maxed out when started back up.

There are currently 447 users online. Thats unique number of people accessing this site right now.

I think its just a case of 

 

I think it'll be a server upgrade next to fix this properly.

[egg]

I feel bad for posting a lot of pics and shit now, perhaps I need to proceed with more mechanical sympathy!

[High Jetter]

14 minutes ago, egg said:

I feel bad for posting a lot of pics and shit now, perhaps I need to proceed with more mechanical sympathy!

Hardly mechanical, more of a bits and bytes issue. Size matters.

[SiC]

37 minutes ago, egg said:

I feel bad for posting a lot of pics and shit now, perhaps I need to proceed with more mechanical sympathy!

No no, keep doing it! It's what this forum is about. 

 

I need to look into resolving the amount of PNGs that keep being uploaded though. They aren't a very well compressed image format and they're eating up large sums of storage at the moment. These usually are screenshots and phones are high resolution now which makes them many megabytes. 

Ideally I'll see if I can recompress them as JPEGs as that should massively reduce space. Just there is no way of the forum software doing it, so I'll have to script up something to happen regularly. 

[scooobydont]

Look's like its starting to slow again right now if it helps identify the intermittency.

[bunglebus]

Getting the 502 thing again on Shite in Miniature.

I hate to say it but is that thread too big? Does it make a difference?

[SEATMad]

5 minutes ago, bunglebus said:

Getting the 502 thing again on Shite in Miniature.

I hate to say it but is that thread too big? Does it make a difference?

Had the same issue with eBay tat, seems to be okay for the time being

[SiC]

Seems to be around 6pm to 7pm it has the biggest issue. I think it's a mix of site traffic increasing substantially (people coming home from work/finishing eating) and the updates kicking in at the same time. 

I might tweak the times the backups start to not coincide with peak traffic. 

[loserone]

Presumably it's not loading the whole thread every time though 

[SiC]

You'd hope not but it's doing a hell a lot of processing considering it only needs to run a straightforward query that's limited to a known number of results (i.e. a page display). God knows what crap they're doing when a post is made or even a thread accessed. 

[eddyramrod]

@SiC it didn't work  I've been trying to get into the same thread for several minutes now to pick up on a notification.  All together now: 502...

[SiC]

33 minutes ago, eddyramrod said:

@SiC it didn't work  I've been trying to get into the same thread for several minutes now to pick up on a notification.  All together now: 502...

Yeah I was on the site at the time. Everything going absolutely dogshit slow again - not just that thread. As someone who spends a bit too much time here, it irritates the hell out of me when it goes slow/breaks too. 

Looking at what's going on with the server at the time it slowed, there was nothing different in what it was doing except being heavily loaded. I struggle to believe that there is a sudden influx of posting all at just that point.

This server is hosted on a machine as a virtual machine that is shared with many other virtual machines. My suspicion is that the underlying server is too heavily loaded and there are other tenants that are hammering it at the same time too. After all everyone wants to get their money's worth by running things hard. Both the users paying for virtual machines and also the hosting provider putting as much on that machine as possible. 

Really I think the only solution is to migrate to the new setup. I can't make any promises but I'll see if I can do something around on Friday and over the weekend. As the root cause isn't obvious, it won't guarantee that it'll fix the problem. But I'm 90% sure both the virtual machine and the underlying machine this is running on is too overloaded. The overload from our virtual machine and others are coinciding. 

[mercedade]

As an aside, I occasionally get duplicate pages showing - the final two pages of a thread are identical. Assuming it's some kind of database/reconciliation issue? (I know nothing about websites).

I thought I was imagining it for a while so didn't say anything, but it's definitely happened a few times now, most recently on the New24 thread.

[Supernaut]

I've noticed some weird behaviour in the text editor box.

I'm on Firefox on Android.

I was writing some text in italics, but then afterwards when trying to put in a line break, it kept just adding my next letters onto the end of the italics without even a space.

I could mash enter loads of times, then as soon as I tried to type anything else it would jump back.

[SiC]

That'll be filed under the "this forum text editor is shit" category unfortunately and highly unlikely caused by the migration. Refreshing the page often helps. 

I bloody hate the editor but Invision give you no other real choice. The editor is end of life about a year ago but they keep rolling on with it - presumably version 5 of the forum software might improve this?

[Zelandeth]

7 hours ago, Supernaut said:

I've noticed some weird behaviour in the text editor box.

I'm on Firefox on Android.

I was writing some text in italics, but then afterwards when trying to put in a line break, it kept just adding my next letters onto the end of the italics without even a space.

I could mash enter loads of times, then as soon as I tried to type anything else it would jump back.

This also happens frequently when trying to embed links or tagging users.  As SiC says, it's just one of the "quirks" of the complete disaster that is the Invision text editor.

If you're on a mobile device, *sometimes* forcing the page to reflow by swapping from landscape to portrait (or vice-versa) and back will kick it into working again.  Not always though.