info Autoshite technical help. Ask questions about how the site works here.

[Dyslexic Viking]

Yes it is as slow as a Norwegian winter.

[SmokinWaffle]

Came to say this too - it is crawling a bit! 

[Dyslexic Viking]

I also now get 502 Bad Gateway now and then.

[danthecapriman]

Slow as hell here too. Bad gateway errors too sometimes.

[bunglebus]

I got a site closed the connection error earlier, in addition to the previous reported sluggishness 

[Jazoli]

It’s glacial at the minute taking over 10 seconds to open a post.

[Split_Pin]

Aye same here, glacially slow were my thoughts too 😞

[SiC]

I've been monitoring these slow points for a while and trying a few things in the backgound. I've kept quiet as I don't want people to do anything differently in their use and thus add another variable.

Right now, I don't know the root cause.

Facts:

• Seems to come and go - sometimes suddenly. Worse during the day.
• CPU load stuck at 100% with either ElasticSearch or database server maxing it out (expected given those processes are doing the most work). This points to a server not network issue to users.
• Search through ElasticSearch or old method made no difference
• Rebuilding index of either search method made no difference
• 1 day search on old method (so least resources) made no difference
• CPU benckmark performance with GeekBench looks normal expected figures for class of server
• Disk I/O performance (with fio) appears to be normal expected figures (5K IOPS on artifact storage + system drive, 15K IOPS on database + search)
• No background tasks appear be running during these slow period. Either internal forum software or backups.
• Users online on the forum makes no difference. Infact during the day there are many less users than the evenings, but slowest during the day.

 
Possiblities:

• Hosting provider has over provisioned the cloud server its hosted on and worse during the day. However I can't get any definitive evidence of this from the benchmarks.
• Network issues between the cloud server and storage. I don't think this as I/O performance is expected values.
• Forum software is doing something weird/buggy. Hard one to prove. But why only during certain periods and varying lengths? I haven't actually looked at the Invision support forum... that'll be my next job.
• Denial of Service attack from elsewhere. I don't have any evidence for this. Network I/O looks normal and tbh its not usually that high. 1-5Mb/s down (peaks higher if images are uploaded to the server) and 2-15Mb/s up. Pipe max is 50Mb/s. User count isn't spiking and that would likely too.

Gut feel is the hosting provider is over provisioned but I can't prove this and so hard to really say its that. Without any hard evidence, I can't really go moaning to them.

So I really don't know. Its frustrating as fuck for me as no doubt for everyone else when it goes to a slooooooooow crawl.

I can't even say move to another provider as with no evidence, it still could happen. That would point to a forum software bug. But also I could move to another provider with better performance and throw resource at it. Potentially cheaper but I'd have to go dedicated box rather than cloud - I'd rather cloud as that means I only have to worry about the software and it outsources keeping on top of potential hardware failures.

[SiC]

...and oh look, its suddenly sped up!

Literally done nothing.

Like this time:

On 28/05/2024 at 19:57, rantingYoof said:

Holy shit, and just like that it's now instant page loads every time! Thanks to whoever gave the server box a virtually kicking!

and this:

On 24/05/2024 at 23:06, Scruffy Bodger said:

It sped up massively this afternoon for me. I assumed Si had done his magic in the background? 

Both times I wasn't at the computer and this time I was writing the above reply...!

Its this sort of sudden changes in speed makes me have the gut feel its the hosting provider. Like some other user on the box has suddenly let up on their load when its outside working hours.

[SiC]

31 minutes ago, SiC said:

Forum software is doing something weird/buggy. Hard one to prove. But why only during certain periods and varying lengths? I haven't actually looked at the Invision support forum... that'll be my next job.

Hmm interesting thread here from the last month with a resolution in the last few days: 🤔

https://invisioncommunity.com/forums/topic/478089-database-performance-issues-after-upgrading-from-4713-to-4716-getitemswithpermission/

So looks like it could be a combination of search engine crawlers and basically rather craply written Invision SQL code for the search system. The search system certainly is getting worked hard at times and its an integral part of making the profiles work properly. Likewise it gets worked hard on the longer threads we have.

There is a plug-in that undoes some of the later changes they did in their last update and people have reported success. Its not officially supported, but I might give it a go...

Other alternative is to put a block in the Huawei search engine crawler.

[SiC]

I've applied that plug-in. We will see what happens. 

There is also another forum update popped up. Not been a full version upgrade since December.

I'll probably schedule it in the next few days. Should be a pretty quick job - about half hour or so.

[loserone]

Fuck me that's lightning fast now

[SiC]

This is what it's been like load average wise during the slow periods. The CPUs are literally hammered to absolute shit but disk and network I/O isn't that high.

Right now...

 

Only instantaneous figures but the top was basically pegged at 100% on all four CPU cores. About 100 extra users online in the second than the first too. 

So much so that it this level of usage stays around the same, I'll probably ratchet it back down a tier on the server class too!

[Split_Pin]

Cheers Si! Whatever you did/thought, it worked!

[SiC]

I can't say for sure that patch will fix this issue as I applied it when the slow period finished. However we'll see in the next few days I guess.

[SmokinWaffle]

Not seen it this fast in a while - its blazing quick and responsive. Fingers crossed that's done it - thanks for your continued troubleshooting and hard word @SiC!

[SiC]

It's best to reserve judgement until a few days/weeks have passed. These sort of things you think you've sorted and declare it fixed, then bites you back on the arse 😅

If we get it again, I might look at a CDN for guest users and potentially blocking some of the more abusive web crawlers especially on profiles. 

[SiC]

Just had a thought. If it is user profiles that are being crawled by search engines, then why not disable non-logged in users from accessing them?

So I'll do that. If you're not logged in, you can't view a users profile details. Hence the web crawlers can't view them.

Tbh I think it probably should be that way anyway? Any immediate objections to this? (I might put a wider post)

I haven't done it yet as I want to try it if/when we get another slow period. Would be magic if it suddenly fixes the slow performance. I might also disable the patch too. 

Basically I want a slow period and try these new tools in my toolbox. That way I can be sure I've fixed the root cause. 

[High Jetter]

I hope we don't get a ticket, full steam ahead now

[SiC]

Going mega slow again. Applied all the above and it hasn't fixed it. 

Right now I've completely disallowed Guests (i.e. not logged in) from viewing the forum entirely. Still chronically slow. While it will be serving up web pages to say disallowed, they shouldn't be taxing the server at all.

Again this is point to my gut feeling that it's the hosting provider has over provisioned the server to other clients who are hammering it during the day. 

[SiC]

I've completely disabled search and no difference either. The search system is still working away as it's used by the software to show forum threads. But says that it's not a quantity of searches causing it either. 

Only 375 users online (that includes guests) right now. Nothing like the 600+ yesterday evening when it was running super quick. 

[SiC]

Bingo. I think I've found the culprit. 

I've disabled Content Discovery for guests. Not sure if this is the root cause (still could be the hosting provider going slow during the day) but it seems to tremendously increased performance again. 

Content discovery is the activity streams that I know many use. It should be working fine for logged in users. Just those not logged in won't be able to use it (they'll need to log in or make an account).

CPU load is still higher than normal but it's as a manageable rate now. 

My suspicion is that maybe a crawler bot (used by search engines for indexing a website) or something else is hammering that feed as it constantly is trying to update. Maybe someone has their work computer with an application that is using that activity feed to give them updates during the day?

I'll keep an eye on it and if we get another bad spike, I may disable it entirely for a bit and see if that resolves the issue. I know people use that frequently so I don't want to disable it permanently. Just right now I want the main bulk of the site to be usable and functional. 

[SiC]

Yeah just flicked it back on for guests and the CPU load shot skywards again with the site going to a crawl. Turn it off and things are usable. 

[loserone]

It's also the Unread content stream, though not sure how that would work as a guest user

[SiC]

15 minutes ago, loserone said:

It's also the Unread content stream, though not sure how that would work as a guest user

Yeah it's my suspicion that it's the unread content which is working the server the hardest. Presumably it assumes everything is unread and just gives the latest posts?

It can be had as an RSS feed too, which makes me suspect it might be a buggy client somewhere hammering it. 

[SiC]

No idea why we had a sudden slow down for 5 to 10mins there. I started shedding some of the forum software services and it wasn't making any difference. Seems to have eased again on its own and dropped right back. Maybe the forum software was running some heavy tasks. No way of knowing from the admin panel if certain tasks are long running or not. 

[EyesWeldedShut]

I sometimes (right now as it happens for example) get this if I go to https://autoshite.com/discover/unread/

I can still browse forums using the top menus just fine 

[SiC]

3 minutes ago, EyesWeldedShut said:

I sometimes (right now as it happens for example) get this if I go to https://autoshite.com/discover/unread/

I can still browse forums using the top menus just fine 

Yeah that's me desperately shedding services above when things were going slow a moment ago to try bringing a bit of normality back to the performance.

That discovery feature I'm permanently disabling for non-logged in users as definitely that is one big thing that causes slowness when too many people hit that page. 

I thought it would have sorted this last go slow tonight but it didn't. Which suggests there are multiple things causing the CPU load to go through the roof. (Invision forum software really is written a bit shitty and inefficient)

 

[EyesWeldedShut]

1 minute ago, SiC said:

(Invision forum software really is written a bit shitty and inefficient)

It's one of those PHP/MySQL gifts from my generation that just keeps on giving  

(Although V 4.x was billed as a total re-write?? Yeah, sure)

[SiC]

2 minutes ago, EyesWeldedShut said:

It's one of those PHP/MySQL gifts from my generation that just keeps on giving  

(Although V 4.x was billed as a total re-write?? Yeah, sure)

They say the next version - v5.x is one too!