❗❗🚨 Major Autoshite Server Maintenance and Downtime - Completed 🚨❗❗

[SiC]

TL;DR the site will go for some major maintenance this coming Friday afternoon/evening. I’ll figure exact times closer to the point. Expect a few hours downtime when this is all going on.

The current server setup has been online for nearly half a decade and in computing world, that’s an actual lifetime. It has had a good innings but it’s coming to end of life.

Not only is the server operating system getting close to not being no longer receiving security updates (end of June 2024 is the cut off) but the underlying server hardware is pretty old technology now. I suspect the hosting provider has chocked that server full of other clients and hammering it hard too. Both reasons are very likely why things seem to go really slow at times and you may on occasions get that very frustrating 502 Bad Gateway errors when trying to access parts of the site. This occurs when the site takes too long to generate a page and times out (around 30 seconds). 

Ultimately, we need to migrate to a new server and do it soon. Hopefully it will eliminate those errors. Of course I can’t guarantee for certain those errors will completely go away but I fully expect them to. Likewise, its important we move to continue getting the latest security updates to keep the site stable and secure.

To use an analogy, previous recent updates have been the oil and tyre changes on the bus. This is basically a whole new bus where the branding/customisations need to be reapplied.

A few months ago in the background, I have built up and started testing a new server enviroment. Some may have seen the previous announcement a few months ago about the testing site at https://autoshitetesting.co.uk - thanks everyone who checked out and gave it a poke. The server that this site is on will become the new server the forum will live on (will still stay as autoshite.com when fully live). This is the latest version of our Server Operating System (Debian) and should give us security support till the end of June 2028. The hardware is the latest technology in CPU, memory and disks. These all should offer multiple times the performance of the existing server setup. I’ve jigged things around so hopefully it will keep it to around the same monthly hosting cost despite the increase in performance.

It’ll take a good few hours to copy all the data from the old to the new and then make the switch. I don’t expect any major hiccups but there maybe a few kinks that may need ironing out once live too. I’m hoping that most machines should pickup the new server IP address automagically and we won’t lose anyone along the way. I’ll try putting up a site somewhere else that we can still provide support if things go wrong (probably https://autoshitetest.co.uk). So note that down now.

I’ve been wanting to do this for a while and have had everything roughly in-place, but I’ve needed to find time to finish this off and go live. Life has been busy for me recently and I’ve needed to find a good slot. Not just a few hours but possibly a day or two if things go really awry. I certainly don't expect any issues (otherwise I wouldn't be kicking this off!) but I need to budget contingency as its a major switch with a few unknowns.

The aim is for possibly Friday afternoon/evening to kick this off. I can’t say exact times yet but hopefully can say closer to the time.

I like being on here as much as many of you do and this work is needed to keep this forum all hunky-dory.

I'll leave this thread unlocked for now if anyone has any questions or concerns on this.

[SiC]

Irritatingly the forum software (Invision Power Board) has just pinged up an update! Its a security update too but thats only in the Commerce part of the forum software package - we don't pay for that or even have it installed. 

I don't really want to apply that update along with this migration at the same time. Doing too much on an update is a receipe for things to fuck up and go wrong in my experience. So I will likely apply that in the next day or so before the migration. Just to ensure that is all stable before doing a big switch. However I don't imagine the downtime from that will be more than half hour or so.

[Twiggy]

A huge thank you from me for all the hard work you put in to keep the site safe and up to date.

[DSdriver]

Keep a cool head and double check everything. We have every confidence and look forward to reality being returned.

[SiC]

Yeah it'll be fine. It's not a complicated setup by any means. Just need some contingency with time as people will moan/worry what's going on if I'm having to take time sorting out fixing stuff that are broken or not functioning correctly. So just setting some expectations before people panic if the site is down for half a day or so. 

[Dave_Q]

3 hours ago, SiC said:

Yeah it'll be fine. It's not a complicated setup by any means. Just need some contingency with time as people will moan/worry what's going on if I'm having to take time sorting out fixing stuff that are broken or not functioning correctly. So just setting some expectations before people panic if the site is down for half a day or so. 

Anyone moaning about this should ask their grandparents about when daveb47 had the keys, the site randomly went down for days and days on end with no real explanation other than sob stories about how he was doing it all as a volunteer and it was very upsetting to be asked to be competent, featuring bonus donation tin rattling.

Having a couple to a few hours downtime, publicised well in advance, in order to upgrade the site for real operational reasons, is nothing for people to be wittering about.

Kids today, etc.

[SiC]

Maybe moan is not the best word for me to have used. More that I know that people always like to know (me included) on roughly how long something is going to be down for. Especially if you're the type that is a bit too addicted to hitting refresh (coming from someone who now past the 20k posts point 🤣). 

[SiC]

On 13/03/2024 at 00:12, SiC said:

Irritatingly the forum software (Invision Power Board) has just pinged up an update! Its a security update too but thats only in the Commerce part of the forum software package - we don't pay for that or even have it installed. 

I don't really want to apply that update along with this migration at the same time. Doing too much on an update is a receipe for things to fuck up and go wrong in my experience. So I will likely apply that in the next day or so before the migration. Just to ensure that is all stable before doing a big switch. However I don't imagine the downtime from that will be more than half hour or so.

This has been on my mind and I really want to get this update on here before the big migration. Then there is a full day to make sure that is (strong and) stable and any new changes aren't related to that update.

Sorry its a bit short notice, but any objections if I apply this in around 8:30pm tonight? Probably will be around half hour outage.

[SEATMad]

5 minutes ago, SiC said:

This has been on my mind and I really want to get this update on here before the big migration. Then there is a full day to make sure that is (strong and) stable and any new changes aren't related to that update.

Sorry its a bit short notice, but any objections if I apply this in around 8:30pm tonight? Probably will be around half hour outage.

Gotta do what you gotta do.. No objections from me. Keep up the good work @SiC your doing a grand job! 

[Dave_Q]

44 minutes ago, SiC said:

Maybe moan is not the best word for me to have used. More that I know that people always like to know (me included) on roughly how long something is going to be down for. Especially if you're the type that is a bit too addicted to hitting refresh (coming from someone who now past the 20k posts point 🤣). 

No worries, I know you're not trying to have a go at anyone, I just wanted to put it on record what a professional job you're doing compared to where we were, just in case anyone does feel like having a moan.

[High Jetter]

Go 4 it if you have the time. Thanks

[SiC]

Todays small bit of work is all done. Nice and easy quick one.

Tomorrow is still looking likely to be the migration day. Not sure what the start time will be - depending when I finish my day job. I imagine it'll be around 6pm or slightly earlier.

I reckon it'll probably be a 2hr job. Most of that will be waiting for the ~220gb of content to copy across.

[egg]

Thanks for that, given I have 60GB of photos and videos filling up my virtual life, 220 for this forum doesn't sound too bad!

[SiC]

17 minutes ago, egg said:

Thanks for that, given I have 60GB of photos and videos filling up my virtual life, 220 for this forum doesn't sound too bad!

Its not terrible but usage is jumping up in recent years. Remember that we knock the resolution down a fair bit and compress it relatively heavily too!

This is the last year or so of uploads:

2.0G    monthly_2022_12
2.2G    monthly_2023_01
2.0G    monthly_2023_02
2.4G    monthly_2023_03
2.8G    monthly_2023_04
2.8G    monthly_2023_05
2.4G    monthly_2023_06
2.5G    monthly_2023_07
2.7G    monthly_2023_08
2.5G    monthly_2023_09
2.9G    monthly_2023_10
2.7G    monthly_2023_11
3.9G    monthly_2023_12
4.0G    monthly_2024_01
3.1G    monthly_2024_02
1.7G    monthly_2024_03

We seem to be averaging around 20k images uploaded per month in the last year or so.

Each 1GB of extra images costs €0.102392 per month to both store and backup. As we never delete images, that is a cumulative amount. So last years images added €3.26 onto the hosting bill every month or €31.80 a year.

Our current (recent) problem is screenshots. They are uploaded as PNGs which aren't compressed as well as JPEGs. With most phones having hi-resolution screens, they are big images too. The plan is to recompress them all as JPEGs automatically and hopefully chop down the usage to a being a bit more reasonable.

[High Jetter]

When I take pics on my phone I always transfer to PC, and edit in ACDSee before posting. Shrinks the file size dramatically.

[egg]

39 minutes ago, SiC said:

They are uploaded as PNGs which aren't compressed as well as JPEGs.

ah well, I still use a 5m pixel 15 year old camera for most of my uploads here, so keeping the resolution as shite as poss 😆

[SiC]

9 minutes ago, High Jetter said:

When I take pics on my phone I always transfer to PC, and edit in ACDSee before posting. Shrinks the file size dramatically.

You really don't need to do that with JPEGs on here. The site does that all automagically when you upload them. 

[Snake Charmer]

Thanks for the info, I will think twice about photos and screenshots in future as I'm guilty as charged my lord. 

[jim89]

9 hours ago, SiC said:

Its not terrible but usage is jumping up in recent years. Remember that we knock the resolution down a fair bit and compress it relatively heavily too!

This is the last year or so of uploads:

2.0G    monthly_2022_12
2.2G    monthly_2023_01
2.0G    monthly_2023_02
2.4G    monthly_2023_03
2.8G    monthly_2023_04
2.8G    monthly_2023_05
2.4G    monthly_2023_06
2.5G    monthly_2023_07
2.7G    monthly_2023_08
2.5G    monthly_2023_09
2.9G    monthly_2023_10
2.7G    monthly_2023_11
3.9G    monthly_2023_12
4.0G    monthly_2024_01
3.1G    monthly_2024_02
1.7G    monthly_2024_03

We seem to be averaging around 20k images uploaded per month in the last year or so.

Each 1GB of extra images costs €0.102392 per month to both store and backup. As we never delete images, that is a cumulative amount. So last years images added €3.26 onto the hosting bill every month or €31.80 a year.

Just start charging @SEATMad👍🏼😁

9 hours ago, SiC said:

 

 

[inconsistant]

It’s awesome that you’re doing this. 

GLWACOY.!

[SiC]

9 hours ago, Snake Charmer said:

Thanks for the info, I will think twice about photos and screenshots in future as I'm guilty as charged my lord. 

Nah carry on as others will do it anyway (including me!). I don't want people to stop posting as it's the whole point of the forum. It's just a technical issue that I need to figure a way to properly address soon. 

[Remspoor]

Do we now call this forum the M25-shitroad Forum? There is a late Friday hiatus associated with them .
Good luck I will be keeping my fingers crossed . 🫰

[SiC]

Who books a meeting at work late on a Friday evening? FFS

Anyway I'm off to the gym first and I'll kick it off when I'm back. Expect it to be around 6:30p to 7p. I'll give a 15mins to 30mins warning before I pull the plug just so anyone can finish writing up a post/PM they might be writing. 

Most of the time will be spend with me sat drumming my fingers waiting until the content is restored. 

[High Jetter]

27 minutes ago, SiC said:

Most of the time will be spend with me sat drumming my fingers waiting until the content is restored. 

We'll all be doing the same! Fingers X it goes smoothly for you.

[grogee]

On 13/03/2024 at 00:01, SiC said:

So note that down now.

 

[grogee]

Good luck @SiC, may the force be with you.

I've just tried setting up a new router but failed miserably so I can't imagine what this new bus business is all about. 

Thanks for your efforts. 

[SiC]

1 hour ago, SiC said:

I'll give a 15mins to 30mins warning before I pull the plug just so anyone can finish writing up a post/PM they might be writing.

This is the 15 minute warning!

[tooSavvy]

"Anything can happen in the next ### minutes!"

#Thunderbirds

👍

[High Jetter]

1 minute ago, SiC said:

This is the 15 minute warning!

Perfect timing for me, off for dinner & TV shortly.  #Selfish

Hope it goes easily for you.

[SiC]

Just now, High Jetter said:

Perfect timing for me, off for dinner & TV shortly.  #Selfish

Hope it goes easily for you.

Well I've just come back from the gym and Mrs SiC is putting tea on (and baking a cake so it seems)