Slrpnk.net outage
Uncategorized
96
Beiträge
52
Kommentatoren
379
Aufrufe
-
Alt account of poVoq@slrpnk.net here.
Our instance is currently down and I can't get remote access to the servers. It appears that there might have been a hardware failure of the main firewall, which is the one thing I can't work around remotely.
I am still trying a few things, but I am not very optimistic that I can get access.
The really unfortunate part is that just now I am on one of my rare work deployments abroad, so I also can't access it physically during the next few weeks and my usual back up that could restart it is not available either.
As something like that never happened in 3 years operating the servers, I thought I can risk it, but murphy's law seems inescapable
I will try to keep you posted here on any updates, but probably there will not be much I can do for a while. Really bad timing
Edit: we might use this "opportunity" to migrate the instance to Piefed, which has been an idea for quite some time now. I will keep you posted on that.
I this it’s a law that servers run 100% perfect until the literal day one leaves town with zero way to return home. One of the many reasons I got all my services off of unraid.
Very cool to learn you’re running your own machines. Do you go into detail about this anywhere?
-
Work trip and server problems. Where have I seen this before?
I've had the opposite--personal trip and work server problems.
-
Well, on the plus side lots of lessons learned and I think I might move at least the xmpp server to an external vps to have a backup communication channel.
Might it also be possible/feasible to send an "email blast" to everyone with a slrpnk.net account? It would probably be quite complicated to send that many emails at once without getting blocked by the major email providers, but hypothetically we've all registered with emails...
Just spitballing ideas for the future. Everything's fine now. Gives me an excuse to spend a little more time on the Canadian end of the fediverse.
-
I this it’s a law that servers run 100% perfect until the literal day one leaves town with zero way to return home. One of the many reasons I got all my services off of unraid.
Very cool to learn you’re running your own machines. Do you go into detail about this anywhere?
We have a small write up about the hardware on our wiki, but it is also down right now.
I think we will share a post-mortem write up of the actual improvements we will do to avoid this in the future.
One thing I will definitly do is to add a KVM remote management console to one of our server boards and move the main firewall into a VM with hardware passthrough of the NICs (this was anyways planned for a 10gbit network upgrade for the second half of 2025). This way I should be able to reboot and even reinstall the main ingress point remotely, so that only the fiber gateway remains as a failure point that requires physical access.
-
Might it also be possible/feasible to send an "email blast" to everyone with a slrpnk.net account? It would probably be quite complicated to send that many emails at once without getting blocked by the major email providers, but hypothetically we've all registered with emails...
Just spitballing ideas for the future. Everything's fine now. Gives me an excuse to spend a little more time on the Canadian end of the fediverse.
As email in profile is an optional feature on our instance about 50% of the userbase opted to not share one so we would only reach some people. But yes, getting spam filtered would be another problem of mass-mailing.
But I found a way to put up a simple html error page and will add this later to inform our members and link to this thread.
-
pretty hands-off
Given that the single admin @SDF@lemmy.sdf.org hasn't been active in two years, yeah, I'd say that counts as "pretty hands-off". The SDF organization as a whole will probably be around for decades to come, but they don't seem to care that much about their Lemmy instance.
They have stepped in when a lot of people complained, it took their whole instance complaining to do so while a Nazi troll spammed hate across the local feed.
-
This is making me want to create an alt on slrpnk.net , since you're so chill and communicative about issues.
Yeah me too, I'd wanted to for a while but never got around to it. Now I probably never will (unless they get the servers back up at some point in the future).
-
Alt account of poVoq@slrpnk.net here.
Our instance is currently down and I can't get remote access to the servers. It appears that there might have been a hardware failure of the main firewall, which is the one thing I can't work around remotely.
I am still trying a few things, but I am not very optimistic that I can get access.
The really unfortunate part is that just now I am on one of my rare work deployments abroad, so I also can't access it physically during the next few weeks and my usual back up that could restart it is not available either.
As something like that never happened in 3 years operating the servers, I thought I can risk it, but murphy's law seems inescapable
I will try to keep you posted here on any updates, but probably there will not be much I can do for a while. Really bad timing
Edit: we might use this "opportunity" to migrate the instance to Piefed, which has been an idea for quite some time now. I will keep you posted on that.
Thanks for all that you do for us and this community! A little downtime won't hurt anyone. I would also be in gavour of PieFed migration
-
Thanks for all that you do for us and this community! A little downtime won't hurt anyone. I would also be in gavour of PieFed migration
I'd probably make an account there if it moves to piefed.
-
I will no longer read that domain as "slurpink" thank you
Oh you will. I know the name for a few months now and I still read it slurpnik
-
Yeah me too, I'd wanted to for a while but never got around to it. Now I probably never will (unless they get the servers back up at some point in the future).
it seems they will, just not too soon
-
Alt account of poVoq@slrpnk.net here.
Our instance is currently down and I can't get remote access to the servers. It appears that there might have been a hardware failure of the main firewall, which is the one thing I can't work around remotely.
I am still trying a few things, but I am not very optimistic that I can get access.
The really unfortunate part is that just now I am on one of my rare work deployments abroad, so I also can't access it physically during the next few weeks and my usual back up that could restart it is not available either.
As something like that never happened in 3 years operating the servers, I thought I can risk it, but murphy's law seems inescapable
I will try to keep you posted here on any updates, but probably there will not be much I can do for a while. Really bad timing
Edit: we might use this "opportunity" to migrate the instance to Piefed, which has been an idea for quite some time now. I will keep you posted on that.
I appreciate the update was thoroughly confused
-
This is making me want to create an alt on slrpnk.net , since you're so chill and communicative about issues.
He's pretty awesome. He has a monthly thread with updates and actually responds to you too.
-
I miss BBS.
you can still have it with NodeBB!
-
Are there modem sounds to play air guitar to?
-
Are there modem sounds to play air guitar to?
beep beep boop boop boop boop beep... EEEEEEEEEEeeeeeeee awwwwwww ka-dong ka-dong da kshhhhhhhhhhhhhhHHHHHHHHHHHHHHHHHHHHHHHHHH
eerie silence
