The mouse did not survive interrogation.
Also mouse not pictured.
The backup server at my parents house really did not want to run. Something kept breaking to the point that no web interface or SSH worked. Now I’m not new to self hosting, so I ran my fair share of HDMI and USB cables across my space to debug something on my server. But at my parents place it’s just that extra bit annoying, since I only bring my laptop. Finding a screen is doable, but the keyboard got me more than once. The only one they have kicking around has a freaking PS/2 cable… not even my salvaged backup hardware is old enough to support that. More than one time I had to wait for friends to get back from work so I could borrow their keyboard just to press ‘any’ key.
I decided to handle the issue by getting an IP KVM, which should help me deal with that, right? Wrong! I installed a new drive and that piece of hardware goes into some secure boot loop or whatever that is too fast for my KVM to turn on…
Not on a computer but I had a JVC TV that started blanking a few times a minute whenever my dvd player was connected. Whilst trying to fix this I attempted to update the TV firmware. That didn’t fix a thing but I noticed that the blanking went away when the USB drive with the firmware update was plugged in. So I just plugged a cheapo USB drive in and left it there for as long as I owned that TV and dvd player.
I used to have a Ryzen 2700X based system as a server. At the time (no clue if it’s still an issue), there was a bug in the interaction between early Ryzen chips and Linux that caused the OS to lock up intermittently. I used a ATmega32u4 board to connect to the reset switch on the PC and as a USB device with a systemd unit that sends a ping to the board every 30 seconds. If the board doesn’t see a ping for a minute, it reboots the PC. It was janky as hell, but it worked for 2+ years!
so you reinvented watchdog?
That reminds me of the old story where they positioned a PC opposite from the server and taped a pen to the CD drive so that opening the CD tray presses the reset button of the server. The PC would automatically detect when the server freezes and open the CD tray. Probably just a legend but still a great example for how a silly and simple solution that just works can often times be as good if not better than a sophisticated but over engineered solution
Way back in the olden days, we had a hard drive with out any backups go bad. I determined that it was the circuit board. So I took a working drive and basically connected the bad driver’s platters to the working drive’s circuit board.
It worked and I was able to back up the data and restore it on a new drive.
My WAN connection would just randomly start losing packets and would not recover, not full disconnect just lots of retries. Didn’t impact LAN just WAN. Unplugged the cable or disabling then re-enabling the interface would fix it for a coupe of days to a week then it would come back.
Time consuming to diagnose as switching cables, ports, etc would require waiting days to see if it fixed it.
Eventually had the ISP come out and swap the fiber media converter and fiber integrity. But again it came back.
After several weeks I stumbled on a FreeBDS bug that was exactly my issue…I was running OpnSense and my gateway had I225s. Applied the scripts and it worked.
“not enough space in the used server for a standard power unit”
I wanted to put a dedicated graphics card in there for interference experiments - but the server is, well, a server - and those high availability PUs are expensive and a hell to cable manage
But a Dremel I had at hand … And a second PU as well. Now my server has a second power unit which is only powering the graphics card and is short circuited to be always-on.
self-made problems require self-made solutions.
mouse not pictured
Got my hopes up for nothing!
I stared at that picture like it was a Where’s Wally for like 10 minutes before I came to check the comments and saw this note in OP’s message…
We are cut of the same cloth.
It’s OK I gotchu
Fun fact - he’s called that because this same thing happened to him with a dead mouse in the pc case.
it is frying here during summer in Australia. I have to buy a seperate airconditioner for the server room and pipe it into the intake of my home lab/nas so it would not shut down on me
My last home I put the server in the closet with my heat pump water heater. They pretty much balanced each other out.
Optical or roller ball?
I have a strong suspicion that the answer is mammilion
That sounds like a lot. Is it more than a bajillion, or less?
Many cells
I got a little fanless atom PC from work that was being tossed because it “didn’t work.” It did work you would just have to apply pressure to it before boot because the CPU must have been coming unsoldered. I just stuck it on the bottom of like 30 pounds of gear and never had a problem.
How on earth did you diagnose that one?
There was an era where the BGA chips giving up the ghost was a common enough occurrence and pushing on things was a relatively normal troubleshooting step
Not exactly self-hosting, but I’ve worked at two different academic labs that had unique issues and small budgets.
The first was on top of a mountain at 3km elevation. The low air pressure meant that all hard drives had a lifespan of no more than a year before starting to have issues. This was in the pre-SSD days. The “solution” was to boot from a floppy, never compress data, and have a student return the drives to a low elevation and swap them for the prior ones every six months.
The second was at a place that was using old Datacube MaxVideo hardware that we would never have the budget to replace. It would crash if the room temperature got above about 18C, so we set the thermostat to 15C. Which doesn’t sound too bad until you’ve tried sitting at a desk for 8 hours where everything is 15C and the heat is just slowly leaving your body into the chair, the desk, and the keyboard. And then you walk outside where it is 32C and instantly start sweating because you’re still in several layers and fingerless gloves.
Back in the day of BBS’ my buddy had a wonky modem. If I picked up phone , pursed my lips and did a squeaky sound the modem worked and I could log into BBS.
No mouse problem, but I’ve had to take measures to prevent cat-related downtime. Cats and power-button at the top is not a good combination.
I have a small piece of acrylic sheet taped over the power and reset buttons on my server for that reason. At least I don’t have to worry about mice.
Meowy Guard. https://en.wiktionary.org/wiki/molly-guard
They make sprung flip-top covers specifically for this reason. It’s just a little adhesive square/circle that you stick over the power/reset button, and it has a flip-top that automatically springs closed again.
When I had this problem I disconnected the power button on the motherboard.
Cant you just assign a different action to that power button instead of messing with it internally?
Connect to spray bottle aimed at… well nope.
You can for a click, but no way for the long press, as that’s handled on a lower level
You got a point. Cats are sneaky 😅
yeah what if cat does a
sudo shutdown -r nowthen we’re really in trouble
Did you reconnect it everytime you had to power it on?
A lot of newer motherboards have a separate power button next to the case pinouts.
Nah just jam a screwdriver in there to jump the pins.
You shut down your PC though?
On one computer I had a recessed reset button, so I used that as power.
Another computer I added a wire to an external button double stick taped to the back of the monitor so I can turn on both at the same time.
You sound like you are one step short of wiring up a giant lever to dramatically pull down when you want to power on the computer.
I would like and subscribe
Oh, my housemates’ cat hits the front power button when he leaps off my tower. I’ve got some leather taped over it. My uptime approaches 1! I love your little weight setup, too!
Instead of cat proofing, you were supposed to get a server room dog…
I have two server room dogs. An English pit, which is my seizure dog, and a Jack Russel. They shed, especially in the summer months, and the floor sweeper gets a daily work out, as well as keeping it out of equipment enclosures. It’s not that I hate cats, they are nice as a pet in your house, just not mine. LOL
My radiator is on top so they’ve learned that’s about the only surface in the house where they’re not allowed to sleep
Aha, I disabled the LEDs by unplugging them from the motherboard, you could do the same with the power sw plugs
I lost a raspberry pi for a couple months. Just couldn’t find the darn thing. It was working fine serving away.
Turns out it was in the basement. I forgot I moved it to be closer to my tilt Hydrometer.
Oh, good thing you wrote this. I’m about to give one of my printers away and there’s still a pi inside
Cups?
OctoPrint (3d printer)
Ooo, another bash mirror. Nice!
YEAH it’s bash.org
To be fair, those things are really tiny!
For a while, the windows machine I was using kept randomly becoming unresponsive (both via network, as well as the keyboard+mouse) and I just could not figure out why. Everytime it did, I’d have to force a shutdown by holding the power button until it shutoff, then press it again to start back up. Sometimes this happened while I wasn’t home, so it had to just stay offline until I was.
I got sick of having to do this; so I replaced the power button with a transistor and a RPI. The PI would ping the server every 5min; if it failed to get a response 3 times in a row, it would trigger the transistor for 10sec, release for 3sec, then press and release again; forcing a poweroff then starting the machine again. It’d also write these events to a log file. I think it was around twice a week ish.
Never did figure out what the cause of the lockup was; but it stopped when I replaced windows with Debian.
I just wrote a comment about a very similar setup! Glad I wasn’t the only one haha
Yeah getting rid of windows solves many issues, LOL. I have always refused to run any server with windows, outside of being forced to at work. 😁
Note to any future lemmings who find this thread troubleshooting the same symptom…
- Rule out unswitched (non-chipset, direct CPU lanes) PCI connectivity and power issues by testing without specific PCI components. Unstable PCI errors can be missed during post and present later after boot, and Windows is especially bad at handling it transparently. Common culprits include GPU sag, dubious riser extensions, and anemic PSUs
- If there is a “high bandwidth” feature toggle for RAM in your BIOS settings that is currently on, try turning it off
- If your system drive is an NVMe blade with a Phizon controller, check if the manufacturer software has a firmware update available.
- If the freeze is regular but not persistent, try turning the polling rate down on your mouse, and if it’s wired, try to make sure it’s plugged into one of the board’s SOC-hosted USB 2.0 ports
Had a zyxel nas that got unresponsive after 10-15days of uptime, thankfully it hat a scheduled shutdown/reboot setting. Runs since 2014 with a reboot every Mondy 3am.
Heh, my router is on one of those timer plug things. No matter how frozen, it will reboot.
I’ve got an extra and don’t know what to do with it. I feel like its bad to pull power unexpectedly once a week though. Especially with a single HDD storage setup. I’d like to setup a RAID config, but its a 28TB drive and additional drives would cost me about $$$$$$$.
I have a NAS that had a similar problem, every week or two it would stop responding. From the logs it was clear that the machine was running fine after going non-responsive, but either the NIC or something else network related was crashing. However, the logs weren’t capturing the exact issue. I wrote a script to ping Google every 10 minutes, write a bunch of debug data to disc if it failed, and then reboot.
And it has never happened in the year since. I have no idea what the root cause was, but sending those pings out seems to have fixed it.
I also had a machine that would randomly crash on Windows, but not on Linux.
I’m pretty sure it was some sort of hardware failure, as it also sometimes failed to boot Linux. But if it booted Linux, it would be stable until I shut it down.
Sometimes Linux handles faults better than Windows
Typically dmesg shows error when the hardware is acting up
Never did figure out what the cause of the lockup was
the mysteries of closed-source software
Using an RPi with a transistor almost seems professional… ITAPPMONROBOT
