new NIC for the storage server: a HP-branded Mellanox ConnectX-4, which I'll be crossflashing to stock CX4 firmware too
2x 25Gb/s SFP28 ports for only $30 USD!
you'd probably have a hard time getting SFP+ NICs for this cheap, even though SFP28 ports are faster and compatible with SFP+ transceivers too
2x 25Gb/s SFP28 ports for only $30 USD!
you'd probably have a hard time getting SFP+ NICs for this cheap, even though SFP28 ports are faster and compatible with SFP+ transceivers too
Forwarded from zhongfu
cheap taobao 86 wall face plate
🥰1
finally got everything tidied up nicely. the fiber's got a bit of a sharp bend in the wall boxes, but it probably shouldn't be a problem at all — light's still going through anyway, and even a 2km transceiver is a bit overkill for a ~50m connection
🥰1
installed 2.5G NICs into two of my Proxmox MFF nodes (EliteDesk 800 G4 DMs), and hooked them up to the 2.5G/SFP+ Hasivo switch I got a while back. I forgot to get a picture, so have this picture instead: on the bottom left is the old VGA flex I/O card from one of them, and the other thing is an old 1G NIC
these are made by a seller on Taobao, who makes these m.2 cards with 1G/2.5G PHYs, along with magnetics/eth jacks on another PCB, with variants for different models of mini desktops (which really reduces the jank factor)
they're also ~$20 each (and are PCIe, of course), so you don't have to deal with USB jitter and unreliability, or pay $200+ for the unobtainium tb3 flex I/O cards + a tb3 NIC
these are made by a seller on Taobao, who makes these m.2 cards with 1G/2.5G PHYs, along with magnetics/eth jacks on another PCB, with variants for different models of mini desktops (which really reduces the jank factor)
they're also ~$20 each (and are PCIe, of course), so you don't have to deal with USB jitter and unreliability, or pay $200+ for the unobtainium tb3 flex I/O cards + a tb3 NIC
(graph of traffic served on ports 80/443 on my internet connection)
haven't updated this for a while, but I guess for the past few days, i've mostly been working on relying less on my main webserver (nginx) serving and proxying requests on ports 80/443 to other devices on my network
for various reasons, the container containing the main webserver is restricted to running only on my storage server, so if I have to take that down for whatever reason, everything else (including services running on other containers) will become inaccessible externally as well
now, i've moved to running nginx on each container that has services (that i'd like to have accessible from the internet on the HTTP/HTTPS ports) for SSL termination and whatnot, and a small haproxy container to proxy TCP streams directly to those containers based on the Host header(s) (for HTTP) or SNI (for HTTPS). there's nothing running on the haproxy container, so i can just let it migrate between my proxmox hosts as required
this also means traffic hitting other services on my networks don't have to pass through the main webserver (or the haproxy container), which is always nice to have
and for services that aren't strictly HTTP/HTTPS (which nginx wouldn't be able to handle, e.g. strelaysrv for Syncthing) or services that manage TLS certificates on their own (e.g. MangaDex@Home), I can now serve them on port 443 so they're accessible even on restrictive networks, woo!
haven't updated this for a while, but I guess for the past few days, i've mostly been working on relying less on my main webserver (nginx) serving and proxying requests on ports 80/443 to other devices on my network
for various reasons, the container containing the main webserver is restricted to running only on my storage server, so if I have to take that down for whatever reason, everything else (including services running on other containers) will become inaccessible externally as well
now, i've moved to running nginx on each container that has services (that i'd like to have accessible from the internet on the HTTP/HTTPS ports) for SSL termination and whatnot, and a small haproxy container to proxy TCP streams directly to those containers based on the Host header(s) (for HTTP) or SNI (for HTTPS). there's nothing running on the haproxy container, so i can just let it migrate between my proxmox hosts as required
this also means traffic hitting other services on my networks don't have to pass through the main webserver (or the haproxy container), which is always nice to have
and for services that aren't strictly HTTP/HTTPS (which nginx wouldn't be able to handle, e.g. strelaysrv for Syncthing) or services that manage TLS certificates on their own (e.g. MangaDex@Home), I can now serve them on port 443 so they're accessible even on restrictive networks, woo!
in other news, open this in chrome for a truly horrifying experience: https://cdn.jsdelivr.net/gh/osnr/horrifying-pdf-experiments@master/breakout.pdf
https://github.com/osnr/horrifying-pdf-experiments
(tl;dr: the PDF specification specifies functionality including embedded Flash, 3D objects, and JavaScript support, which you can see in action in that pdf above)
https://github.com/osnr/horrifying-pdf-experiments
(tl;dr: the PDF specification specifies functionality including embedded Flash, 3D objects, and JavaScript support, which you can see in action in that pdf above)
❤1
(pic: State Railway of Thailand Special Express No. 46 sleeper train car, with upper bunk compartment unfolded but not set up yet)
recently decided to go on a trip to Bangkok by train, and it's a pretty decent experience if you've got the time to spend (or waste)
my journey was split up into four legs:
- JB Sentral to Gemas on KTM Intercity
- Gemas to KL Sentral on KTM ETS
- KL Sentral to Padang Besar on KTM ETS
- Padang Besar to Krung Thep Aphiwat on SRT Special Exp No. 46
I spent a bit more than I needed to, but it still ended up costing a bit less than a direct flight to BKK/DMK, and it's one of those things that'll be harder to do when you no longer have all that time to burn for fun :)
recently decided to go on a trip to Bangkok by train, and it's a pretty decent experience if you've got the time to spend (or waste)
my journey was split up into four legs:
- JB Sentral to Gemas on KTM Intercity
- Gemas to KL Sentral on KTM ETS
- KL Sentral to Padang Besar on KTM ETS
- Padang Besar to Krung Thep Aphiwat on SRT Special Exp No. 46
I spent a bit more than I needed to, but it still ended up costing a bit less than a direct flight to BKK/DMK, and it's one of those things that'll be harder to do when you no longer have all that time to burn for fun :)
🥰2
zhongfu's inane ramblings
this thing allegedly suffers from port flaps (which is why it was so cheap... for a ccr2004-1g-12s+2xs), so I'm gonna be looping traffic between the ports all night and see if it still does that (I bet it's because it's running routeros 6 though)
finally got the replacement sunon fans! the 4-pin PWM ones had mount holes that were a bit too big for the original screws, so I gave up on using them — at least I still had the 3-pin ones, which were going to go into the CCR1036
it seems like the CCR2004 comes with 3-pin fans (probably voltage control w/ tachometer then), and plugging the 4-pin sunon fans in seems to result in no tach readings. maybe I gotta reboot the router for it to work properly, but it works, so whatever
the new fans are a lot quieter (but to be fair, they are less deep and draw only up to 1/3 of the power). you could probably run the CCR2004 fanless even in the insanely warm cabinet I chucked everything in, but I don't know if that's healthy for the transceivers and PSUs, so I just let it run at ~2500rpm minimum speed. but even at full speed (~5000rpm), the fan sound is quite imperceptible, especially relative to my old CCR1036
it seems like the CCR2004 comes with 3-pin fans (probably voltage control w/ tachometer then), and plugging the 4-pin sunon fans in seems to result in no tach readings. maybe I gotta reboot the router for it to work properly, but it works, so whatever
the new fans are a lot quieter (but to be fair, they are less deep and draw only up to 1/3 of the power). you could probably run the CCR2004 fanless even in the insanely warm cabinet I chucked everything in, but I don't know if that's healthy for the transceivers and PSUs, so I just let it run at ~2500rpm minimum speed. but even at full speed (~5000rpm), the fan sound is quite imperceptible, especially relative to my old CCR1036
so with that, i've finally swapped my old router (the CCR1036) out for the CCR2004. it's definitely a downgrade in terms of CPU performance:
- the CCR2004 tops out at ~14Gbps of throughput with IP filter rules — definitely no match for the CCR1036, which can do ~22Gbps with its 36 cores. that'd be the main bottleneck if I was pushing download and upload hard on my 10Gbps internet connection
- WireGuard (which I do use quite a lot) is definitely no exception either, and I don't think there's any form of hardware offload that I can make use of, so hitting even 1Gbps might be hard (especially if I'm also pushing other normal traffic)
but those trade-offs are pretty acceptable, since I'd probably not come close to saturating my internet connection(s) yet (?), and the extra SFP+ (and SFP28!) ports I get on the CCR2004 means I could get another one of those cheap 2.5Gbps broadband connections without having gigabit Ethernet ports limiting them to 1Gbps.
and of course, the CCR2004 draws less power, is much more modern, and has a massive heatsink in the back that lets it run fanlessly too
- the CCR2004 tops out at ~14Gbps of throughput with IP filter rules — definitely no match for the CCR1036, which can do ~22Gbps with its 36 cores. that'd be the main bottleneck if I was pushing download and upload hard on my 10Gbps internet connection
- WireGuard (which I do use quite a lot) is definitely no exception either, and I don't think there's any form of hardware offload that I can make use of, so hitting even 1Gbps might be hard (especially if I'm also pushing other normal traffic)
but those trade-offs are pretty acceptable, since I'd probably not come close to saturating my internet connection(s) yet (?), and the extra SFP+ (and SFP28!) ports I get on the CCR2004 means I could get another one of those cheap 2.5Gbps broadband connections without having gigabit Ethernet ports limiting them to 1Gbps.
and of course, the CCR2004 draws less power, is much more modern, and has a massive heatsink in the back that lets it run fanlessly too
zhongfu's inane ramblings
new NIC for the storage server: a HP-branded Mellanox ConnectX-4, which I'll be crossflashing to stock CX4 firmware too 2x 25Gb/s SFP28 ports for only $30 USD! you'd probably have a hard time getting SFP+ NICs for this cheap, even though SFP28 ports are…
also got a replacement for this and installed it in my storage server — this time, there's no massive air gap in the thermal paste, so I didn't bother repasting it.
it's sitting at a comfortable 85 degrees now (warning at 105degC, soft shutdown at 115degC, hard shutdown at 130degC), unlike the old card, which got to 115degC pretty quickly. this is despite it sitting between my GPU (a Tesla P100) and 4 NVMe drives! i don't know if it'll get even warmer if I start pushing 25Gbps through it, but that won't be happening any time soon
in any case, if something does go wrong with it, I can always fall back onto the existing 1Gbps connections — it'd suck a little, but it would still work fine
meanwhile, I'll probably wait for my replacement fuses for the old card to come, so I can fix it
it's sitting at a comfortable 85 degrees now (warning at 105degC, soft shutdown at 115degC, hard shutdown at 130degC), unlike the old card, which got to 115degC pretty quickly. this is despite it sitting between my GPU (a Tesla P100) and 4 NVMe drives! i don't know if it'll get even warmer if I start pushing 25Gbps through it, but that won't be happening any time soon
in any case, if something does go wrong with it, I can always fall back onto the existing 1Gbps connections — it'd suck a little, but it would still work fine
meanwhile, I'll probably wait for my replacement fuses for the old card to come, so I can fix it
zhongfu's inane ramblings
today's ($10) gamble: an APC Smart-UPS 750 (from the previous generation, i.e. the SUA750I) seller says it doesn't power up no matter what they try, including "with different power cords". doesn't seem like they've tried anything else (including looking at…
another update on this: i finally got around to installing the replacement batteries (and the network management card) I bought off taobao, only to realize that there's probably something wrong with voltage regulation on the battery charging bits — it's only putting out 22V unloaded, which is way way way too low for 2 lead-acid batteries in series
guess that means i'll have to eventually pull it apart and see which capacitors (or other bits) are failing. not fun :(
(also, the batteries seem to be a hair too wide for the UPS, and I really had to struggle to pull the batteries out after i put them in. oddly enough, this isn't the first time i've had this problem. maybe taobao batteries just have poorer tolerances for physical dimensions? they seem to work fine nonetheless)
guess that means i'll have to eventually pull it apart and see which capacitors (or other bits) are failing. not fun :(
(also, the batteries seem to be a hair too wide for the UPS, and I really had to struggle to pull the batteries out after i put them in. oddly enough, this isn't the first time i've had this problem. maybe taobao batteries just have poorer tolerances for physical dimensions? they seem to work fine nonetheless)
😢2