Yeah… when you pull up stats for Netflix library, you learn some things… Like how little content they actually had. Never cracked 7000 movies… And while that may seem like a lot to a lot of people out there. Those of us that remember blockbuster stores, you ignore like 90% of them cause they’re dumb or silly movies that you’d never watch anyway (or stuff you’ve already watched). Then you can put actual numbers to it… If each of these are full bluray rips (which they’re not as far as Netflix goes) they only take up 175TB… It’s not a lot of movies at all.

https://www.businessinsider.com/how-netflix-movie-and-tv-show-catalog-changed-over-time-2020-2

It’s pretty easy to see how an individual could collect more content than netflix easily. Now add money to the equation… I think it would be possible to collect double or triple netflix easily.

The real fun is going to be when he’s finally up and running… I have ~250TB of data on the Truenas box. Initial sync is going to take a hot week… or 2…

Edit: 23 days at his max download speed :(

Fine… a hot month and a half.

Ceph has been FANTASTIC for me. I’ve done the dumbest shit to try and break it and have had great success recovering every time.

The key in my experience is OODLES of bandwidth. It LOVES fat pipes. In my case 2x 40Gbps link on all 5 servers.

Nah, that’d be mean. It isn’t “simple” by any stretch. It’s an aggregation of a lot of hours put into it. What’s fun is that when it gets that big you start putting tools together to do a lot of the work/diagnosing for you. A good chunk of those tools have made it into production for my companies too.

LibreNMS to tell me what died when… Wazuh to monitor most of the security aspects of it all. I have a gitea instance with my own repos for scripts when it comes maintenance time. Centralized stuff and a cron stub on the containers/vms can mean you update all your stuff in one go

40 ssds as my osds… 5 hosts… all nodes are all functions (monitor/manager/metadataservers), if I added more servers I would not add any more of those… (which I do have 3 more servers for “parts”/spares… but could turn them on too if I really wanted to.

2x 40gbps networking for each server.

Since upstream internet is only 8gbps I let some vms use that bandwidth too… but that doesn’t eat into enough to starve Ceph at all. There’s 2x1gbps for all the normal internet facing services (which also acts as an innate rate limiter for those services).

I could… But it would be a royal pain in the ass to find a VPS that has a clean address to use (especially for email operations).

Exactly this. 2 phase into house, batteries on each leg. While it would be exceedingly rare for just one phase to go out… i can in theory weather that storm indefinitely.

Nope 240. I have 2x 120v legs.

I actually had verizon home internet (5g lte) to do that… but i need static addresses for some services. I’m still working that out a bit…

Fire extinguisher is in the garage… literal feet from the server. But that specific problem is actually being addressed soon. My dad is setting up his cluster and I fronted him about 1/2 the capacity I have. I intend to sync longterm/slow storage to his box (the truenas box is the proxmox backup server target, so also collects the backups and puts a copy offsite).

Slow process… Working on it :) Still have to maintain my normal job after all.

Edit: another possible mitigation I’ve seriously thought about for “fire” are things like these…

https://hsewatch.com/automatic-fire-extinguisher/

Or those types of modules that some 3d printer people use to automatically handle fires…

Absurdly safe.

Proxmox cluster, HA active. Ceph for live data. Truenas for long term/slow data.

About 600 pounds of batteries at the bottom of the rack to weather short power outages (up to 5 hours). 2 dedicated breakers on different phases of power.

Dual/stacked switches with lacp’d connections that must be on both switches (one switch dies? Who cares). Dual firewalls with Carp ACTIVE/ACTIVE connection…

Basically everything is as redundant as it can be aside from one power source into the house… and one internet connection into the house. My “single point of failures” are all outside of my hands… and are all mitigated/risk assessed down.

I do not use cloud anything… to put even 1/10th of my shit onto the cloud it’s thousands a month.

The site is already available in HTTPS. Why would you even serve content non-encrypted?

If you need an education on the matter… Here you go. https://www.cloudflare.com/learning/ssl/why-use-https/

“I don’t handle sensitive information on my website so I don’t need HTTPS”

A common reason websites don’t implement security is because they think it’s overkill for their purposes. After all, if you’re not dealing with sensitive data, who cares if someone is snooping? There are a few reasons that this is an overly simplistic view on web security. For example, some Internet service providers will actually inject advertising into HTTP-served websites. These ads may or may not be in line with the content of the website, and can potentially be offensive, aside from the fact that the website provider has no creative input or share of the revenue. These injected ads are no longer feasible once a site is secured.
Modern web browsers now limit functionality for sites that are not secure. Important features that improve the quality of the website now require HTTPS. Geolocation, push notifications and the service workers needed to run progressive web applications (PWAs) all require heightened security. This makes sense; data such as a user’s location is sensitive and can be used for nefarious purposes.

I don’t feel the need to be your teacher. You can easily google why you should always be using HTTPS. There’s numerous reason… all overwhelmingly obvious. Forget the basic “Not every ISP is an angel, and they all will collect as much information as they can get”. But I already said that… “It’s still best practice to limit sniffing.” Not sure why I need to elaborate any more on that. Very much akin to “why close your window blinds”, because nobody likes a peeping tom.

Ultimately for this specific website it’s literally changing a couple lines of code in their apache or nginx instance (or whatever proxy they’re using). It’s called best practice for a reason.

Edit: Hell it’s even a bit more of a guarantee that your site makes it to the consumer unaltered. Would be odd for that site to have it’s packets intercepted and midget porn be added to every page wouldn’t it? Think that would hurt the guys reputation?

An SSL here is nothing more than security theater, or marketing.

Or like I already said… is best practice.

Why would the lack of SSL concern you?

Because it means my traffic to that site is in the clear. And while we’re not transacting anything sensitive necessarily. It’s still best practice to limit sniffing.

Automatically swapping to https should be default behavior for every website.

I just want to point out the Technitium project as an alternative to unbound and bind resolver as well.

Regardless, it’s really easy to setup your own DNS resolver that resolves to DNS roots.

That’s total draw of the whole rack. No indicative of power per vm/lxc container. If I pop onto management on a particular box it’s only running at an average of 164 watts. So for all 5 processing nodes it’s actually 953 watts (average over the past 7 days). So if you’re wanting to quantify it that way, it’s about 10W per container.

Truenas is using 420 watts (30 spinning disks, 400+TiB raw storage…closer to 350 usable. Assuming 7 watts per spinning drive were at 210Watts in disks alone, spec sheet says 5 at idle and 10 at full speed). About 70 watts per firewall. Or 1515 for all the compute itself.

The other 1000-ish watts is spent on switches, PoE (8 cameras, 2 HDHR units, time server and clock module,whatever happens to be plugged in around the house using PoE). Some power would also be lost to the UPS as well because conversions aren’t perfect. Oh and the network KVM and pullout monitor/keyboard.

I think the difference here is that I’m taking my whole rack into account. Not looking at the power cost of just a server in isolation but also all the supporting stuff like networking. Max power draw on an icx7750 is 586Watts, Typical is 274 according to spec sheet. I have 2 of them trunked. Similar story with my icx7450s, 2 trunked and max power load is 935W each, but in this case specifically for PoE. Considering that I’m using a little shy of 1k on networking I have a lot of power overhead here that I’m not using. But I do have the 6x40gbps modules on the 7750.

With this setup I’m using ~50% of the memory I have available. I’m 2 node redundant, and if I was down 2 nodes I’d be at 80% capacity. Enough to add about 60GB more of services before I have to worry about shedding load if I were to see critical failures.

I run 4 vms that are seedboxes. Each are on vpns and I’ve observed them to hit up to ~90MBps, just shy of gigabit (I have 8gbps). Most of the time I’m not uploading much at all. It’s not just you bandwidth and the VPN that matter. But also the peers that’s making the request. Sometimes they just don’t talk to you but the other peers. Or they don’t have that much download bandwidth. The only real way to test is to self host your own torrent and grab it from a controlled outside peer. See what it actually gets when you pull directly.

On the Sata SSD ceph storage. That’s just live stuff on the containers/vms. I’m at 20% usage of the 70TiB usable at the moment. I don’t use it all that heavily. Because of the way ceph works it’s really ~23 TiB of usable space and ~4.5 TiB written since it writes 3 copies in my cluster.

On the slow storage node it’s running Truenas with 28 spinning disks at 16TB each. 2 hot spares, and 2 ssds each for cache, log, and metadata (eating up total of 36 bays). That’s 342.8TiB usable after raidz nonsense. And I’m 56% usage. I have literally everything I’ve done that I cared to save from like 2005 or 2006 or so. Backups for the ceph storage (PBS). Backups for computers I’ve had over the years. Lots of linux ISOs(105TiB) archived, including complete sets of gaming (37TiB) variants. Oh and my full steam library as well which currently sits at 14TiB. Flashpoint takes up a few TiB as well…

5 node proxmox cluster (each node on 40gbps networking[yes ceph…], ~80TB of SSD storage, 180cores, ~630GB of ram total)
1 slow storage node (~400TB)
2x opnsense servers in HA
2x icx7750s
2x icx7450s

PoE to all the things… and 8gbps internet.

Usually run ~15-17amps. So about 2000 watts. It’s my baby datacenter.

Sometime this month I’ll be installing 25000kwh solar system on my roof and batteries.

As far as heat goes… It’s in the garage with an insulated door, heat pump water heater, and there’s a tripplite ac unit in the bottom of the rack. The waste air(from the a/c) exhausts outside through a direct vent in the wall. The garage is downright tolerable to me for extended periods of time. The servers don’t complain at all.

Reading about all you guys being under 200w or whatever makes me wonder if it’s worth it. Then I realize that the cost to do even a 1/4 of what I do in the cloud is more expensive than buying my solar.

Power costs for the rack is about $100-120 a month. If it wasn’t for solar.

Edit: 75 LXC containers, 22VMs.

“I’m paying with exposure.”

In this case though, advertising otherwise would have actually cost them money… This is the one time that it actually is a “decent” although not “great” argument. The exposure dynamic to companies is completely different than for workers.

Torrents only here… I have 8gbps internet. I’m privileged, so I seed (10x or one year). I don’t see a point to paying to be part of a usenet in my situation. I have a few private trackers I’m on. I should see about getting into a few more though to spread the bandwidth wealth. 4 seedbox vms to roundrobin the new torrents that get added.