Posted by afilipovski 16 hours ago
Software like https://restic.net/ does a good job. Few choices of backend
Accepts stuff from pipe too so you can just pipe mysqldump or pg_dumpall without intermediate files
Very decent options for checking repo integrity, personally (well, at work too, we use it on few hundred user machines and servers) I also added "roll a dice for backup and try to restore it" test script to make sure it is working
Decent deduplication too so "store year's worth of weekly snapshot" is very sensible strategy once you exclude the "constantly changing and useless" (caches etc.) files out of it
can mount FUSE directory with all backups on sensible OSes.
https://kopia.io/ does the same +GUI but a bit worse on CLI front (it insists on keeping local config which makes scripting a bit more involved, but not much)
IMHO, the easiest way to perform a restore test is to use your production backups to refresh a lesser environment on a weekly cadence. Naturally this has to be appropriate to the type of data you're restoring, e.g., our E2E (end to end) testing environment has been built to the same risk tolerances of production so it's suitable for production data and the PII (Personally Identifiable Information) it hosts. If this isn't in your own risk tolerances, you can still perform the same test but ensure the data from prod is anonymised or alternatively destroyed, and perform a second restore after production restore with the dataset that preceeded it that has no PII data or similar.
With this the E2E tests confirm the backups are expected and you can tick a box that when shit does eventually hit the fan, data corruption isn't a problem. Normal caveats apply, you must make sure your backups are immutable so they cannot be modified or changed _after_ the tests have been ran.
uh no, in a company, as a sysadmin, you try your backup files at least every month
For example:
Requirement:
"Just copy a file from folder A to folder B."
Minimal translation:
"Implement the capability to copy any file, in any format and of any size from SharePoint, located at a configurable path, with appropriate authentication and access control checks, then stream it in chunks to a different, configurable path inside an S3 bucket, also with appropriate authentication and access controls in place. Ensure that any disruption in either service which may occur while the file is streaming can be recovered from at the point it failed instead of having to restart from the beginning. Ensure that the retry mechanism is built-in and that the retry window is configurable; if the file cannot be copied within the specific time window, then an error should be sent via email to a configured email address. Ensure that the entire transfer is encrypted in transit... If the file happens to be a folder, then you must copy across all of its contents recursively up to a certain configured MAX_DEPTH to avoid DoS and ensuring that the system does not get caught in an infinite loop due to symlinks pointing to a parent folder... In this case, send an email to the configured address... Etc... Etc..."
And the thing is; if you tell AI "Just copy a file from folder A to folder B." - It will not meet your 'basic' needs because even if it does a great job at filling the gaps in your requirements, it will still take shortcuts. In order for an AI to avoid taking shortcuts, it would have to make you fill out a questionnaire and make you sign up for and configure services; it would not be a pleasant user experience. The user experience cannot be pleasant, because the AI cannot read your mind and it cannot know your intent.
"Just copy a file from folder A to folder B."
All that access control and authentication and symlink nonsense? That's a self-inflicted problem that exists only in enterprise, and shouldn't be assumed - much less created - for regular users.
So who are we? A household with a localhost administrator?[0]
Then you are surely doesn't need encrypted, chunk-level corporate level bla-bla-bla.
You need a Syncthing copy to some other device not at home - and an additional backup procedure to maintain the history and protect against PEBKAC errors - which can run on your local Syncthing copy or/and the other one.
And no, nobody needs your 555GB of RAWs of sunsets/flowers/precious_family_moments you shot - not even you.[1]
[0] well considering the tone of TFA and most of the comments here
[1] that's Instagram/Google Photos/iCloud for nowadays, totally with "Remember this day N years ago?"
In my case, yes. I've gotten into the trap of building a homelab that over time became pretty important, more like a small prod environment. We take for granted the "stability" that cloud services provide (until the terms of service change and you are screwed over by either loss of privacy, loss of access, etc). Self-hosting these things yourself exposes you to this complexity.
Photos aren't that important, how about your passwords and documents and the whole setup around all this?
I also do a part of this just for the love of the game.