Need to organize things

_{Tags:
misc}

I really need to organize my files, every time I try I end up overwhelmed, and the usual "Do it in small steps" suggestion is counterproductive because files are just virtualized data, I can effortlessly shuffle them all around and put them in neat folders if I wanted but that solves nothing and more often than not it just ends up with me having more stuff to sort.

I already do that, I wrote a small go script that takes a folder and just organizes everything into the big SORT folder at /home

user@computer:~/SORT$ ls
 Audios     Minecraft       Software
 Books     'Old Projects'   sort.log
 Fonts      Operation       Texts
 Images     Other           weirdos
 Manually   Programming     Zips

Some are further divided into more folders but it's still flatter than most of the computer, it's still not a good solution by any means, I thankfully discovered very early that running my script recursively is a very bad idea, it's obvious but I didn't realize while I was coding it, but file identification is context dependent, so when I was running it on a small subset of the last files I had downloaded I saw as the folder Factorio/ got completely mangled just before my eyes.

Since I am incredibly paranoid about losing files I made very sure as I was writing the script that nothing I did was irreversible, and properly logged, so I was able to recover things just as they were just by reading the log and just undoing each file move.

Still being delicate with moving files doesn't change the fact that the sorter was now at most half as effective, I can run it non recursively in a folder with no inherent context such as Downloads and it'll just move the Images, Audios, Books, etc to their "respective places", then I have to go to each individual sub-folder remaining and run it again, it wouldn't be that bad if not for the fact that I am just moving the problem out of sight instead of solving it.

Each and every one of those "Respective places" files go to has a problem of it's own, let's list them.

`Texts/`

Overall the most successful, it has simple text files in plain text but aren't scripts, problem is I need to manually move them in from weirdos half of the time.

`weirdos/`

for the files that don't fit anything else I've seen; corrupted files, weird formats and similar, it's not really sorted but it has what it says it has.

`Other/`

Not so weird stuff that is still hard to sort; some saved webpages, some captchas I scrapped when making a solver, some drawings, some backups, the uncompressed PngSuite, some chess .pgns, some Courses, stuff like that.

`Fonts/`

Contains a grand total of 3 different fonts, that would be bad on it's own but it gets worse; none of those fonts is actually "sorted" by being here, for a font to be installed it needs to be in the .fonts folder in the first place, so those 3 fonts are either inert, and therefore useless, or duplicated and therefore actively harmful since when I change OS I take the whole .fonts folder with me anyway and dump it somewhere else.

`Manually/`

It's empty and the script conscientiously avoids putting stuff on it, it's one of the whitelisted paths where I allowed the script to run recursively so the intended use was probably FolderToSort/* -> Manually -> [Sort], but it didn't take long for me to realize that it's far easier to make a new temp folder at FolderToSort and move what I want to automatically sort there, so this has remained empty for all it's existence.

`Operation/`

I've got no clue what that is, it only contains an empty file named nthumbnail.py, the sorter script doesn't mention it at all either.

`Audios/Music/ & Audios/Unsorted/`

Everything new comes in to Unsorted and some of it is supposed to go into Music eventually, but it never does, audio is the one thing I've never managed to handle, I just can't hear the difference between some 36mb lossless .flac vs a -b:a 24k opus, yet I store both in the same place all the same, it is also full of things I don't really know where to put like pink noise, recordings of background and some very short audio clips that probably came from a video, like this: I don't think that should be put in the music folder despite it being indisputably music.

That of course it is paired with the fact that Music/ isn't any good either, it contains 8 folders; 3 are sets I torrented, they come with the cover and separated in neat folders for each disk despite me only caring about 1 or 2 of the songs inside, I won't delete them tho, 4 of them are basically the same except every song is optimized~~???~~, their hashes are all different but their sizes aren't considerably so, I know some have metadata and some don't but they aren't organized by folder, and I can't really compare them because I can't hear a damn difference. Finally the last folder; TODO/ isn't even audio files, it's full of videos that I was going to convert to audio but I never ended up doing so, to begin with I usually listen to music from video files in the first place so there is little point of converting them to audio, but I can't (Well, I don't want to) do the same the other way around, so they will stay as audio.

`Books/`

Has like 6 books and the rest is random .pdf files that can't really be called books, wouldn't be so bad if not for the fact that I have some 30gb worth of books somewhere in the computer (I'll read them one of these days I swear), yet the modest subset of them in this folder barely reaches a few MB, I've got no idea where the rest of books is but I'd consider this a sorting failure.

`Zips/`

While this accomplishes having only it's subject matter inside, the category of "zips" is terribly useless and overlaps with a lot of stuff, from manga to minecraft modpacks to random source code to some really bizarre files I made when messing with raw binary data to some who knows what password protected archives(probably some game considering it's size and name) to actual important backups to whatever else the directory has. Having a "Zips" folder accomplishes absolutely nothing more than a "Misc Junk" folder would, except this obfuscates it's content and makes finding content inside it almost impossible.

Now it's time for the ugly ones

`Software/`

Originally meant for installers and similar; .deb, .flatpak, .snap, .apk, etc or direct executables; bins, appimages, .desktop launchers, etc. After the Factorio accident I also gave it the additional meaning of storing all folders that are unsorteable because they are required to have a structure for some software to run, I soon realized that:

I have no idea what software requires what structure to run
I am already putting most "folders required to have a structure" elsewhere, this is redundant
Since this is impossible to automate, all folders that get into this are sorted manually
Folders that require structure often have reasons to stay in the place they were created and not in my big SORT folder

And as you might've realized by no I suck at organizing stuff, so point 3) in particular is really bad, many files in there don't actually fit the criteria like 1 off installer scripts and folders containing nothing but them.

The folder also has .isos and ROMs.

`Programming/`

Small scripts in various languages, mostly python, bash, lua and js, those are not meant to be software project or anything like that, they're just scripts to perform a single task, often meant to be run only once, let's see t.4 (copy 1)-1.py

from subprocess import run, PIPE
with open("P4.txt")as f, open("O.txt")as safe: i = {t.strip() for t in f}-{t.strip() for t in safe}
for t in i:
	print(t)
	run([*"adb shell pm uninstall --user 0".split(),t],stdout=PIPE)

It's very simple, it just uninstalls android apps listed in P4.txt but not on O.txt, it has some other garbage commented out at the end but it still is an incredibly simple script...

The thing is, I don't have a P4.txt nor O.txt in Programming/, and thus this script is completely useless, not only this one, pretty much every single script in this dir has been stripped from the one context they were designed to be in, rendering most of them completely useless, they're not that valuable in the first place (I hope) so nothing depended on them (I think) and as such nothing has broken when I moved them here (I lie to myself).

`Minecraft/`

A folder for Minecraft as the name implies, it has several subfolders:

user@computer:~/SORT/Minecraft$ ls
Instances  OtherVersions
Modpacks   Resourcepacks
Mods       Shaders

`Modpacks/`

Contains some modpacks as .zip, it doesn't have many since they are really hard to recognize programatically (Zips/ has several files that are just collections of standalone mods, those should be here), other than that it is solid.

`Resourcepacks/`

Pretty much the same, easier to detect too so more of them are sorted there, no complaints.

`Mods/`

The gates to hell, it has some source code as folders and some standalone mods already as jars, a subset of the former is the the code of the mods I've made myself, those folders aren't really stored here, but rather they are symlinks to the the code stored in the dreaded "Projects" folder, I won't stride further into it quite yet.

`Shaders/`

The opposite is true in this case, this is a regular folder containing many shaders, when I create a new instance I then make a symlink to this folder instead of downloading and configuring shaders all over again, I've got no idea how far has this folder spread nor what depends on it, so I rather not touch it.

`OtherVersions/`

Clones, Reimplementations of both servers and clients, Pocket Edition, PS3 edition, PI edition, etc, basically anything that is Minecraft but it isn't the Java version or Shitrock.

`Instances/`

Many MANY instances coming from many different launchers, it's a gigantic mess and that's only the surface, once you go a few folders deep you'll come to realize the fact that many inner folders are in fact symlinks to somewhere else, most to some folder inside ~/Projects (Another horror unrelated to ~/SORT), furthermore it's filled with actually important worlds so I can't just carelessly move it around, yet many are duplicate instances and backups of other instances elsewhere, worst part is that this folder isn't really unique to SORT/Minecraft, I have several such dirs everywhere in my computer and I can't really centralize them because the symlink situation.

With 200gb it is the singular largest folder at this level(inside of one of SORT's main subdirectories), yet it is not quite as problematic as some other folders since this is rarely accessed and grouped into mid to big sized files, there are still worse places.

`Images/`

Unlike every other folder in the SORT/ this is one I use very often and it contains many files, in fact this is a bucket for every png, jpg, webp and gif in the main drive and, as you might expect, it's packed full, if it were a flat directory it would crash thunar every time I tried to open it so I separated it in 256 folders given by the first 2 digits of the blake3 hash of the file content, so every folder is ~700 big (I have some 180k images stored here), this would be very manageable and quite a good solution if it was cold storage but it is not, this is a folder I index often and I am terribly inefficient at finding anything I want on it, I would like to tag them but this folder has a few main problems:

I have many copies of some images all with different hashes.
I have many images where the important is not the image but the data they contain (more on that in a future post).
For some idiotic reason many images are the thumbnails of other images in the folder, including some that are thumbnails of thumbnails, and thumbnails of thumbnails of thumbnails, and so on.
I am a very lazy person, unless I find an equally simple alternative where I can press a keybind and it does everything for me I doubt I'll change the current system.

`'Old Projects'/`

The ugliest in SORT, as it's name implies it is full of "Projects", those projects contain everything I've discussed above and more, regular programming, scripts, thousands of images generated by those scripts, Minecraft mods, obscure technologies and their guide books, manufactured special weird files, scrapped texts, drawn fonts, generated audio, etc, etc, etc.

The Old part comes from the fact that this isn't actually my current project folder but rather the merging of some very old snapshots of the current one, I moved, renamed, edited, and generally changed around so many things that I don't have a reliable way of matching the ones that were left here with the upstream ~/Projects so I had to relegate them here (Projects itself has it's own Old folder with many more), it follows no definite structure other than folders roughly named after the name of at least one of the project they contain, sometimes.

In conclusion

Idk, don't be like me I guess and sort your files properly, often an unplanned attempt hurts more than doing nothing and just accumulate stuff in a structured bucket.

I wish I had more drives to test many different topologies, but I guess the only solution is to get better at organizing things.