Today i combined tags, random and search into a single script using the new big metadata files. So far so good! Still working on importing .metadata files into the big ones, decided to pre-generate and distribute clean values instead of having the user generate them themselves.

Decided I no longer need to separate files into smaller folders with a specific size. Instead, I can just scan any directory for files and add a symlink to it's location.

For example,

ae63e83e3bf87b6710ca3951a5863634de1cd972e2dc4ebf430955f25f6c13f9

would have a link created in

/files/a/e/6/3/e

I've already written out the script and added optional file integrity checking. I'll roll this out completely once I've finished migrating over to the big metadata .csv files. When complete, it should be easy to generate a magnet link for each file and cache the results, so seeding and webtorrent implementations should be possible.

Wrote a script to generate a torrent from a file and write the magnet link into magnet.csv

New Metadata Editor is complete. All values that can have a multiple values (names, artists, descriptions, etc) can be easily be assigned.

Still generating the large .csv files for metadata values.

Big plans in the works, recreated the message board/blog/comment/messaging feature. Decided to create large dictionary with each term hashed. Instead of distributing metadata with the application, users will manage their own list of files and metadata (in hashed format, it will be compared against the main TPS dictionary). Users will be able to "trust" other users (which entails importing their PGP key and having a copy of their userdata) and when they do so, they will be able to see all files/metadata/posts the other user has created.

This basically means we are shifting towards becoming a file categorization protocol, rather than just a pirate archive. Stay tuned for more updates!

Haven't shared an update in a while. The .csv files have been done for a couple months, I abandoned the personal dictionary feature due to over-complication and took a little break on development of the application and started working on converting all of my NTFS partitions over to LUKS encrypted ext4 partitions, doing some metadata scraping projects, and mass downloading new content.

Almost finished the LUKS migration, just working on the backup drives now. Also decided to abandon the folder structure for files (thepiratesea_1, thepiratesea_2, etc). Each file will have a symlink directly to the root directory of it's respective drive. This will make it easier to check for corrupted files, create a torrent for each individual file, and not limit files to a max size.

Thanks for sticking around, I'll share more soon!

Short update: Currently working generating a torrent for every file in the archives, and storing the info hashes into a .csv file. It's a slow process, but going pretty smooth

The last few days I've slapped together a stand-alone Youtube scraper and put it up on Github. I'll eventually incorporate it into the main TPS application when it's launched, but for now it's fully functional and in it's own repository. Feel free to test it out and start scraping, I'll eventually throw together something to share and merge results. https://github.com/thepiratesea/youtube_scraper/

Very simple to run on Linux only:

git clone https://github.com/thepiratesea/youtube_scraper

bash youtube_scraper.sh

Cleaned up the landing page @ thepiratesea.org

This project ain't dead dudes. I'm still scraping away, I'll share some info soon

I bit the bullet and ordered another 20TB. I haven't been scraping Youtube videos for about a year, I was hoping HDD prices would come back down but I figure it's time to catch back up.

I have however, been scraping metadata. These are my results with the youtube scraper script so far. "Video Metadata" is the title, date, description and channel ID for the specific video. About 1.55M of them are videos that I've actually downloaded and archived, the rest are just random from scraping.

Merry Christmas fellow pirates! The downloader tool to archive Youtube videos/metadata, strip the ID3 data, hash the files and add all of metadata to our format is finally complete and pushed to the Github repository.

Since Youtube has gotten increasingly aggressive, I've added a little script that will automatically switch VPN nodes if there is an error (currently only works for PIA, but I can add support for other popular VPNs later).

I'm working towards phasing out Invidious where possible since it's been unstable (it's still required for certain features like pulling channel descriptions and getting recommended videos). So I definitely recommend running a VPN if you use this script so your main IP address doesn't get flagged as a bot and throttled.

Eventually I will set up a private git repository to merge everybody's collected metadata and a file server to upload the video results.

Stay tuned for more updates!

Rearranged the datacenter room to allow for more expansion. We now have a setup that will allow us to scale up in the foreseeable future. I've been thinking about launching a series of websites that are dedicated to streaming specific shows. This should be relatively easy to achieve with the scripts I've written to run a local playlist, and just publish the results to webserver. Hopefully if we target the proper shows, we can start bringing in a little bit of revenue that will help fund the project and hardware expansion.

Arghhh, we av' decided it's about time t' undock this fine vessel and let er' sail.

Oe'r th' next few weeks, we'll be rewritin' The Pirate Sea from scratch wit' multi-user functionality in mind, an' pushin' our progress t' Github in real time as we go. A basic login and updater are already live, ye' can clone/run it by typin'

git clone https://github.com/thepiratesea/thepiratesea.git . && bash thepiratesea.sh

in yer terminal

Goin' forward, we will maintain a list of domains that link t' webservers, where th' client will grab the /users/ directory from each server an' merge all user data locally. Keep checking fer updates, they should be plentiful for the foreseeable future!

Introducing The Pirate Sea's new script, the Youtube Raider. A great quality of life browser script that allows you to manage subscriptions, history, whitelisting/blacklisting (parental controls) and scrape metadata without effort as you browse Youtube.

More info and source code - https://github.com/thepiratesea/youtube-raider

Our website is properly configured and we compiled an appimage for the client -- thepiratesea.org