nothing@nowhere - 2021-05-13

Hydrus Network Setup

In this tutorial we're going to see how to use Hydrus Network, which is a personal booru client used primarily to hoard as many images as possible from image posting boards like 4chan.org.

Initial Setup

[ 10.66.66.2/32 ] [ /dev/pts/3 ] [~/Tools]
→ git clone https://github.com/hydrusnetwork/hydrus
Cloning into 'hydrus'...
remote: Enumerating objects: 16952, done.
remote: Counting objects: 100% (1094/1094), done.
remote: Compressing objects: 100% (431/431), done.
remote: Total 16952 (delta 695), reused 1011 (delta 649), pack-reused 15858
Receiving objects: 100% (16952/16952), 101.25 MiB | 5.32 MiB/s, done.
Resolving deltas: 100% (14113/14113), done.

[ 10.66.66.2/32 ] [ /dev/pts/3 ] [~/Tools/hydrus]
→ ls -l
total 88
drwxr-xr-x 2 nothing nothing  4096 Jun 28 23:44  bin
-rw-r--r-- 1 nothing nothing   251 Jun 28 23:44  client.py
-rwxr-xr-x 1 nothing nothing   251 Jun 28 23:44  client.pyw
-rw-r--r-- 1 nothing nothing   484 Jun 28 23:44  COPYING
drwxr-xr-x 2 nothing nothing  4096 Jun 28 23:44  db
drwxr-xr-x 2 nothing nothing 12288 Jun 28 23:44  help
-rw-r--r-- 1 nothing nothing  2089 Jun 28 23:44 'help my client will not boot.txt'
drwxr-xr-x 7 nothing nothing  4096 Jun 28 23:44  hydrus
-rw-r--r-- 1 nothing nothing   884 Jun 28 23:44  license.txt
-rwxr-xr-x 1 nothing nothing  2066 Jun 28 23:44  README.md
-rw-r--r-- 1 nothing nothing   303 Jun 28 23:44  Readme.txt
-rw-r--r-- 1 nothing nothing    38 Jun 28 23:44  requirements_macos_build.txt
-rw-r--r-- 1 nothing nothing   376 Jun 28 23:44  requirements_macos.txt
-rw-r--r-- 1 nothing nothing    44 Jun 28 23:44  requirements_ubuntu_build.txt
-rw-r--r-- 1 nothing nothing   376 Jun 28 23:44  requirements_ubuntu.txt
-rw-r--r-- 1 nothing nothing    77 Jun 28 23:44  requirements_windows_build.txt
-rw-r--r-- 1 nothing nothing   376 Jun 28 23:44  requirements_windows.txt
-rw-r--r-- 1 nothing nothing   251 Jun 28 23:44  server.py
drwxr-xr-x 7 nothing nothing  4096 Jun 28 23:44  static
-rw-r--r-- 1 nothing nothing   247 Jun 28 23:44  test.py

i'm on kali (debian based distro) so i get the required python dependencies for ubuntu (which is also debian based):

[ 10.66.66.2/32 ] [ /dev/pts/3 ] [~/Tools/hydrus]
→ pip3 install -r  requirements_ubuntu.txt
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: beautifulsoup4>=4.0.0 in /usr/lib/python3/dist-packages (from -r requirements_ubuntu.txt (line 1)) (4.9.3)
Requirement already satisfied: chardet>=3.0.4 in /usr/lib/python3/dist-packages (from -r requirements_ubuntu.txt (line 2)) (4.0.0)
Collecting cloudscraper>=1.2.33
  Downloading cloudscraper-1.2.58-py2.py3-none-any.whl (96 kB)
     |████████████████████████████████| 96 kB 264 kB/s
Requirement already satisfied: html5lib>=1.0.1 in /usr/lib/python3/dist-packages (from -r requirements_ubuntu.txt (line 4)) (1.1)
Requirement already satisfied: lxml>=4.5.0 in /usr/lib/python3/dist-packages (from -r requirements_ubuntu.txt (line 5)) (4.6.3)
Collecting lz4>=3.0.0
  Downloading lz4-3.1.3-cp39-cp39-manylinux2010_x86_64.whl (1.8 MB)
     |████████████████████████████████| 1.8 MB 952 kB/s
Collecting nose>=1.3.0
  Downloading nose-1.3.7-py3-none-any.whl (154 kB)
     |████████████████████████████████| 154 kB 2.6 MB/s
[...]

[ 10.66.66.2/32 ] [ /dev/pts/3 ] [~/Tools/hydrus]
→ python3 client.py

[ 10.66.66.2/32 ] [ /dev/pts/3 ] [~/Tools/hydrus]
→ chmod +x ./client.py

[ 10.66.66.2/32 ] [ /dev/pts/3 ] [~/Tools/hydrus]
→ ./client.py

Now we simply run client.py to open the hydrus client, so we can also create a shortcut (symbolic link) to it and add it into our PATH:

[ 10.66.66.2/32 ] [ /dev/pts/3 ] [~/Tools/hydrus]
→ sudo ln -s /home/nothing/Tools/hydrus/client.py /usr/local/bin/hydrus_client.py

Now simply launch it:

[ 10.66.66.2/32 ] [ /dev/pts/0 ] [~/Tools/hydrus]
→ hydrus_client.py

Now right off the bat hydrus is completely white themed so to fix this goto file > options > style > Hydracula.qss and to colours > colourset > darkmode

Next step here is to change the bandwidth limits into network > data > review bandwidth usage and rules and in here we need to edit the default bandwidth rules, edit each of those depending on how much data you intend to download every day / hour:

For simplicity's sake i just set them as MBps:

Hoarding from chans

hydrus basically has a way of downloading every image from a thread on a 4chan board, there is also a feature to paste in multiple threads at once, so you can have the idea of downloading every image of every thread of a board, For example, to get every image from /g/ to do so you can do as follows:

F12 > console > allow paste
const links2 =[...new Set(Object.keys(window.catalog.threads))].map(id => `https://boards.4channel.org/g/thread/${id}`).join('\n')
console.log(links2)

Now you have a list of links of all the threads of that one board, and so we can create a new download page to bulk download from them pages > new download page > watcher:

And here you can see that we have the ability to copy in as many threads as we want, so get the previous links in your clipboard and click that logo:

So now on this page hydrus is going to download /g/ basically. If you want to see every new image that gets downloaded, select all the watchers with CTRL+A and right click > show all watchers' new files:

And that's it! That's basically how i use hydrus, i also hoarded a collection of wallpapers from /wg/ with the following firefox console command:

F12 > console > allow paste
const links3 =[...new Set(Object.keys(window.catalog.threads))].map(id => `https://boards.4channel.org/wg/thread/${id}`).join('\n')
console.log(links2)

Take note that i changed the variable to being links3 and the string to /wg/thread/$(id), and then same process we just send all the links from our clipboard to the watchers page in hydrus.

Importing an existing database

Now if you're like me and you have multiple OSes to use, you will probably have to install hydrus more than once, and one particular problem can be to import a database, and it's not as simple as you might think:

To explain this problem we're going to pick our database, which is basically a folder that we choose, i have an existing one in /mnt/vault/DUMP/, so you would think that i can import them by going to database > migrate database > add new location for files

Next we remove the previous db location:

Basically we can't really move the database to a non empty location, nor discover it as an option, trying to do this will result in the following:

TLDR if you have an existing database you need to basically tell hydrus to pick the database path as you launch it, so we're going to do it using a bashscript after we remove the shortcut we did earlier:

[ 10.66.66.2/32 ] [ /dev/pts/0 ] [Github/blog/HTB]
→ sudo rm /usr/local/bin/hydrus_client.py
[sudo] password for nothing:

[ 10.66.66.2/32 ] [ /dev/pts/0 ] [Github/blog/HTB]
→ sudo vim /usr/local/bin/hydrus_client.sh

[ 10.66.66.2/32 ] [ /dev/pts/0 ] [Github/blog/HTB]
→ cat /usr/local/bin/hydrus_client.sh
#!/bin/bash

/home/nothing/Tools/hydrus/client.py -d="/mnt/vault/DUMP"

[ 10.66.66.2/32 ] [ /dev/pts/0 ] [Github/blog/HTB]
→ sudo chmod +x /usr/local/bin/hydrus_client.sh

So we run hydrus once again, this time when we launch hydrus we need to repopulate the database since we know that it has the correct database location, you can check it by doing file > open > database directory. So we repair our database by discovering the files by going to file > import files

Wait until the database gets completely parsed and then hit 'import now':

And there you go, we managed to import our existing database. (be patient because importing takes a long time)