Hello everyone! 
In this blog post we’re exploring the option of self hosting Anubis.
Anubis is a software that defends and protects your services from AI Scrapers and LLMs. Since AI started to get popular more and more scrappers started appearing.They grab your data in order to use it to train their AIs while ignoring any rules such as robots.txt, licenses and intellectual property laws.Meta even torrented 82TB of books to train their AI.
You may have seen when visiting the some websites a pop-up screen like this:
This is Anubis, it works by presenting your browser with a proof of work challenge which require some CPU resources to solve. The challenge makes scrapping website a bit costlyfor scrappers, instead of just issuing a request they now need to have a browser with a Window, and spend CPU resources solving the challenge in order to bypass Anubis.
Popular websites such as Gnome Gitlab, Arch Wiki already use Anubis to protect their data, and you can use it too!
To run the playbook you will need make and ansible. You can download the Playbook from: https://forge.nuculabs.de/dnutiu/ansible-playbooks/src/branch/master/anubis
Then you will need to modify the inventory.ini and variables.yaml and run make install && make run
Anubis needs a target URL of the service it’s protecting, since I’m running Forgejo inside a Podman container the target url for me is: http://host.docker.internal:3000.You can change it in variables.yaml.
Esentially variables.yaml already contain some default settings, you may want to change the instance name of this Anubis instance, the http port, the target (url) andif you’re exposing Anubis directly the setup firewall option.
The inventory.ini should only contain the IP address of the server you want to install Anubis, the connection is made via SSH and you should have the key loaded inside your agent.
That’s about it.
In this blog post we’re exploring the option of self hosting Anubis.
Anubis is a software that defends and protects your services from AI Scrapers and LLMs. Since AI started to get popular more and more scrappers started appearing.They grab your data in order to use it to train their AIs while ignoring any rules such as robots.txt, licenses and intellectual property laws.Meta even torrented 82TB of books to train their AI.
You may have seen when visiting the some websites a pop-up screen like this:
This is Anubis, it works by presenting your browser with a proof of work challenge which require some CPU resources to solve. The challenge makes scrapping website a bit costlyfor scrappers, instead of just issuing a request they now need to have a browser with a Window, and spend CPU resources solving the challenge in order to bypass Anubis.
Popular websites such as Gnome Gitlab, Arch Wiki already use Anubis to protect their data, and you can use it too!
Ansible Playbook
I’m protecting my Forgejo instance with Anubis and to make it easier to deploy Anubis I’ve created an Ansible Playbook.To run the playbook you will need make and ansible. You can download the Playbook from: https://forge.nuculabs.de/dnutiu/ansible-playbooks/src/branch/master/anubis
Then you will need to modify the inventory.ini and variables.yaml and run make install && make run
Anubis needs a target URL of the service it’s protecting, since I’m running Forgejo inside a Podman container the target url for me is: http://host.docker.internal:3000.You can change it in variables.yaml.
Esentially variables.yaml already contain some default settings, you may want to change the instance name of this Anubis instance, the http port, the target (url) andif you’re exposing Anubis directly the setup firewall option.
Code:
# variables.yaml
setup_firewall: false # exposes the anubis firewall port
setup_firewall_for_metrics: false # exposes the firewall port for metrics
anubis:
instance_name: "forgejo"
base_directory: "anubis"
container_image: "ghcr.io/techarohq/anubis:v1.16.0"
difficulty: 5
http_port: 4000
metrics_port: 4001
serve_robots_txt: "true"
target: "http://host.docker.internal:3000"
og_passthrough: "true"
og_expiry_time: "24h"
og_cache_consider_host: "false"
webmaster_email: ""
policy_file_name: ""The inventory.ini should only contain the IP address of the server you want to install Anubis, the connection is made via SSH and you should have the key loaded inside your agent.
Code:
# inventory.ini
[nuculabs]
hosts.example.com ansible_user=ansible
[local]
localhostThat’s about it.