My mission today was not really simple but it was still quite needed: I wanted to implement my deployment and server setup via Ansible. There might be better options out there but I already use Ansible at work and thus wanted to learn a little more about it.

This had the added bonus that I finally had a neat way to automate my deployments in a better way than before. Previously I had a small makefile that just used rsync to copy over the files and ssh to restart the webserver with the new config. Don't get me wrong, this is definitely not a bad approach to take; it's simple and works but it has it's drawbacks. It doesn't give you a lot of feedback and (in my opinion) it's a little bit spaghetti if you want to extend it to a system that gives you good options, feedback and is scaleable. You might as well just use existing automation systems.

Setting up Ansible for the first time, when you don't really have an idea how it all fits together is a little confusing: there's a lot of new terminology and concepts to understand. Luckily these systems are widely used and LLM's can generate some good examples if you prompt for the right thing :)

After having built everything it's on to executing the playbook. So i ran it, checked if my sites are still up aaaaaaaand I get a 403...hm >:T Luckily my site is smol and I don't sell people things they don't need so downtime is basically not relevant...but it still sucks.

Onto debugging then...

  • Caddy (my http server): healthy
  • Logs: Don't say anything interesting
  • File-owner: webserver role
  • File-permissions (of everything): 0644 (rw-r--r--)

So far everything looks like it should not error...I'm slowly running out of ideas...

I searched the internet a bit and...nothing. So I tested around with file permissions a little more and found something interesting: setting file permissions to 0500 (r-x------) fixed the issue and the files could be served again. What a weird thing, right? Why would caddy need execute permissions for the files it's serving? I mean clearly you would only have to read the files and that'd be that, right?

Have you already noticed what the actual issue was?

I'll give you a hint: there was no issue with the html files. Caddy really only needs read permissions for them. If you want to figure it out for yourself now's your time.

Want to know what the issue really was? The problem was that there were no execute permissions on the directories! :^)

If you didn't know (like me before): the execute bit on directories is for accessing the contents inside the directory. The read bit only allows you to list the files inside, NOT read them. I think this is confusing until you think about what to do with the execute bits on directories really. I mean you can't execute a directory, so what else to do with it?

I fixed it by changing this Ansible task:

- name: Copy website content to the web server
  ansible.builtin.copy:
    src: "{{ item.src }}"
    dest: "{{ item.dest }}"
    owner: "{{ web_user }}"
    group: "{{ web_group }}"
    directory_mode: '0500'
  loop: "{{ sites }}"

To this one:

- name: Copy website content to the web server
  ansible.builtin.copy:
    src: "{{ item.src }}"
    dest: "{{ item.dest }}"
    owner: "{{ web_user }}"
    group: "{{ web_group }}"
    mode: "0400"
    directory_mode: '0500'
  loop: "{{ sites }}"

Notice that I just added a mode entry which set's the file modes and the directory_mode sets the mode for - you guessed it - the directories.

I think this was an odd thing to debug but I definitely learned something along the way. So basically:

Remember to correctly set your file permission, kids!