posted on Apr 25, 2017

Fiddling With SELinux Policies

SELinux is enabled by default on both RedHat and CentOS servers. It can be run in three modes, namely enforcing, permissive, or disabled.

Quite a few developers have the habit of disabling SELinux when configurations breach existing policies. Two of the most common approaches are to either globally disable SELinux,

setenforce 0

or to set it to a permissive mode.

setenforce 2

In permissive mode, SELinux allows all operations, but logs any that would have breached existing policies. This allows us to look into the audit logs to determine which policies need to be enabled for our applications to work as expected.

A common scenario in which SELinux permissions are breached is while trying to serve applications proxied behind nginx. SELinux by default does not allow nginx to connect to remote web, FastCGI or any other servers.

Setting Up The Environment

In this blog post, we will recreate the above scenario. We would need Vagrant and VirtualBox installed to follow through.

Once the applications are installed we can pull the relevant vagrant boxes.

$ git clone https://gitlab.com/wismutlabs-public/selinux-example.git
$ cd selinux-example
$ vagrant up

This will pull the relevant vagrant box and set it up with the necessary hostname and IP address. The image that vagrant pulls has nginx and a simple python app installed. The app is served by nginx using the following config file:

upstream app_servers {
    server 127.0.0.1:8282;
}

server {
    listen       80; 
    server_name  nginx.wismutlabs.dev;

    access_log  /var/log/nginx/nginx.wismutlabs.dev.access.log;
    error_log  /var/log/nginx/nginx.wismutlabs.dev.error.log error;

    location / {
        proxy_pass         http://app_servers;
        proxy_redirect     off;
        proxy_set_header   Host $host;
        proxy_set_header   X-Real-IP $remote_addr;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Host $host;
    }
}

In this configuration, nginx proxies any connections received for the host nginx.wismutlabs.dev to the webapp running on port 8282.

Testing The Setup

Before we proceed to setup SELinux policies, it is always a good idea to test out if everything is working fine as expected.

Access the VM using the following command:

$ vagrant ssh

Once we are in the VM we can check whether nginx and the webapp are running using systemd’s systemctl command

$ systemctl status nginx
● nginx.service - The nginx HTTP and reverse proxy server
   Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2017-04-24 14:14:59 UTC; 10min ago
 Main PID: 14044 (nginx)
   CGroup: /system.slice/nginx.service
           ├─14044 nginx: master process /usr/sbin/nginx
           └─14045 nginx: worker process

$ systemctl status webapp
● webapp.service
   Loaded: loaded (/etc/systemd/system/webapp.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2017-04-24 14:15:00 UTC; 10min ago
 Main PID: 14142 (gunicorn)
   CGroup: /system.slice/webapp.service
           ├─14142 /home/webapp/app/env/bin/python2 /home/webapp/app/env/bin/gunicorn --bind 127.0.0.1:8282 app:app
           └─14150 /home/webapp/app/env/bin/python2 /home/webapp/app/env/bin/gunicorn --bind 127.0.0.1:8282 app:app

We should also test whether the webapp is working properly.

$ curl localhost:8282
{
  "Accept": "*/*", 
  "Host": "localhost:8282", 
  "User-Agent": "curl/7.29.0"
}

Now that we know the webapp is working fine, and that both nginx and the webapp are running, it is time to test the application from the ”outside world”.

The Vagrantfile provided sets up the VM with the hostname nginx.wismutlabs.dev and IP address 10.10.10.21 and injects these values into the host machines /etc/hosts file. This was done when we executed, like so:

$ vagrant up

If this set did not happen, we should manually add these entries to our /etc/hosts file. The assumption here is we are working on a Unix-like machine.

To test this out, we can try pinging this from our host machine:

$ ping -c3 10.10.10.21
PING 10.10.10.21 (10.10.10.21): 56 data bytes
64 bytes from 10.10.10.21: icmp_seq=0 ttl=64 time=0.280 ms
64 bytes from 10.10.10.21: icmp_seq=1 ttl=64 time=0.249 ms
64 bytes from 10.10.10.21: icmp_seq=2 ttl=64 time=0.310 ms

--- 10.10.10.21 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.249/0.280/0.310/0.025 ms

$ ping -c3 nginx.wismutlabs.dev
PING nginx.wismutlabs.dev (10.10.10.21): 56 data bytes
64 bytes from 10.10.10.21: icmp_seq=0 ttl=64 time=0.264 ms
64 bytes from 10.10.10.21: icmp_seq=1 ttl=64 time=0.277 ms
64 bytes from 10.10.10.21: icmp_seq=2 ttl=64 time=0.302 ms

--- nginx.wismutlabs.dev ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.264/0.281/0.302/0.016 ms

Note that if everything was setup properly, nginx.wismutlabs.dev should resolve to 10.10.10.21 as shown above.

Open a browser and point it to http://nginx.wismutlabs.dev. We would be presented with the error code 502 - Bad Gateway.

On the VM, issue the following command, and revisit the page on the browser. This will disable SELinux.

$ sudo setenforce 0

We would now be able to see a response from the webapp similar to what is shown below:

{
  "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", 
  "Accept-Encoding": "gzip, deflate", 
  "Accept-Language": "en-sg", 
  "Connection": "close", 
  "Host": "nginx.wismutlabs.dev", 
  "Upgrade-Insecure-Requests": "1", 
  "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_4) AppleWebKit/603.1.30 (KHTML, like Gecko) Version/10.1 Safari/603.1.30", 
  "X-Forwarded-For": "10.10.10.1", 
  "X-Forwarded-Host": "nginx.wismutlabs.dev", 
  "X-Real-Ip": "10.10.10.1"
}

What just happened

While SELinux is enabled, it prevents nginx from proxying any connection to the webapp, hence the 502 response page. Once SELinux is disabled, nginx is able to proxy the connection to the webapp, allowing the webapp to serve content to us.

Writing SELinux policies

Before we can write an SELinux policy for this, we need to determine which SELinux policies were violated.

To determine this, we need to install a few packages. This would give us tools such as audit2allow, audit2why, sesearch and so on.

On the VM, issue the command:

$ sudo yum install -y policycoreutils-python setools

It is best to become the root user for the rest of the tasks.

$ sudo -i

To determine what policies were violated, let’s revert SELinux to enforcing:

# setenforce 1

On CentOS machines, all audit information is logged to the file /var/log/audit/audit.log

We can access the webapp from our browser again after tailing this file in follow mode. This will show us which policies were violated:

# tail -f tail -f /var/log/audit/audit.log

We would see an entry that looks something like this:

type=AVC msg=audit(1493044451.697:1387): avc:  denied  { name_connect } for  pid=14045 comm="nginx" dest=8282 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket
type=SYSCALL msg=audit(1493044451.697:1387): arch=c000003e syscall=42 success=no exit=-13 a0=e a1=7f6387838300 a2=10 a3=7ffe5c963a30 items=0 ppid=14044 pid=14045 auid=4294967295 uid=996 gid=994 euid=996 suid=996 fsuid=996 egid=994 sgid=994 fsgid=994 tty=(none) ses=4294967295 comm="nginx" exe="/usr/sbin/nginx" subj=system_u:system_r:httpd_t:s0 key=(null)

Here we see an audit message code (1493044451.697:1387). To understand this message further we need to pass it through the audit2why command:

# grep 1493044451.697:1387 /var/log/audit/audit.log | audit2why
type=AVC msg=audit(1493044451.697:1387): avc:  denied  { name_connect } for  pid=14045 comm="nginx" dest=8282 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket

    Was caused by:
    One of the following booleans was set incorrectly.
    Description:
    Allow httpd to can network connect

    Allow access by executing:
    # setsebool -P httpd_can_network_connect 1
    Description:
    Allow nis to enabled

    Allow access by executing:
    # setsebool -P nis_enabled 1

The output itself is rather self explanatory. It even goes as far as to describe how we could fix the problem.

In this post, we will be focusing on the the boolean httpd_can_network_connect. Let us check the value of this boolean using the getsebool command:

# getsebool httpd_can_network_connect
httpd_can_network_connect --> off

We can now enable this boolean by using the suggestion from audit2why. We will do this without the -P flag for now. This flag is used to set the boolean permanently across reboots.

# setsebool httpd_can_network_connect 1

Now if we retrieve the value of the boolean we will see the following:

# getsebool httpd_can_network_connect
httpd_can_network_connect --> on

We can open the browser window and test out the webapp now. We would be able to see the output served by the webapp.

We can use the sesearch command to determine what exactly is allowed by turning on this boolean:

# sesearch -A -s httpd_t -b httpd_can_network_connect
Found 1 semantic av rules:
   allow httpd_t port_type : tcp_socket name_connect ;

This indicates that by setting this boolean, it allows httpd_t to connect to all TCP socket types that have the port_type attribute.

To find out what are the attributes associated with port_type, we can run the following command:

# seinfo -a port_type -x
   port_type
      afs3_callback_port_t
      afs_bos_port_t
      afs_fs_port_t
      .....
      .....
      dhcpd_port_t
      dict_port_t
      distccd_port_t
      dns_port_t
      .....
      .....
      http_cache_port_t
      http_port_t
      .....
      .....

There are over 200 types that are allowed. To determine which ports are associated with these types we can execute the following:

# semanage port -l

While this solves our problem, it is not a clean solution. We just allowed a blanket rule that allows httpd_port_t to connect to any port, while what we are trying to achieve is to only allow access to our webapp’s port.

Creating an SELinux Policy

We can use the command audit2allow to determine how to create an SELinux policy that caters for our requirements.

Let us pipe our audit log through audit2allow to determine the necessary course of action.

# grep 1493044451.697:1387 /var/log/audit/audit.log | audit2allow


#============= httpd_t ==============

#!!!! This avc is allowed in the current policy
allow httpd_t unreserved_port_t:tcp_socket name_connect;

The tool, audit2allow, shows that the policy that had previously blocked our proxy request is now allowed. This is because we set the boolean earlier. Let us revert it to how it was earlier and then proceed to run audit2allow.

# setsebool httpd_can_network_connect 0
# grep 1493044451.697:1387 /var/log/audit/audit.log | audit2allow


#============= httpd_t ==============

#!!!! This avc can be allowed using one of the these booleans:
#     httpd_can_network_connect, nis_enabled
allow httpd_t unreserved_port_t:tcp_socket name_connect;

This basically says that we need to extend the http_t policy to allow access to unreserved ports. In order to do this, we need to generate a policy file.

# grep 1493044451.697:1387 /var/log/audit/audit.log | audit2allow -m nginx > nginx.te

The contents of this nginx.te file is as follows:

# cat nginx.te 

module nginx 1.0;

require {
    type httpd_t;
    type unreserved_port_t;
    class tcp_socket name_connect;
}

#============= httpd_t ==============

#!!!! This avc can be allowed using one of the these booleans:
#     httpd_can_network_connect, nis_enabled
allow httpd_t unreserved_port_t:tcp_socket name_connect;

We can compile this policy using the following set of commands:

# checkmodule -M -m -o nginx.mod nginx.te 
checkmodule:  loading policy configuration from nginx.te
checkmodule:  policy configuration loaded
checkmodule:  writing binary representation (version 17) to nginx.mod

This can, however, be done using the audit2allow command as well by using the -M flag.

# grep 1493044451.697:1387 /var/log/audit/audit.log | audit2allow -M nginx
******************** IMPORTANT ***********************
To make this policy package active, execute:

semodule -i nginx.pp

We could then install this policy as suggested by the output above and check if it was successfully installed.

# semodule -i nginx.pp
# semodule -l | grep nginx
nginx   1.0

If we point our browser to http://nginx.wismutlabs.com, we would now be able to see the response from the webapp.

An Alternative Approach

While we could compile a policy using the aforementioned method, it still has a shortcoming when trying to achieve our objective. The above policy allows access to all unreserved ports. Our objective is to allow access to a particular port, and not all unreserved ports.

Let us uninstall the nginx policy and startover.

# semodule -r nginx
libsemanage.semanage_direct_remove_key: Removing last nginx module (no other nginx module exists at another priority).
# semodule -l | grep nginx
#

If we point our browser to http://nginx.wismutlabs.com, we would again be faced with the 502 Bad Gateway response.

To achieve our desired result, we can add the webapp’s port 8282 to the http_port_t type using the semanage command.

# semanage port -a -t http_port_t -p tcp 8282

We can check whether http_port_t contains our tcp port by using semanage.

# semanage port -l | grep http_port_t
http_port_t                    tcp      8282, 80, 81, 443, 488, 8008, 8009, 8443, 9000

If we now point our browser to http://nginx.wismutlabs.com we would be able to see the response from the webapp again.

With this approach, we can now change the webapp port on the fly.