Find Your Hidden Services

Liunx Administration

When you are inventorying network services as part of a security audit, make sure that you find all your hidden services!

I don’t mean a hidden server as with the so-called “Deep Web”, something only accessible through Tor, but instead a network service that you might overlook on a known server. I was reminded of this recently when I first thought that an NFS server wasn’t available when it really was already working just fine.

“Everything Works Like A File”

That’s a common observation on all the UNIX-family operating systems, and with some changes in terminology it seems to be a good description for any multi-tasking operating system. What can you do with a file?

  • Open it
  • Write data into it
  • Read data out of it
  • Close it

That’s also true for devices, network sockets, and pipes used for inter-process communication. One programming interface lets you learn and apply one common method for dealing with everything.

The lsof program lists open files in the broadest sense. You can list everything that a specified process has open — the file holding its executable code, all its shared libraries, and all the files and devices and sockets and more that it is reading from, writing to, listening to, and more.

You can answer the question “Which of my log files are currently being read or written?”

# lsof $( find /var/log /var/www/log -type f )

When you are working with network services, you need to answer the question “What is the list of all active UDP and TCP services?”

# lsof -i

Where Did My NFS Service Go?

This is where it gets tricky! Here is the result of looking for the TCP services on a system that is supposed to be an NFS server:

# lsof -i tcp
COMMAND    PID  USER  FD   TYPE DEVICE SIZE/OFF NODE NAME
rpcbind   1865  root   8u  IPv4   5894      0t0  TCP *:sunrpc (LISTEN)
rpc.mount 2307  root   9u  IPv4   6242      0t0  TCP *:38207 (LISTEN)
rpc.mount 2307  root  11u  IPv4   6260      0t0  TCP *:44624 (LISTEN)
rpc.mount 2307  root  13u  IPv4   6276      0t0  TCP *:36905 (LISTEN)
sshd      2443  root   3u  IPv4   6398      0t0  TCP *:ssh (LISTEN)

I see the RPC port mapper and some mount daemons, but I don’t see the expected NFS service listening on TCP port 2049. Let’s see what’s running:

# rpcinfo -p | egrep 'program|portmapper|nfs'
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100227    2   tcp   2049  nfs_acl
    100227    3   tcp   2049  nfs_acl
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100003    4   udp   2049  nfs
    100227    2   udp   2049  nfs_acl
    100227    3   udp   2049  nfs_acl

# ps axuww | egrep 'PID|nfs'
USER   PID %CPU %MEM  VSZ  RSS TTY  STAT START  TIME COMMAND
root  3485  0.0  0.0    0    0 ?    S<   21:50  0:00 [nfsd4]
root  3486  0.0  0.0    0    0 ?    S<   21:50  0:00 [nfsd4_callbacks]
root  3490  0.0  0.0    0    0 ?    S    21:50  0:00 [nfsd4]
root  3491  0.0  0.0    0    0 ?    S    21:50  0:00 [nfsd4]
root  3492  0.0  0.0    0    0 ?    S    21:50  0:00 [nfsd4]
root  3493  0.0  0.0    0    0 ?    S    21:50  0:00 [nfsd4]
root  3494  0.0  0.0    0    0 ?    S    21:50  0:00 [nfsd4]
root  3495  0.0  0.0    0    0 ?    S    21:50  0:00 [nfsd4]
root  3496  0.0  0.0    0    0 ?    S    21:50  0:00 [nfsd4]
root  3497  0.0  0.0    0    0 ?    S    21:50  0:00 [nfsd4]

There are the NFS service processes. Ah, but look at how they are listed in square brackets. These are kernel threads, not regular processes. Let’s ask lsof for all the details on a specific one:

# lsof -p 3485
COMMAND  PID USER   FD     TYPE DEVICE SIZE/OFF NODE NAME
nfsd4   3485 root  cwd      DIR    8,6     4096    2 /
nfsd4   3485 root  rtd      DIR    8,6     4096    2 /
nfsd4   3485 root  txt  unknown                      /proc/3485/exe

That is strangely terse compared to what we see for normal processes. And look, no sign of activity on TCP port 2049.

The same lsof program runs on all UNIX-like operating systems, it has to make do with whatever the local kernel provides to it. Simply running lsof -i asks it to list all the TCP/IP sockets that processes have open. The kernel would handle connection requests on TCP port 2049 and hand them off to the NFS server threads.

Using An Additional Tool

This is a case where a port scanner provides further information:

# nmap -sS -sV -n 10.1.1.232

Starting Nmap 6.47 ( http://nmap.org ) at 2015-06-21 09:25 EDT
Nmap scan report for 10.1.1.232
Host is up (0.00154s latency).
Not shown: 997 closed ports
PORT     STATE SERVICE VERSION
22/tcp   open  ssh     OpenSSH 6.0p1 Debian 4+deb7u2 (protocol 2.0)
111/tcp  open  rpcbind 2-4 (RPC #100000)
2049/tcp open  nfs     2-4 (RPC #100003)
MAC Address: B8:27:EB:69:BE:BB
Service Info: OS: Linux; CPE: cpe:/o:linux:linux_kernel

This is why we show you several overlapping tools in Learning Tree’s Linux server administration course — it’s good to investigate one problem with multiple tools!

Type to search blog.learningtree.com

Do you mean "" ?

Sorry, no results were found for your query.

Please check your spelling and try your search again.