Ssh honeypot

overview

As usual my server ssh port is being attacked. I have fail2ban installed that is applying geoip and failed attempts filters to block unauthorised access. But ssh attempts are persistently being handed off to other clients originating at a different IP, some of which are in AU.

I am interested in finding out what user and passwords credentials they are attempting and from what IP address, for those that are in AU. This knowledge may help improve my security for unattended systems. So something like an ssh honeypot is called for.

Note: my source-code changes to sshd are different top the usual honey pot, for I only log failed attempts. A successful password value is never logged. Also note that there are two paths for password login:

• regular ssh_auth_password
• PAM sshpam_auth_password

Note: the client connection's details are contained within struct * ssh which is not directly available to sshpam_auth_password() without changing its parameters.

Since writing this page I have developed listsd.

requirements

• log user id and passwords securely
  • password log to be only accessible via root privileges (logs to /var/log/auth.log)
  • don't log passwords with access from my local network (not implemented see next line)
  • don't log users and passwords from white-listed users and IPs (I mainly satisfied this by only logging failed attempts)
• log the remote IP address in the same log entry (some other part of login is logging the client ID (done for failed attempts)
• block black-list remote login if it is not from its listed IP address, or from my internal network. (not implemented, see above)

choices

• install a full ssh honeypot
• make my own version of ssh
• use PAM

I decided to build an ssh from source code and to modify it to meet my requirements. This may be done is stages as I need to work out how sshd works.

hardening

it is important, and has been shown that root should not be able to sshd login. May attacks target root. By extension it is important not to let the attacker know what operating system you are running, so make these apply:

• do not permit root to ssh login
• obfuscate the SSH header so the operating system and version is not sent back to attackers.

I hacked my deployed sshd binary with a hex editor so it does this: (See SP007 production hardening)

telnet watchdog.server 22
Trying 10.10.88.2.
Connected to watchdog.server.
Escape character is '^]'.
SSH-2.0-xxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxx

The honey pot changes go one further, I added the words 'Honey Pot' to the sshd banner - just to let them know they are wasting their time:

nc watchdog.server 22

SSH-2.0-XXX-XXX Honey Pot

Note that the former hack, and the later compiled honey pot version do not disclose the version of SSH nor the operating system.

Building ssh from source

There are minimal dependencies required to build OpenSSH.

If sshd is not already installed on your system you will need to create a chroot.

• check if sshd is there

ssh -V

• (optional) if ssh is already installed, else execute the following:

 mkdir /var/lib/sshd
 chmod -R 700 /var/lib/sshd/
 chown -R root:sys /var/lib/sshd/
 useradd -r -U -d /var/lib/sshd/ -c "sshd privsep" -s /bin/false sshd

obtain build dependencies

• obtain the build dependencies

apt install build-essential zlib1g-dev git autoconf -y

• got to a work area

cd /usr/src

build openssl

(optional) you may need to build ssl if it is not installed or its version is less than 1 or you experience an error with linking libcrypto.a when building openSSH. Note do not use -prefix=/usr unless you install the same version as that which Debian is already running!

• obtain the installed version

openssl version

• obtain the openssl source (if version < 1 or you have problems building openssh)

git clone git://git.openssl.org/openssl.git

• You may list all the git tags via

git tag -n

• choose an appropriate tag

cd openssl
git checkout -f tags/OpenSSL_1_1_1l

• tar it up (in case you have to build again or checkout a different tag and build again)

 tar cvf ../openssl.tar ../openssl

• now configure

export PREFIX_DIR=/usr/local
make dclean
./config --prefix=$PREFIX_DIR --openssldir=$PREFIX_DIR 

• make (make sure you use -j n to speed it up)

make -j 5
make tests
make install

• now check the build

export LD_LIBRARY_PATH=/usr/local/lib:/usr/local/lib64:$LD_LIBRARY_PATH
export LD_RUN=$LD_LIBRARY_PATH
/usr/local/bin/openssl version

build openssh

• obtain openssh source

git clone https://github.com/openssh/openssh-portable

Running with this will result in building with the head version, you maylist the tags and choose a more stable version if you wish

git tag -n

Checkout

git checkout -f tags/<tag>

• configure the build

cd openssh-portable
autoreconf
export PREFIX_DIR=/usr/local
export LD_LIBRARY_PATH=$PREFIX_DIR/lib:$PREFIX_DIR/lib64:$LD_LIBRARY_PATH
export LD_RUN=$LD_LIBRARY_PATH
./configure --prefix=$PREFIX_DIR --exec-prefix=$PREFIX_DIR --with-ssl-dir=$PREFIX_DIR --libdir=$PREFIX_DIR --with-pam --with-md5-passwords --with-ipaddr-display --with-pie --with-privsep-path=/var/lib/sshd --sysconfdir=/etc/ssh --enable-pkcs11

• make

make -j 5
make tests
make install

• create a test configuration

vi /usr/local/etc/sshd_config 

# RBH - so I can test
Port 23
#ListenAddress 0.0.0.0
#ListenAddress ::
PermitRootLogin yes

• run a test

export LD_RUN=$LD_LIBRARY_PATH
/usr/local/sbin/sshd -f /usr/local/etc/sshd_config

modify openssh

Now you can make the Honey Pot changes, since you have proved that the above build works, and that sshd runs.

• the first change is to modify the version which is presented in the banner

vi version.h

#define SSH_VERSION     "XXX-XXX Honey Pot" // "OpenSSH_8.7"

• the next change is to remove the minor version and operating system et al from the sshd banner returned to every client.

vi ssh_api.c

• make the following changes to truncate the banner returned to the client (I am not sure I changed this much in the end as I changed SSH_VERSION afterwards)

/* Send our own protocol version identification. */
int
_ssh_send_banner(struct ssh *ssh, struct sshbuf *banner)
{
        char *cp;
        int r;

        // RBH
        if ((r = sshbuf_putf(banner, "SSH-2.0%s\r\n", SSH_VERSION)) != 0)
                return r;
        if ((r = sshbuf_putb(ssh_packet_get_output(ssh), banner)) != 0)
                return r;
        /* Remove trailing \r\n */
        if ((r = sshbuf_consume_end(banner, 2)) != 0)
                return r;
        if ((cp = sshbuf_dup_string(banner)) == NULL)
                return SSH_ERR_ALLOC_FAIL;
        debug("Local version string %.100s", cp);
        free(cp);
        return 0;
}

• now log failed password attempts (to auth.log)

vi auth-passwd.c

• include a call to logit() on failure for auth-passwd.c (logit is a macro defined in log.h)

...
#ifdef USE_PAM
        if (options.use_pam) {
                // RBH
                return (sshpam_auth_passwd(ssh, password) && ok);
        }
#endif
#if defined(USE_SHADOW) && defined(HAS_SHADOW_EXPIRE)
        if (!expire_checked) {
                expire_checked = 1;
                if (auth_shadow_pwexpired(authctxt))
                        authctxt->force_pwchange = 1;
        }
#endif
        result = sys_auth_passwd(ssh, password);
        if (authctxt->force_pwchange)
                auth_restrict_session(ssh);

        // RBH include the next 3 lines
        if (!result) {
                logit("Honey: failed attempt for Username<%s> Password<%s> IP<%s> port<%d>", authctxt->user, password,ssh_remote_ipaddr(ssh), ssh_remote_port(ssh));
        }
        return (result && ok);
}

• modify the parameter passed to sshpam_auth_passwd()

vi auth-pam.h

• update to

// RBH
int sshpam_auth_passwd(struct ssh*, const char *);

• include logit within auth-pam.c

 vi auth-pam.c

• update to reflect change below

/*
 * Attempt password authentication via PAM
 */
int
// RBH pass struct ssh * instead of Authctxt *
sshpam_auth_passwd(struct ssh *ssh, const char *password)
{
        // RBH obtain the authctxt
        Authctxt *authctxt = ssh->authctxt;
        
        int flags = (options.permit_empty_passwd == 0 ?
            PAM_DISALLOW_NULL_AUTHTOK : 0);
        char *fake = NULL;

        if (!options.use_pam || sshpam_handle == NULL)
                fatal("PAM: %s called when PAM disabled or failed to "
                    "initialise.", __func__);

        sshpam_password = password;
        sshpam_authctxt = authctxt;

        /*
         * If the user logging in is invalid, or is root but is not permitted
         * by PermitRootLogin, use an invalid password to prevent leaking
         * information via timing (eg if the PAM config has a delay on fail).
         */
        if (!authctxt->valid || (authctxt->pw->pw_uid == 0 &&
            options.permit_root_login != PERMIT_YES))
                sshpam_password = fake = fake_password(password);

        sshpam_err = pam_set_item(sshpam_handle, PAM_CONV,
            (const void *)&passwd_conv);
        if (sshpam_err != PAM_SUCCESS)
                fatal("PAM: %s: failed to set PAM_CONV: %s", __func__,
                    pam_strerror(sshpam_handle, sshpam_err));

        sshpam_err = pam_authenticate(sshpam_handle, flags);
        sshpam_password = NULL;
        free(fake);
        if (sshpam_err == PAM_MAXTRIES)
                sshpam_set_maxtries_reached(1);
        if (sshpam_err == PAM_SUCCESS && authctxt->valid) {
                debug("PAM: password authentication accepted for %.100s",
                    authctxt->user);
                return 1;
        } else {
                // RBH include logging on password failure
                logit("Honey: User<%s> Password<%s> IP<%s> port<%d>",authctxt->user, password, ssh_remote_ipaddr(ssh), ssh_remote_port(ssh));
                debug("PAM: password authentication failed for %.100s: %s",
                    authctxt->valid ? authctxt->user : "an illegal user",
                    pam_strerror(sshpam_handle, sshpam_err));
                return 0;
        }
}

• then build via:

make install

Installation

After you have run from /usr/local/sbin and thoroughly tested you can then re-configure and build for your machine using --prefix=/usr

Note: ensure that you use the same openssl version as what was installed on your machine when you install openssl from a source build, otherwise you could lose your package manager and all sorts of utlities.

It is suggested that you temporarily install telnetd on the target system so if you destroy sshd you can still telnet in headless.

• on the remote install

apt install telnetd

• on the client install

apt install telnet

I tested the source build on the following system and proved it using an external USB HDD so I could carry the results around across system rebuilds. So now to backup watchdog and try it.

spare raspberry pi 2

how to

• list all keys for known server

ssh-keygen -l -F watchdog.server -f ~/.ssh/known_hosts

• remove all keys for known server

ssh-keygen -R watchdog.server

• add keys for known server

ssh-keyscan -H watchdog.server >> ~/.ssh/known_hosts

• disable ssh password access https://www.cyberciti.biz/faq/how-to-disable-ssh-password-login-on-linux/
• 2FA Two_Factor_Authentication