Open in app

Sign up

Sign in

Write

Sign up

Sign in

CloudOps At Scale: Secure access patterns using Bastion Host and Transfer.Sh

Bhuvanesh
Searce
Published in
6 min readMar 4, 2019

--

In the Cloud world, security is paramount. Many cloud providers are still coming up with new features to make the infra secure. In this post, Here we are going to setup a bastion server to secure the SSH access on any Cloud infra like AWS, GCP, and Azure.

Is Bastion same as Jump host?

Actually not. Jump hosts are living in the same network as you want to connect. But the Bastion host is located somewhere else and need additional security options to access other networks.

At Searce, we manage cloud infrastructure for multiple customers across AWS, GCP and Azure on 24*7 basis. When have to provide access to our CloudOps teams — it’s not efficient making writing multiple firewall rules / security groups, hop on to customer’s bastion host etc. So we have deployed our own bastion Server to access all the customer's infra.

What’s new here?

Its just bastion host setup, why we need to blog about it?
There are so many blogs and documentation about setting up a bastion server. But here we have added few ingredients to the recipe.

  1. SSH session recording
  2. 2Factor authentication enabled.
  3. Transfer.sh for sharing the files between any servers.
  4. Automated user management.
  5. Easy to restore the users on the new bastion server.

Enable SSH Session Recording:

This option will track every command that an user executed along the command results. With the help of this, we can easily find out if someone did something wrong. This is already published in AWS Blog, but we have done this on GCP.

Create the directory for session recording log files.

mkdir /var/log/bastion

Here out admin user is sqladmin. So make this user to own this bastion log directory.

chown sqladmin:sqladmin /var/log/bastion
chmod -R 770 /var/log/bastion
setfacl -Rdm other:0 /var/log/bastion

Make OpenSSH execute a custom script on logins.

echo -e "\nForceCommand /usr/bin/bastion/shell" >> /etc/ssh/sshd_config

Remove some features from SSH and make it more restricted.

awk '!/AllowTcpForwarding/' /etc/ssh/sshd_config > temp && mv temp /etc/ssh/sshd_config
awk '!/X11Forwarding/' /etc/ssh/sshd_config > temp && mv temp /etc/ssh/sshd_config
echo "AllowTcpForwarding no" >> /etc/ssh/sshd_config
echo "X11Forwarding no" >> /etc/ssh/sshd_config

Create a shell script to capture the user's session.

mkdir /usr/bin/bastion
cat > /usr/bin/bastion/shell << 'EOF'if [[ -z $SSH_ORIGINAL_COMMAND ]]; thenLOG_FILE="`date --date="today" "+%Y-%m-%d_%H-%M-%S"`_`whoami`"
  LOG_DIR="/var/log/bastion/"echo ""
  echo "NOTE: This SSH session will be recorded"
  echo "AUDIT KEY: $LOG_FILE"
  echo ""SUFFIX=`mktemp -u _XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX`script -qf --timing=$LOG_DIR$LOG_FILE$SUFFIX.time $LOG_DIR$LOG_FILE$SUFFIX.data --command=/bin/bashelse
  echo "This bastion supports interactive sessions only. Do not supply a command"
  exit 1fi
EOFchmod a+x /usr/bin/bastion/shell
chown root:sqladmin /usr/bin/script
chmod g+s /usr/bin/script

Prevent bastion host users from viewing processes owned by other users, because the log file name is one of the scipt execution parameters.

mount -o remount,rw,hidepid=2 /proc
awk '!/proc/' /etc/fstab > temp && mv temp /etc/fstab
echo "proc /proc proc defaults,hidepid=2 0 0" >> /etc/fstab

Restart the SSH service to apply /etc/ssh/sshd_config modifications.

service sshd restart

You can sync the logs files which are in /var/log/bastion/* to S3, GCS or Azure Blob.

To verify this session recording, we can login to the bastion server and we’ll get the below welcome message.

Also we can find the log files.

ls /var/log/bastion/[root@XXXXXXX bhuvaneshwaran_r]# ls /var/log/bastion/
2019-02-26_06-38-27_bhuvaneshwaran_r_eZpI0sOqp3VGJi8z73sHV9nikMIZXY1Z.data
2019-02-26_06-38-27_bhuvaneshwaran_r_eZpI0sOqp3VGJi8z73sHV9nikMIZXY1Z.time
2019-02-26_17-55-02_bhuvaneshwaran_r_MFUQ71xnZ6tDrNfHSzIuP10vlXBmpgRT.time
2019-02-26_17-59-07_bhuvaneshwaran_r_yrOZFBA6XOzkNiMXYoNfRFz45jIWyD3N.data
2019-02-26_17-59-07_bhuvaneshwaran_r_yrOZFBA6XOzkNiMXYoNfRFz45jIWyD3N.time
2019-02-26_18-01-56_sqladmin_z1KsXV4J9cwWSJnPlbFi2AqfgbBkolh1.data
2019-02-26_18-01-56_sqladmin_z1KsXV4J9cwWSJnPlbFi2AqfgbBkolh1.time
2019-02-26_18-04-00_sqladmin_5DRUr5fh0OwVKMkh98sumeBFhrMsrjIn.data

Configure SMTP with Postfix and integrate AWS SES:

Before installing google authenticator and other things, we are going to install postfix mail agent and integrate with AWS SES. (if you have any own mail servers then you can ignore this part)

yum remove sendmail
yum install postfix
yum install cyrus-sasl-plain
alternatives --set mta /usr/sbin/postfix#Replace email-smtp.us-west-2.amazonaws.com with your SES region or SMTP server.
vi /etc/postfix/main.cf
smtp_tls_note_starttls_offer = yes
smtp_tls_security_level = encrypt
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous
relayhost = [email-smtp.us-west-2.amazonaws.com]:587
smtp_sasl_auth_enable = yes
smtp_use_tls = yes
smtp_tls_CAfile = /etc/ssl/certs/ca-bundle.crtvi /etc/postfix/sasl_passwd
[email-smtp.us-west-2.amazonaws.com]:587 ACCESSKEY:SECRETKEYsudo postmap hash:/etc/postfix/sasl_passwd
sudo chown root:root /etc/postfix/sasl_passwd /etc/postfix/sasl_passwd.db
sudo chmod 0600 /etc/postfix/sasl_passwd /etc/postfix/sasl_passwd.dbsudo postconf -e 'smtp_tls_CAfile = /etc/ssl/certs/ca-bundle.crt'
sudo postfix start; sudo postfix reload

Send the test message:

echo "test mail from postfix" | mailx -r "sender@domain.com (Bastion Admin)"  -s "Bastion host test" receiver.r@searce.com

Enable 2FA with Google Authenticator:

Install Google authenticator:

sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
sudo yum install google-authenticator

Setup Google Authentication in SSH Config

vi /etc/pam.d/sshd#remove this line or comment it.
auth       substack     password-auth#add this line in the end
auth required pam_google_authenticator.sovi /etc/ssh/sshd_config
#Please take a copy of the sshd_config file#Use custom SSH Port
Port 46285#Disable root login
PermitRootLogin no#Enable Key based authendication and disable password authendication
PubkeyAuthentication yes
PasswordAuthentication no#Enale PAM
UsePAM yes#My admin user is sqladmin, so I don't want to use MFA for this user, so if Google Authendicator is not working, I can login with this admin user and fix it. So exclude sqladmin user from MFA.Match User "sqladmin"
        AuthenticationMethods "publickey"
Match User "*,!sqladmin"
        AuthenticationMethods "keyboard-interactive,publickey"

Save the config file and restart it.

service sshd restart

Automate User Creation:

This shell script will help to create the user with google authenticator along with the Publickey authentication. Then automatically then send the private key and Google authenticator’s QR code in email.

Also it’ll sync the user’s public/private keys and google authenticator files to GCS bucket. (you can modify this script to sync this information to S3 or Azure blob.

Input values:

  1. Username: Name of the user
  2. Email: email address to send the private key and google QR code.
  3. Super User: Does this user need root access? If yes 'Y' else 'N'

Create an user and test the MFA:

./bastio-user-create.shUser Name: bhuvanesh
Email Address: bhuvaneshwaran.r@searce.com
Do you want Super User access [Y or N]: Y

Configure Transfer.sh on bastion:

Now we are going to setting up transfer.sh via docker.

yum install docker
service docker startdocker pull dutchcoders/transfer.sh:latest

I have a 100G volume and attached it to /transfer and I want all the uploaded files should go to this mount point.

  • tmp dir: /transfer/tmp
  • upload files location: /transfer/uploads
docker run -d --restart always --privileged=true -i -v "/transfer/uploads:/uploads" -v "/transfer/tmp:/tmp"  --publish 80:8080 dutchcoders/transfer.sh:latest --provider local --basedir /uploads --temp-path=/tmp

Open your browser and hit the IP of the bastion server.

You can use the custom port, SSL and etc in this Docker container. From our setup this port is only opened to customer’s jump host. So we’ll transfer the files from our computers to the bastion server and download it from customer’s jump box.

Conclusion:

This is our current setup, but you can do a lot of customizations. This bastion server only helps you to switch to Linux servers, not windows. Use custom port for bastion server’s SSH or restrict SSH port from the security group. You can take advantage of login page for transfer.sh web interface and many things.

We used the below lines inside the user creation shell script to transfer files in one command.

#Setup Transfer.sh alias
cat<<EOT >>/home/$username/.bashrc
transfer() {
    curl --progress-bar --upload-file "$1" http://10.10.10.10/$(basename $1) | tee /dev/null;
}
alias transfer=transfer
EOT
Bastion
Ssh
Security
Gcp
AWS

--

--

Bhuvanesh
Searce
Follow

Written by Bhuvanesh

836 Followers

CTO @ShellKode | https://thedataguy.in

Follow

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams