It's a UNIX system! I know this!

1

https://jurassicsystems.com/

So I’ve been immersing myself in the world of Linux for the last 12 months, to bring my decades of Windows experience in line.

I’ve been a command line non-enthusiast for basically my whole life, so I find that it takes a looooong time for me to memorize the commands that are essential, but I’m slowly getting there. This topic is a series of cheatsheets I’ll refer to as I go. If you have suggestions to improve anything I’ve posted here, please do!

:warning: I assume you are root for many of these commands. If you aren’t, you’ll get errors trying to do some of the sys-admin-y kinds of things I describe below. So use sudo -s to become root as you need to, and exit to stop being root.

:bell: When you are root your command line will end in #, and $ otherwise.

3 Likes
2

Initial Build and Burn-in testing

RAM

Memtest86, any version, from bootable media. One complete error-free pass. 64GB takes 2 hours, 128GB 4 hours, and so on.

OS / DISK

Install Ubuntu Server LTS x64.

:bell: Make sure you enable OpenSSH as part of the install so you can SSH in later over the network!

badblocks -sv /dev/sda

apt-get install smartmontools
smartctl -i /dev/sda 
smartctl -c /dev/sda
smartctl -t short /dev/sda
smartctl -l selftest /dev/sda

I consider installing an OS and a basic self check “good enough” for working disks. If you want to be ultra-sure, try the FreeNAS recommendations.

CPU

mkdir mprime
cd mprime
wget ftp://mersenne.org/gimps/p95v287.linux64.tar.gz
tar xzvf p95v287.linux64.tar.gz
rm p95v287.linux64.tar.gz

About 8 hours suffices in my experience.

Network

Set up server

apt-get install iperf
iperf -s

Set up client, connect to server IP

apt-get install iperf
iperf -c 10.0.0.1

Gigabit is 120 MB/sec or 960 Mbits/sec. 10 gigabit is 10× that :stuck_out_tongue:

2 Likes
3

Reconfiguring Network

What’s the network hardware look like, and the current status (up, down, whatever)

ip addr
ifconfig

Ubuntu 16 and earlier

Change the network config, IP addresses, etc

nano /etc/network/interfaces

sample

## primary
auto p3p1
iface p3p1 inet static
  address 10.0.0.180
  gateway 10.0.0.254
  netmask 255.255.255.0
  network 10.0.0.0
  broadcast 10.0.0.255

## secondary
auto p2p1
iface p2p1 inet dhcp

:bell: if you have multiple network interfaces, remove the gateway line from the one you don’t want to be the default gateway!

restart networking after changes

ifdown --exclude=lo -a && ifup --exclude=lo -a

:bell: loopback has to be excluded here or apparently weirdness ensues.

Ubuntu 18 and later

The files are at /etc/netplan now.

See How to configure a static IP address in Ubuntu Server 18.04 | TechRepublic

1 Like
4

Updating Ubuntu

Check major and minor version

lsb_release -a

Normal update

:bell: This includes point releases such as 14.02, 14.03 etc and is generally not very risky.

apt-get update
apt-get dist-upgrade

Set up automatic security updates

:loudspeaker: I strongly recommend you have automatic security updates on by default!

dpkg-reconfigure -plow unattended-upgrades

Clean up old updates

Ubuntu likes to keep old kernels around after updating, and they can take up considerable amounts of space, many gigabytes after a year. This will clean up the old, archived stuff:

apt-get autoclean && apt-get autoremove -y

Speed up boot

nano /etc/default/grub
GRUB_TIMEOUT=0
update-grub

:bell: Also check your BIOS for fast boot options, which can considerably shorten time spent in BIOS when restarting.

Full major version upgrade

:warning: This is a major in place upgrade, going from 14.xx to 16.xx and can be risky on a production system!

apt-get install update-manager-core
do-release-upgrade
1 Like
5

Have you considered vimtutor or vim-adventures.com? If you’re going to spend significant time remoted into other machines over SSH you will grow tired of nano's limitations, probably sooner rather than later. It’s an investment to learn, definitely, but at least you have good resources available.

ETA: I am aware of the irony of me showing up 15 minutes into your post to tell you “I’m not trying to start an editor war, but…” :smiley:

2 Likes
6

Websites / Internet

When does this site’s SSL cert expire?

echo | openssl s_client -connect example.com:443 2>/dev/null | openssl x509 -noout -dates

Who owns the domain and when does it expire?

whois example.com

Who owns / hosts this IP address?

dig -x 192.168.1.1

What’s in DNS?

dig example.com
dig example.com NS ← nameservers
dig example.com MX ← mailservers
dig example.com TXT ← text records
dig example.com A ← ipv4 records
dig example.com AAA ← ipv6 records
dig example.com CNAME ← CNAME records

How many network hops between me and this server?

traceroute example.com

1 Like
9

What the Heck is Going on With This Machine?

Memory

free -m

:bell: you want a swapfile rougly equal to memory size, otherwise processes may terminate due to out of memory errors under memory pressure

Disk

df -m

Show all files larger than 100M:

find / -xdev -type f -size +100M

Show largest directories on the whole drive

ncdu /

Use the ← and → keys to expand or collapse the largest directories.

CPU Load

load average 0.00, 0.01, 0.05

An idle computer has a load number of 0 and each process using or waiting for CPU (the ready queue or run queue) increments the load number by 1. Most UNIX systems count only processes in the running (on CPU) or runnable (waiting for CPU) states. However, Linux also includes processes in uninterruptible sleep states (usually waiting for disk activity), which can lead to markedly different results if many processes remain blocked in I/O due to a busy or stalled I/O system. This, for example, includes processes blocking due to an NFS server failure or to slow media (e.g., USB 1.x storage devices). Such circumstances can result in an elevated load average, which does not reflect an actual increase in CPU use (but still gives an idea on how long users have to wait).

  • last minute, last 5 minutes, last 15 minutes
  • load is not exactly CPU time, but tasks busy
  • load average equal to total cores (including threaded cores) is usually safe, much higher is a sign of trouble

CPU Usage

htop

Think of it in terms of resources: memory, disk, CPU. Which processes in htop are using the most of each?

  • F6 to sort
  • F4 to filter by name
  • F9 to kill processes
1 Like
10

Stop logging in as root

Generally you don’t want to log in as root, and you should avoid it.

  • On default Ubuntu installs the root account has no password so you can’t log in as root.

  • It’s better to use SSH keys for login, and Digital Ocean provides methods for this, but without SSH keys – you’ll log in as root.

  • You can disable SSH access for root, but that requires a second account.

Create a new account

sudo or be root first, then

adduser jane

This user will need sudo permissions, so

visudo

edit and add a line here:

# User privilege specification
root    ALL=(ALL:ALL) ALL
jane    ALL=(ALL:ALL) ALL

Log in as the new account

Log out and log back in via SSH as the new account, before making any other changes. Verify that you still have remote access and can sudo as you expect.

Disable Root SSH Login

nano /etc/ssh/sshd_config

Turn off root logins

PermitRootLogin no

Turn off passwords (in favor of SSH keys)

PasswordAuthentication no

restart the sshd service

service ssh restart

Verify that root is no longer allowed

Try to log in again via SSH as root; you shouldn’t be able to.

You’ve reduced the login attack surface considerably, since “root” is no longer a valid username to log in with via SSH. And that’s the most common point of attack.

1 Like
11

Copy files from a USB drive

Welcome to one of the most common search terms in the world: linux usb drive.

You know how on most operating systems you just plug in the USB drive and then copy to it? Well forget all that, because

The automatic way

You can install usbmount to get auto-mounting of USB drives.

Once you do this, you’ll see a bunch of USB drivers under /media:

ls /media
usb  usb0  usb1  usb2  usb3  usb4  usb5  usb6  usb7

When I plugged in my drive, it was available under /media/usb so presumably if you plug in more than one, you can disambiguate with the numbers.

It’s a little weird because all these mount points show up, and you can ls them, but they’ll be empty until you plug a USB drive in.

The manual way

lsblk
mkdir /media/usbstick
mount -t vfat /dev/sdb1 /media/usbstick

12

Finding files

Looking for a particular file and can’t find it? Be at the root, then

find . -type f -name 'filename'

15

Basic setup for new mini-servers

Install current version of Ubuntu Server LTS. Take all the defaults. Reboot.

Log in and grab your SSH key from the public Ubuntu listing service at https://launchpad.net/ – DO NOT DO THIS AS ROOT!

ssh-import-id {name}

now sudo -s to root and get the latest updates:

apt-get update
apt-get dist upgrade
apt autoclean
apt autoremove -y

Reboot again so any OS initiated microcode CPU updates, etc can take effect.

Make sure unattended upgrades are installed so the system will get critical updates automatically

dpkg-reconfigure -plow unattended-upgrades

Make sure multiple failed SSH login attempts are automatically blocked

apt-get install fail2ban -y

Turn on the built in firewall, so only web and SSH traffic is allowed

ufw allow http
ufw allow https
ufw allow ssh
ufw enable

Turn off SSH password and root logins, so the only way to log in via SSH is with proper key files

nano /etc/ssh/sshd_config

The lines you want to change read

PermitRootLogin no
PubkeyAuthentication yes
PasswordAuthentication no

Log out, wait a minute then :electric_plug: disconnect power to the box. Then re-connect power and make sure it comes back up! This usually requires you to set BIOS options for what happens for resume on power loss, but it is critical for a box in a server role, so you need to verify that if the power goes out, your server will automatically come back up.

1 Like