Monitoring your linux system is one of the main roles played by the administrators in their day-to-day life. Though linux is equipped with many monitoring tools that provide us many options to monitor the system the way we want it to. We will discuss few that are largely used, easy to remember and use. These tools can be built-in commands or programs and add-on tools.
Monitoring any system includes cpu usage, process activity, memory consumption, storage (disk) usage, network statistics and many more. These all criteria can be observed through command line and also there are a few graphical tools that can be useful too. We will first of all go through the built-in commands and tools that can help us in monitoring the linux machine.
Monitoring CPU
Using
Sysstat
You need to install this package from the ubuntu repository.
$ sudo apt-get install sysstat
After which we can do deep monitoring of cpu using
mpstat command.
$ mpstat
This command helps in monitoring smp box where multiple cpu are present We can use mpstat to display individual cpu utilization.
The command mpstat displays the average activities of all present processors.
$ mpstat -P ALL
This will display all the processors present in the system with their invidual stats.
On my laptop
baban@hashprompt:~$ mpstat -P ALL
Linux 3.2.0-35-generic (hashprompt)
Sunday 20 January 2013
_x86_64_
(4 CPU)
12:20:19 IST CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
12:20:19 IST all 5.09 0.01 3.47 1.41 0.00 0.03 0.00 0.00 89.99
12:20:19 IST 0 6.44 0.01 3.41 2.01 0.00 0.12 0.00 0.00 88.01
12:20:19 IST 1 4.09 0.01 3.43 2.24 0.00 0.02 0.00 0.00 90.22
12:20:19 IST 2 6.55 0.00 3.25 0.93 0.00 0.00 0.00 0.00 89.27
12:20:19 IST 3 3.29 0.00 3.79 0.49 0.00 0.00 0.00 0.00 92.44
baban@hashprompt:~$
On SMP box
HPC2:~ # mpstat -P ALL
Linux 2.6.16.60-0.35-default (
HPC2)
Sunday 20 January 2013
01:14:19 IST CPU %user %nice %sys %iowait %irq %soft %steal %idle intr/s
01:14:19 IST all 27.25 0.00 4.28 3.11 0.01 0.04 0.00 65.31 33521.58
01:14:19 IST 0 57.27 0.00 16.10 0.11 0.00 0.00 0.00 26.52 251.21
01:14:19 IST 1 61.99 0.00 21.39 0.09 0.00 0.00 0.00 16.53 250.30
01:14:19 IST 2 20.57 0.00 0.47 3.25 0.00 0.00 0.00 75.71 255.70
01:14:19 IST 3 14.18 0.00 0.30 2.35 0.00 0.00 0.00 83.17 264.58
01:14:19 IST 4 19.85 0.00 0.48 4.35 0.00 0.00 0.00 75.32 255.94
01:14:19 IST 5 19.73 0.00 0.52 4.44 0.00 0.00 0.00 75.31 252.76
01:14:19 IST 6 18.51 0.00 0.42 4.35 0.00 0.00 0.00 76.71 252.80
01:14:19 IST 7 17.23 0.00 0.46 4.41 0.00 0.00 0.00 77.90 253.14
01:14:19 IST 8 15.15 0.00 0.25 0.52 0.00 0.00 0.00 84.08 285.23
01:14:19 IST 9 11.64 0.00 0.17 0.69 0.00 0.00 0.00 87.49 256.51
01:14:19 IST 10 60.21 0.00 21.76 0.10 0.00 0.00 0.00 17.93 250.22
01:14:19 IST 11 51.33 0.00 20.29 0.53 0.00 0.00 0.00 27.85 250.50
01:14:19 IST 12 29.35 0.00 7.06 4.45 0.00 0.00 0.00 59.14 253.43
01:14:19 IST 13 18.03 0.00 1.25 4.46 0.00 0.00 0.00 76.26 252.86
01:14:19 IST 14 17.11 0.00 1.84 4.52 0.00 0.00 0.00 76.53 252.98
01:14:19 IST 15 19.60 0.00 1.92 4.49 0.00 0.00 0.00 73.99 253.35
01:14:19 IST 16 21.50 0.00 0.54 4.23 0.00 0.00 0.00 73.73 255.17
01:14:19 IST 17 24.73 0.00 0.46 4.20 0.00 0.00 0.00 70.61 253.58
01:14:19 IST 18 21.40 0.00 0.42 4.24 0.00 0.00 0.00 73.94 254.26
01:14:19 IST 19 18.53 0.00 0.43 4.42 0.00 0.00 0.00 76.62 254.53
01:14:19 IST 20 24.57 0.00 0.47 4.15 0.00 0.00 0.00 70.81 253.20
01:14:19 IST 21 19.74 0.00 0.46 3.91 0.00 0.00 0.00 75.90 252.75
01:14:19 IST 22 21.53 0.00 0.46 4.00 0.00 0.00 0.00 74.01 252.93
01:14:19 IST 23 18.15 0.00 0.55 4.18 0.00 0.00 0.00 77.13 257.21
01:14:19 IST 24 22.35 0.00 0.58 2.84 0.00 0.00 0.00 74.23 257.91
01:14:19 IST 25 23.74 0.00 0.52 2.79 0.00 0.00 0.00 72.95 252.30
01:14:19 IST 26 23.79 0.00 0.44 2.69 0.00 0.00 0.00 73.08 253.89
01:14:19 IST 27 27.48 0.00 0.44 2.74 0.00 0.00 0.00 69.34 254.86
-----output truncated-----
Using
SAR (System Activity Recorder)
To make the sysstat utility to collect system activity information we have to edit the file /etc/default/sysstat and set the option "ENABLED" as "true". Refer the below screenshot for more details.
$ sar -u
$ sar -u 2 5
$ sar -u -P ALL
%user - Percentage of CPU utilization that occurred while executing at the user level (application).
%nice - Percentage of CPU utilization that occurred while executing at the user level with nice priority.
%system - Percentage of CPU utilization that occurred while executing at the system level (kernel).
%iowait - Percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request.
%steal - Percentage of time spent in involuntary wait by the virtual CPU or CPUs while the hypervisor was servicing another virtual processor.
%idle - Percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request.
Using
iostat
$ sudo iostat -xtc
Using
vmstat
$ sudo vmstat 2 5
Monitoring Memory
Using
/proc/meminfo
The /proc/meminfo file stores memory usage statistics.
The above shown screenshot is the truncated output. As you can see that the total memory of my laptop is 3838124kB. Below calculation shall give us the the total memory used.
Mem Used = MemTotal - MemFree - Buffers - Cached
= 1434868kB
We can compare and verify this output with the gnome-system-monitor ouput.
As you can see both the outputs are same.
Using free
$ free -m
free -m command shows memory utilization in MB. Here, 1364MB is the actual used memory whereas 2383MB is the actual free memory.
$ free -m -t
This command displays the total memory present, utilized and free respectively which includes the swap memory also.
Using vmstat
$ vmstat 2 5
Using top
$ top
Using SAR
$ sar -r 2 5
The above command also helps to calculate the exact memory utilization in ubuntu.
Monitoring Processes
Using
Top
Top is a program that provides the real-time dynamic view of running system. By default it displays the cpu intensive processes that are ordered by highest percentage of CPU usage on the top. The top area contains the uptime, load average, no of users and processes and memory consumption.
Few hot-keys that have various usages in top are listed below.
1 - Displays the cpu states of multiple numbers processors present in the system.
A - Displays the top resource hungry processes
k - to kill a particular PID
z - to choose between colour and mono
Using
ps
ps is another built-in command which is very useful to monitor processes. It displays the snapshot of the running processes at particular instance of time the command is run. Below mentioned are few common examples of ps command.
$ ps -ejH
This command helps to view the process hierarchy.
PID PGID SID TTY TIME CMD
933 933 933 ? 00:00:00 bluetoothd
934 930 930 ? 00:00:03 rsyslogd
938 938 938 ? 00:00:03 NetworkManager
2235 2235 938 ? 00:00:00 dhclient
2348 2348 938 ? 00:00:00 dnsmasq
965 903 903 ? 00:00:13 polkitd
969 968 968 ? 00:00:00 avahi-daemon
972 968 968 ? 00:00:00 avahi-daemon
990 990 990 ? 00:00:00 cupsd
2543 2543 2543 tty7 01:15:55 Xorg
2641 1092 1092 ? 00:00:00 lightdm
2731 2731 2731 ? 00:00:00 gnome-session
2766 2766 2766 ? 00:00:00 ssh-agent
2783 2731 2731 ? 00:00:13 gnome-settings-
2822 2731 2731 ? 00:00:10 syndaemon
2814 2731 2731 ? 00:10:33 cinnamon
3034 2731 2731 ? 00:00:12 nemo
3099 2731 2731 ? 00:00:02 /usr/bin/termin
3109 2731 2731 ? 00:00:00 /usr/bin/termin <defunct>
3110 3110 3110 pts/0 00:00:00 bash
3166 3166 3110 pts/0 00:00:00 sudo
3176 3166 3110 pts/0 00:02:05 vmware
3797 2731 2731 ? 00:11:35 chromium-browse
3815 2731 2731 ? 00:00:11 chromium-browse
3921 2731 2731 ? 00:00:08 chromium-browse
9084 2731 2731 ? 00:09:58 chromium-browse
10311 2731 2731 ? 00:00:18 chromium-browse
22853 2731 2731 ? 00:00:06 /usr/bin/termin
22864 2731 2731 ? 00:00:00 /usr/bin/termin <defunct>
22865 22865 22865 pts/2 00:00:00 bash
23495 23495 22865 pts/2 00:00:00 ps
2844 2731 2731 ? 00:00:00 nm-applet
2848 2731 2731 ? 00:00:00 polkit-gnome-au
2852 2731 2731 ? 00:00:00 gnome-fallback-
3029 2731 2731 ? 00:00:00 gdu-notificatio
3063 2731 2731 ? 00:00:01 zeitgeist-datah
3064 2731 2731 ? 00:00:01 gnome-screensav
1106 1106 1106 ? 00:00:03 irqbalance
1278 1278 1278 ? 00:00:00 vmware-vmblock-
----- Output truncated -----
$ pstree
Shows the process hierarchy.
$ ps -Al
Shows long output of all processes.
$ ps -AlF
Shows long with some extra full output.
$ ps ax
Prints all processes
pgrep & ps -p
Displays only the process id(s) associated with the process.
pmap -d
Reports memory map of a process.
Using
pidstat
Lists all running processes and the resources that they are using.
Pidstat can be used with options like -u (CPU utilization) and -r (memory utilization) with time intervals.
System Running Status
Using
uptime
uptime gives you the duration for which the server has been up and running, number of users logged into the system and the load averages for the past 1min, 5mins and 15mins respectively.
$ uptime
User Stats
Using
w
w command lists the users currently logged into the system and the processes they are using.
$ w
We can also list the processes used by particular user.
$ w baban
NFS Statistics
$ nfsstat
Lists NFS statistics. To run this command we need to install the nfs-common package using below command.
$ sudo apt-get install nfs-common
Other Tools
Using
Gnome System Monitor
Gnome System Monitor tool is the best graphical monitor tool that comes with the Ubuntu operating system. This tool is very helpful in monitoring CPU, Memory & swap, Network, Processes and File Systems.
Using atop
This command is similar to top command. atop also displays used and free memory like top but differs only in some advanced options. atop does not come bundled with the os. It has to be installed from the ubuntu repository using the below mentioned command.
$ sudo apt-get install atop
$ atop
Using htop
htop is also a similar process as top and atop. It is very interactive as it gives a dynamic graphical representation of cpu utilization, memory and swap usage.
$ sudo apt-get install htop
$ htop
Using
Glances
Glances is another free monitoring tool like top/htop which can be used through text interface. It is developed in python. The most unique thing in this software is that this software can determine the status of system resources on the system and can indicate the status in various colors which helps in easy monitoring. Also there is a provision to set the threshold values for the indicators.
This software is maintained by
Nicolargo. You can download software and easy step-by-step instructions for installation can be found at
github.
Below shown is a screenshot of Glances on my system.
Using
GKrellM
GKrellM is a GTK-based stacked monitor program that charts SMP CPUs, memory, swap, disks, load, active net interfaces and internet connections.
It can be installed using the below mentioned command.
$ sudo apt-get install gkrellm
There are also various exciting skins available at
muhri.net. You can download the tarball of all skins from
here. Extract the tarball using below command.
$ sudo tar -xzvf GKrellM-Skins.tar.gz
After extraction you will find many skins in tar.gz format. To extract all .tar.gz files run the below command.
$ for i in *.tar.gz; do tar -xzvf $i; done
Now move all the extracted skin directories to themes directory inside .gkrellm2 directory of your home directory. To auto-start it you can add it to "Startup Applications". See the below screenshot for more details.
My system has got GKrellM with invisible skin. Heres what it looks like...
Using
Munin
Munin is a graphical monitoring tool for both network and system which presents its output as graph through a web interface.
Installation and configuration of munin can be found
here.
Using
Conky
It's a light-weight system monitor for X window system. Conky scripts are available for almost all desktop environments like Unity, Gnome, Mate, Cinnamon, etc. You can download the script and edit it as per your requirements. I too have a conky running on cinnamon desktop shown below.
There are lots of conky scripts all over the internet. I had some scripts from this
noobslab website.
Note: Please feel free to share any other commands and tools that I have missed.