Blog #6 – Memory usage
A. When to use these tools
Most of ASE’s memory usage will be in shared memory. However, there are times when you may want to examine what other memory ASE may be using. For instance, if your system is showing signs of memory contention you may want to verify exactly what memory ASE is using, and what it is using it for. Note also that these tools can be used to examine other processes that may be running on the system to take a look at their memory usage as well.
B. What to look for
An important concept to understand is that on Unix/Linux a process will have two values for memory usage. There is virtual memory, which is (roughly) the amount of memory the process has told the OS that it might use; and there is resident set memory, which is the actual physical memory in use by the process. The key concept is that virtual memory has no direct relationship to how much memory a process is using. To illustrate, here are a couple examples taken from a Linux system using the ‘ps’ command. The VSZ column is the virtual memory size and the RSS column is the resident set size. On linux these sizes are given in KB.
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 2441 0.0 0.0 535296 5200 ? Ssl Jul08 0:08 automount --pid-file /var/run/autofs.pid
dputz 21350 9.4 1.0 618624 156588 ? Ssl 11:27 0:01 /work/dputz/ase157/ASE-15_0/bin/dataserver
Here we see that the automount system daemon has over 500MB of virtual memory but is only using about 5MB of real memory. The dataserver process has just a little more virtual memory (600MB) but is using 156MB of real memory.
When we are looking at memory usage we will want to concentrate on the real memory in use by a process – that is what is actually taking up space in the system’s memory.
C. What tools to use
There are several useful tools we can use to look at memory usage. In the section above we saw how the ‘ps’ command can show the resident set size on Linux. In addition, there are utilities such as pmap, procmap, ipcs, and top that can let us look at processes memory usage. These can vary quite a bit by platform; here is a short list of what is available and what they show:
Linux
------
- ps aux - Shows VSZ (virtual memory size) and RSS (resident set size) for all processes on the
system.
- pmap -x <process id> - Shows the Kbytes (virtual memory size) and RSS (resident set size) for
each address space in the processes memory map. Note that this will
show memory usage for all shared libraries separately as well. The ASE
shared memory will show up as well – see the example 3. below.
- ipcs -ma - Shows the shared memory segments in use on the system. If the “nattch” column
shows a 0 that indicates that there are no running programs currently attached.
- top -a – Shows the top users of memory, sorted in descending order by resident set size (RES).
- cat /proc/meminfo – Can be used to help show if ASE is using huge pages.
Solaris
--------
- ps -elfy - Shows SZ (virtual memory size) and RSS (resident set size) for all processes on the
system.
- pmap -x <process id> - Shows the Kbytes (virtual memory size) and RSS (resident set size) for
each address space in the processes memory map. Note that this will
show memory usage for all shared libraries separately as well. The ASE
shared memory will show up as well.
- ipcs -ma - Shows the shared memory segments in use on the system. If the “NATTCH” column
shows a 0 that indicates that there are no running programs currently attached.
- top -o res – Shows the top users of memory, sorted in descending order by resident set size (RES).
AIX
---
- ps axv - Shows SIZE (virtual memory size for just the data segment of the process) and RSS
(resident set size) for all processes on the system.
- procmap <process id> - Shows the virtual memory size for each address space in the
processes memory map. Note that this will show memory usage for all shared libraries
separately as well. Note that on AIX the ASE shared memory will not show up in this list.
- ipcs -ma - Shows the shared memory segments in use on the system. If the “NATTCH” column
shows a 0 that indicates that there are no running programs currently attached.
- topas -P – Shows an output similar to the ps command. It will include the DATA RES and TEXT
RES columns, which make up the processes resident set size. This is an interactive
display, you can pick which column to sort on by using the left and right arrow keys.
- svmon -P <process id> - Shows a list of all memory segments and sizes for the specified process.
It will also show memory usage by pagesize, which can help identify if ASE is using any
large pages.
HP-UX
------
- ps -elf - Shows SZ (resident set size) for all processes on the system. Note that on HP-UX you can
set the environment variable UNIX_STD=2003 and get both resident set size (SZ) an
virtual memory size (VSZ) by running ps with a command line like
ps -A -o pid -o pcpu -o sz -o vsz -o comm
- pmap <process id> - Shows the VSZ(virtual memory size) and RSZ (resident set size) for
each address space in the processes memory map. Note that this will
show memory usage for all shared libraries separately as well. The ASE
shared memory will show up as well at the end of this output.
- ipcs -ma - Shows the shared memory segments in use on the system. If the “NATTCH” column
shows a 0 that indicates that there are no running programs currently attached.
- top - Shows the top users of cpu, sorted in descending order by current cpu usage. Virtual
memory size (SIZE) and resident set size(RES) are shown.
- glance – This is an HP-UX interactive utility. To see processes sorted in order by the resident set
size; on the main screen enter “o” followed by “1”. Skip down to the “sort key” and
type “rss” , enter, then “y”.
D. Examples
- A question arises on a Linux system showing memory paging regarding exactly how much memory ASE is using. The ASE “max memory” is set to 3720000 (about 7.5GB)
and “allocate max shared memory” is set to 0.
- ps output
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
sybase 21293 0.6 5.9 8382844 906676 ? Ssl 11:25 0:57 /work2/ase16/ASE-16_0/bin/dataserver
- pmap output
Address Kbytes RSS Dirty Mode Mapping
0000000000400000 32720 12492 0 r-x-- dataserver
00000000025f3000 5684 2048 172 rwx-- dataserver
0000000002b80000 1488 208 208 rwx-- [ anon ]
0000000004b4a000 1352 1220 1220 rwx-- [ anon ]
0000000144c6c000 7341860 885920 885920 rwxs- [ shmid=0xe8012 ]
0000003735e00000 128 116 0 r-x-- ld-2.12.so
000000373601f000 4 4 4 r-x-- ld-2.12.so
0000003736020000 4 4 4 rwx-- ld-2.12.so
…
---------------- ------ ------ ------
total kB 8382844 906676 888784
Note that on Linux, “Dirty” pages are those that the process has written to; meaning that if the system needed to reuse that memory it would forst have to write those pages out to swap space.
From these two outputs we can see that, while the virtual memory size of ASE is quite large (over 8GB) the actual resident set memory being used is less than 1GB.
2. On AIX, the system administrator has collected ‘ps’ output that shows the SZ column
for each of 8 ASE engines to be much less than the configured ‘max memory’ of about 2GB. They want to know how much memory ASE is really using.
- The issue here is that AIX does not include shared memory in the resident set size
- The ps avxw output shows:
PID TTY STAT TIME PGIN SIZE RSS LIM TSIZ TRS %CPU %MEM COMMAND
8060938 pts/5 A 0:00 0 10768 33292 xx 44555 22524 0.0 0.0 /sybase/ase157/ASE-15_0/bin/dataserver -ONLINE:7,0
8192142 pts/5 A 0:01 0 10764 33288 xx 44555 22524 0.0 0.0 /sybase/ase157/ASE-15_0/bin/dataserver -ONLINE:3,0
8847466 pts/5 A 0:00 21 12096 34620 xx 44555 22524 0.0 0.0 /sybase/ase157/ASE-15_0/bin/dataserver -ONLINE:1,0
7864604 pts/5 A 0:50 1203 13624 36148 xx 44555 22524 0.1 0.0 /sybase/ase157/ASE-15_0/bin/dataserver -sdwpaix157
7930130 pts/5 A 0:00 2 10768 33292 xx 44555 22524 0.0 0.0 /sybase/ase157/ASE-15_0/bin/dataserver -ONLINE:2,0
8847650 pts/5 A 0:00 0 10768 33292 xx 44555 22524 0.0 0.0 /sybase/ase157/ASE-15_0/bin/dataserver -ONLINE:6,0
8913190 pts/5 A 0:00 0 10768 33292 xx 44555 22524 0.0 0.0 /sybase/ase157/ASE-15_0/bin/dataserver -ONLINE:5,0
8978768 pts/5 A 0:01 9 11996 34520 xx 44555 22524 0.0 0.0 /sybase/ase157/ASE-15_0/bin/dataserver -ONLINE:4,0
The RSS columns (in KB units on AIX) all add up to only about 270 MB. In order to calculate the actual total memory in use we will need to add in the size of the
shared memory. To get that, we can use the “ipcs –ma” command. It shows:
T ID KEY MODE OWNER GROUP CREATOR CGROUP NATTCH SEGSZ CPID LPID ATIME DTIME CTIME
m 78643205 0x970ef015 --rw------- sybase ipsec sybase ipsec 8 2252800000 7864604 8060938 12:04:48 12:04:48 12:04:39
The SEGSZ gives us the size of the shared memory (in bytes) which we can see matches the configured “max memory”.
c. Another way to look at this would be to use the “svmon” command; taking one of the dataserver process ids:
svmon -P 7930130
-------------------------------------------------------------------------------
Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd 16MB
7930130 dataserver 610150 558551 0 604497 Y Y N
PageSize Inuse Pin Pgsp Virtual
s 4 KB 562662 550007 0 557009
m 64 KB 2968 534 0 2968
L 16 MB 0 0 0 0
This shows 562662 4KB pages and 2968 64KB pages in use; which all added up comes to about 2.4GB.
3. Someone has asked whether ASE running on Linux is experiencing a “memory leak”; which is a condition where memory gets allocated by a program but never freed. They have monitored the resident set size of ASE from boot time and see the following pattern:
At boot – resident set size = 1023000
After running 1 hour resident set size was 1392248
After running 2 hours it was 3101496
After running 3 hours it was 3588204
After running 4 hours it was 3925660
Then after running a large batch application they saw it grow to 7952832
After that, it stayed fairly constant, with little change for a couple of weeks.
They questioned whether this showed that ASE had some sort of memory leak as it booted up and ran for a while.
Using the pmap –x utility we were able to show that all of the growth occurred in the
Shared memory segment. On linux, that line in the pmap output looked like:
Address Kbytes RSS Dirty Mode Mapping
0000000000400000 32720 15064 0 r-x-- dataserver
00000000025f3000 5684 2648 172 rwx-- dataserver
0000000002b80000 1488 208 208 rwx-- [ anon ]
0000000003df3000 1352 1220 1220 rwx-- [ anon ]
0000000143f15000 7341860 6977384 3977384 rwxs- [ shmid=0x130012 ]
…
This showed that the resident set size growth was due entirely to ASE using more and more of the data cache spaces in the shared memory segment. There was no memory leak; this is normal and expected behavior as ASE uses more and more of the cache.
E. Conclusion
Using the system tools to look at memory usage can help determine what the complete memory usage of ASE (and other processes) is. This will help in diagnosing any issues that might arise from a system being low on memory (such as memory paging). i