Particular thanks to:
"Eric J. Ostrander" <ejo@astro.phys.cmu.edu>
Tom Vayda <vayda_tom@jpmorgan.com>
David Robson <robbo@box.net.au>
Andy_Paton@worktech.co.uk
Ross Bennett <ross@gcs.com.au>
My original question is at the end of this message.
Both Andy and Ross came up with the real goodies in suggesting the
following kernel tuning additions to /etc/system:
set max_nprocs=4096
set strctlsz=4096
set shmsys:shminfo_shmmax=268435456
set shmsys:shminfo_shmseg=600
set shmsys:shminfo_shmmni=512
set shmsys:shminfo_shmmin=1
set semsys:seminfo_semmns=4096
set semsys:seminfo_semmni=4096
set semsys:seminfo_semmnu=4096
set semsys:seminfo_semume=64
set semsys:seminfo_semmap=512
set semsys:seminfo_semmsl=128
These are Lotus recommendations for running their product on Solaris. It
seems to have cured the problem.
Tom Vayda also suggested amending the notes.ini file to include
Disable_Locking=1
Apparently this will avoid several kinds of file locking problems. I have
to admit I haven't tried this yet. I'm waiting to see the impact of the
other changes first.
As background, Ross also came up with the following article which I found
very useful.
-----------------------------Article start
--------------------------------------------
Below are the recommended System tuning parameters to be used with Lotus
Domino 4.51 & Solaris 2.5.1. These recommendations have come from the
Lotus UNIX Support Team, Mr. Scott Hopper and Mr. Bruce Chapman - Sun
Software Engineer who is working with Lotus as part of the Notes UNIX
Development Team within Lotus .
This is all that needs to be set, the other parameters are carryovers from
3.x/4.1x where System V semaphores were used on the Solaris code those
setting are no longer required.
set max_nprocs=4096
set strctlsz=4096
*set shmsys:shminfo_shmmax=268435456
set shmsys:shminfo_shmseg=600
set shmsys:shminfo_shmmni=512
set shmsys:shminfo_shmmin=1
also there is the Notes_SHARED_DPOOLSIZE parameter:
For c-shell in the .cshrc add:
setenv Notes_SHARED_DPOOLSIZE 4096000
or bourne or korn shell add in the .profile or .kshrc, respectively:
Notes_SHARED_DPOOLSIZE= 4096000
export Notes_SHARED_DPOOLSIZE
* this parameter could be much less, however it is a limit for an
individual process. Notes will only consume whatever
NOTES_SHARED_DPOOLSIZE is set to.
The following information has been supplied to us by Mr. Bruce Chapman -
Sun Software Engineer who is working with Lotus as part of the Notes UNIX
Development Team within Lotus
Some facts about Notes_SHARED_DPOOLSIZE :
It is a misnomer. It does not limit size, but simply indicates
allocation chunk size for ALL shared memory (not only POOLs)... Maybe
it should have been "Notes_SHARED_ALLOC_SIZE" or the like, but it is
most closely tied with the NSF_BUFFER_POOL_SIZE, and so got its (bad)
name.
The actual amount of shared memory utilized is controlled by several
factors. NSF_BUFFER_POOL_SIZE is a (modifiable via notes.ini) variable
which controls the maximum size of the largest consumer of shared
memory. Other consumers of shared memory are the global variables, the
NIF buffer pool, and the NSF buffer control pool; while
NSF_BUFFER_POOL_SIZE is set to 1/4 physical memory by default, others
are either fixed in size or relatively small (total of all other shared
memory users is at most < 20MB, probably < 8MB).
There is an interaction between Notes and Solaris system 5 shared memory
such that, on some 2.5/2.5.1 systems, the Notes server cannot allocate
more than 100 shared memory segments. As such, the default 1MB
DPOOLSIZE can lead to the dreaded "...insufficient memory..." message
on a server if you have more than 320MB of physical memory (and hence
80MB NSF pool maximum + < 20MB max other shared memory) or have set the
NSF_BUFFER_POOL_SIZE manually in notes.ini to be more than 80MB (we have
always seen near peak results without setting this in notes.ini, but
letting the server do its 1/4 physmem calculation).
Given #3: there are occasions, when you need > 100MB total shared memory
for your notes server, that you must bump the value of
Notes_SHARED_DPOOLSIZE higher than it's default 1MB. You will not need
to do so unless you have >320MB RAM on the system or have overriden the
NSF_BUFFER_POOL_SIZE > 80MB. The Lotus support bulletin I've seen
mentions changing based upon the number of users; while the number of
users may be a rough approximation of how much of the max buffer pools
are utilized, it is not the correct criteria for setting this variable.
It also appears their base value for 100+ users is about 5 times higher
than it should be, and the recommendation of doubling that for 200+
users can only be
based on incorrectly interpreting the meaning of this variable
(because of it being named incorrectly, no doubt).
We performed several tests with varying DPOOLSIZEs and found little
difference between 1MB and 4MB (up until we hit the 100MB limit), but
some definite performance degradation (on the order of 7% or so) at the
high end loads using 16MB sizes. It was apparent that the less
frequent, yet more costly, allocation of larger chunks hurt performance;
and, unless you have a system with 1.6 GB RAM, 4MB DPOOLSIZE should be
adequate to prevent running out of shared memory.
Almost all of my information is from knowing the code and doing
benchmarks. If someone has hard evidence from actual customer sites
that shared memory usage is beyond the parameters I describe above, I
would really like to see it.
In Notes 4.52, we should not need any of this, as we plan to use
mmap()ed memory in place of system V shared memory on Solaris (by
default, although we plan to offer the old behavior if you want it).
Testing has also shown some performance improvement given these changes.
---------------------------Article end
----------------------------------------------
--------------------------------- My original question
--------------------------
Hi Guys
Normally I would hesitate to ask this list about an application like notes,
but my user is chuntering on about "this doesn't happen on my NT
boxes......" So here goes.
Boxes are both SparcStation 20s, configured as below:
As you can see, both have 128Mb RAM and are running at 2.5 and 2.5.1.
Notes 4.5 for Solaris is installed on both.
NSF Buffer pool size in Notes is set to 128000000 (slightly smaller than
the maximum allowable size of 128M). Notes stats memory report is "plentiful"
vmstat reports bugger all happening while the notes server is running (in
background, not attached to an openwin console)
procs memory page disk faults cpu
r b w swap free si so pi po fr de sr s1 s3 -- -- in sy cs us sy id
0 0 0 201104 61604 0 0 6 0 0 0 0 0 1 0 0 25 73 70 0 0 99
0 0 0 192044 46424 0 0 0 0 0 0 0 1 7 0 0 52 34 72 0 0
100
0 0 0 192044 46424 0 0 0 0 0 0 0 0 0 0 0 21 36 70 0 0
100
0 0 0 192044 46424 0 0 0 0 0 0 0 0 0 0 0 19 36 69 0 0
100
0 0 0 192044 46424 0 0 0 0 0 0 0 0 0 0 0 21 35 69 0 0
100
0 0 0 192044 46424 0 0 0 0 0 0 0 0 0 0 0 21 60 70 1 0 99
0 0 0 192044 46424 0 0 0 0 0 0 0 0 0 0 0 20 36 68 0 0
100
0 0 0 192044 46424 0 0 0 0 0 0 0 0 0 0 0 20 34 68 0 0
100
But whenever my user (singular at this point) opens a database (of any
size) or accesses the server from either a client of the console, notes
reports "Insufficient memory available - increase NSF_BUFFER_POOL_SIZE",
the box panics and reboots.
This activity is consistent on both 20s. Ironically, the same software runs
with no problems on pentium based NT servers with only 32Mb. I know less
than zilch about Notes (and it is entirely possible that my installation on
the 20s is at fault.) But the last thing I want to see is a bunch of NT
boxes creeping onto my net.
If anybody has any ideas at all, I'll be more than grateful
Mick
--------- config of 20s -----
Box1
DATE: Fri Jun 13 17:42:10 BST 1997
USER: root
HOSTNAME: le0 = XXXXXXXX
IP ADDRESS: le0 = XXXXXXXXX
MODEL: SUNW,SPARCstation-20
CPU: SUNW,sx
CPU: TI,TMS390Z55
CPU: TI,TMS390Z55
CPU: TI,TMS390Z55
CPU: TI,TMS390Z55
FRAME BUFFER(S): cgsix
SunOS RELEASE: 5.5
TYPE: unknown (/=unknown, swap=, /usr=unknown, /home=unknown)
MEMORY: 128MB
SWAP: 226.6MB total, 38.8MB used, 187.8MB available
LOAD AVERAGE: 0.16, 0.04, 0.03
DEFAULT PRINTER: none
ETHERNET ADDRESS: XXXXXXXXXXXXXX
HOSTID: XXXXXXXXXX
DISK: c0t1d0 "SUN1.05" (1050MB unformatted)
PARTITION: c0t1d0s0 mounted on /home (846MB, 26% full)
DISK: c0t3d0 "SUN1.05" (1050MB unformatted)
PARTITION: c0t3d0s0 mounted on / (51MB, 26% full)
PARTITION: c0t3d0s6 mounted on /usr (253MB, 34% full)
PARTITION: c0t3d0s5 mounted on /opt (432MB, 32% full)
SWAP PARTITION: c0t3d0s1 (128MB)
TAPE: rst4 - unknown (no tape loaded)
Box2
DATE: Fri Jun 13 19:02:04 BST 1997
USER: root
HOSTNAME: le0 = XXXXXXXXXXXX
IP ADDRESS: le0 = XXXXXXXXXXX
MODEL: SUNW,SPARCstation-20
CPU: SUNW,sx
CPU: Ross,RT626
FRAME BUFFER(S): cgthree
SunOS RELEASE: 5.5.1
TYPE: unknown (/=unknown, swap=, /usr=unknown, /home=unknown)
MEMORY: 128MB
SWAP: 227.9MB total, 35.1MB used, 192.8MB available
LOAD AVERAGE: 0.15, 0.05, 0.03
DEFAULT PRINTER: none
ETHERNET ADDRESS: XXXXXXXXXXXXXXXX
HOSTID: XXXXXXXX
DISK: c0t1d0 "SUN2.1G" (2126MB unformatted)
PARTITION: c0t1d0s6 mounted on /home/home1 (1716MB, 8% full)
DISK: c0t3d0 "SUN2.1G" (2126MB unformatted)
PARTITION: c0t3d0s0 mounted on / (27MB, 38% full)
PARTITION: c0t3d0s6 mounted on /usr (108MB, 64% full)
PARTITION: c0t3d0s4 mounted on /var (108MB, 3% full)
PARTITION: c0t3d0s7 mounted on /home (1036MB, 10% full)
PARTITION: c0t3d0s5 mounted on /opt (216MB, 5% full)
PARTITION: c0t3d0s3 mounted on /usr/openwin (108MB, 85% full)
SWAP PARTITION: c0t3d0s1 (128MB)
----------------------------------------------------------------------------
----
Mick Morgan Tel: 01603-704859
Advanced Technology Group Fax: 01603-704817
CCTA http://www.open.gov.uk/
-----------------#include<standard_disclaimer.h>-------------------
----------------------------------------------------------------------------
----
Mick Morgan Tel: 01603-704859
Advanced Technology Group Fax: 01603-704817
CCTA http://www.open.gov.uk/
-----------------#include<standard_disclaimer.h>-------------------