Find it

Tuesday, August 30, 2011

Disable Large Segment Offload (LSO) in Solaris 10

In this blog article, I will share my understanding on Large Segment Offload (LSO). I got a task to disable the LSO on few of the servers (include zones).

Let's first understand what is LSO stands for and what is the purpose of using LSO.

As you see above LSO stands for Large Segment Offload.

TCP Offload Engine is an embryonic technology which is designed to offload TCP stack handling from the main system CPU to a processor built into NIC cards, hence the no CPU cycle and kernel time will get consumed.

LSO saves valuable CPU cycles by allowing the network protocol stack to handle large segments instead of the traditional model of MSS (TCP Maximum Segment Size) sized segments. In the traditional network stack, the TCP layer segments the outgoing data into the MSS sized segments and passes them down to the driver. This becomes computationally expensive with 10 GigE networking because of the large number of kernel functional calls required for every MSS segment. With LSO, a large segment is passed by TCP to the driver, and the driver or NIC hardware does the job of TCP segmentation (LSO offload the segmentation job on Layer 4 to the NIC driver). An LSO segment may be as large as 64 KByte. The larger the LSO segment, better the CPU efficiency since the network stack has to work with smaller number of segments for the same throughput.

So in simple words, use LSO for better network performance while reducing processor (CPU) utilization.

Segmentation is needed if a full TCP segment does not fit into the Ethernet Maximum Transmission Unit (MTU) size. With LSO, TCP segments do not need to get split in software implementation, this is done on the interface card hardware instead. Being much more effective, this improves the network performance while reducing the workload on the CPUs. LSO is most helpful for 10 Gigabit Ethernet network interfaces and on systems with slow CPU threads or lack of CPU resources.

Solaris LSO can be used if all of the three conditions are met :

1.The TCP/IP stack integrates LSO,
2.The Network Interface Card hardware supports it (for e.g. drivers like e1000g,ixgb,ixgbe etc),
3.The driver for this network card is capable of handling it.

Sadly, in most of the cases LSO seems to be not working that well hence it leads to disable the LSO support. Here is the ways to disable the LSO.

Ways to disable LSO -

Disable LSO by adding the following line in the /kernel/drv/e1000g.conf file (I’m using the e1000g interface/driver hence the file that I'm using is /kernel/drv/e1000g.conf) :

lso_enable=0,0,0,0,0,0,0,0;

After making the changes reboot is required or else if reboot is not possible then you can use ndd utility/command to disable it on a temporary basis and not persist across the reboot.

Using ndd you can disable it as shown below -

# ndd -set /dev/ip ip_lso_outbound 0

Also if you don't want to reboot the server after modifying the file /kernel/drv/e1000g.conf you can simply unplumb all of your e1000g interfaces with ifconfig, do "update_drv e1000g" to reload the .conf file, and then replumb and reconfigure the interfaces with ifconfig however still if I'm going to unplumb the network interfaces then eventually I'll be disturbing the services so reboot is the best option.

I had to disable the LSO as our application folks were experiencing slowness in their web application (response time etc.) It looks like LSO cause unstable connections & hence there are few observations like dropped sockets, dropped packets, packet reordering, packet retransmits and ultimately application folks observed slowness in their web application, NFS stuffs etc.

Friday, August 26, 2011

Installing Perl modules on Solaris 10 - An experience

In recent times I got a task to perform few Perl modules installation. To build the module in question (Compress::Raw::Zlib), I downloaded the module from CPAN (Comprehensive Perl Archive Network), verified that the MD5 checksum was correct, and used the following steps to compile the module:


# perl Makefile.PL
# make
# make install


The ‘make Makefile.PL’ completed successfully, but the make failed with the following errors:

# make
cp lib/Compress/Raw/Zlib.pm blib/lib/Compress/Raw/Zlib.pm
AutoSplitting blib/lib/Compress/Raw/Zlib.pm (blib/lib/auto/Compress/Raw/Zlib)
/usr/bin/perl /usr/perl5/5.8.4/lib/ExtUtils/xsubpp -typemap /usr/perl5/5.8.4/lib/ExtUtils/typemap -typemap typemap Zlib.xs > Zlib.xsc && mv Zlib.xsc Zlib.c
cc -c -I./zlib-src -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -xarch=v8 -D_TS_ERRNO -xO3 -xspace -xildoff -DVERSION=\"2.037\" -DXS_VERSION=\"2.037\" -KPIC "-I/usr/perl5/5.8.4/lib/sun4-solaris-64int/CORE" -DNO_VIZ -DGZIP_OS_CODE=3 -DUSE_PPPORT_H Zlib.c
/usr/ucb/cc: language optional software package not installed
*** Error code 1
make: Fatal error: Command failed for target `Zlib.o'

Above output indicate an error related to compiler & it is clearly showing that it is looking for cc complier however I was trying to build the module using gcc and not the cc (Sun studio) compiler but it seems like by default modules Makefile is referring to Sun Studio compiler and not the gcc complier. I do not have Sun Studio compiler on our systems & as this is being license software I can not even simply install it on any server due to compliance restrictions.

After doing a little bit research on the issue, I found that there is a way to fix this problem.  If you want to use gcc to build all Perl modules on a system, you can permanently remove the Sun Studio compiler references by adjusting the “cccdlflags” and “optimize” variables in /usr/perl5/5.8.4/lib/sun4-solaris-64int/Config.pm:

root@XXXX:/usr/perl5/5.8.4/lib/sun4-solaris-64int# egrep '(KPIC|O3)' Config.pm
cccdlflags='-KPIC'
optimize='-xO3 -xspace -xildoff'

After removing the above entries make was successful with gcc complier.

There is yet another method to do the Perl module installation even not performing above changes to Config.pm. You can simply build Perl module using –

- /usr/perl5/bin/perlgcc Makefile.PL
- /usr/sfw/bin/gmake
- /usr/sfw/bin/gmake test
- /usr/sfw/bin/gmake install

What we were discussing in above is how to install Perl modules manually and how to tackle with complier related issues. There is yet another automated method to install Perl module and which is most preferred and quick method to do it.

Simply export the http_proxy environment variable with your proxy server name or IP address and valid port number so that you can directly reach to the internet via proxy.


# http_proxy=myproxy:8080; export http_proxy
# perl -MCPAN -e shell

If this is the first time this script is run it will ask a lot of questions and store the results in Config.pm

Then simply run –

1. Start the CPAN shell script: /usr/perl5/bin/perl -MCPAN -e shell.
2. Install a module, e.g.: install CGI::Session
3. Exit the shell script: exit

To check the already installed modules execute –


# perl -e 'use HTML::Parser;'

If nothing is returned, Perl was able to locate the module. Otherwise, you will see “Can't locate HTML/Parser.pm in @INC”

One more experience to share here. I installed Threads Perl module however user was not able to execute his scripts due to current Perl distribution was not build or complied with “useithreads” configuration option hence he requested to recompile Perl distribution with threads option enable.

Here is a string that I use to compile Perl distribution with threads option.


# sh Configure -Dusethreads -Dprefix=/usr/perl5/5.14.1

With new Perl compilation user was able to execute his scripts with threads module.

Hope this document will help to those who are new to this work.