You are hereFeed aggregator
Feed aggregator
Hardware Engineer
Planet Illumos - Tue, 02/14/2012 - 22:24
Back at Fishworks, my colleagues had a nickname for me: Adam Leventhal, Hardware Engineer. I wasn’t designing hardware; I wasn’t even particularly more involved with hardware specs. The name referred to my preternatural ability to fit round pegs into square holes, to know when parts would bend but not break (or if they broke how [...]
Solaris 11: Setting up Sendmail / SASL to handle SMTP AUTH
Planet Illumos - Tue, 02/14/2012 - 22:24
I thought I would finally get SMTP AUTH working on my Solaris 11 servers, so that I could then setup my phone to send e-mail correctly. If you have seen my previous post you will see that Solaris 11 can't do it with the standard software.Build Cyrus SASLInstall the packages which we requite and download Cyrus SASL Library# sudo pkg install pkg:/developer/build/make system/header developer/gcc-3 text/gnu-grep database/berkeleydb-5# echo $PATH/usr/sbin:/usr/bin# export PATH=$PATH:/usr/gnu/bin:/usr/sfw/binAt this time it will not compile, so alter the following lines.# gzip -dc cyrus-sasl-2.1.25 | tar xf - # cd cyrus-sasl-2.1.25 # vi ./lib/saslutil.c ./plugins/kerberos4.c ./plugins/digestmd5.c utils/Makefile.am./lib/saslutil.c85c85extern int gethostname(char *, int);change to:extern int gethostname(char *, unsigned int);./plugins/kerberos4.c113c113extern int gethostname(char *, int);change to:extern int gethostname(char *, unsigned int);./plugins/digestmd5.c106c106extern int gethostname(char *, int);change to:extern int gethostname(char *, unsigned int);./utils/Makefile.am59a60,62add the following lines:saslpasswd2_LDFLAGS = -rpath $(libdir)dbconverter_2_LDFLAGS = -rpath $(libdir)pluginviewer_LDFLAGS = -rpath $(libdir)Now we are ready for building.# ./configure --enable-cram --enable-digest --enable-plain \ --enable-login --disable-krb4 --disable-anon \ --disable-gssapi --with-saslauthd=/var/run/saslauthd# make# sudo make install Add the following links:# sudo ln -s /usr/local/lib/sasl2 /usr/lib/sasl2# sudo ln -s /usr/local/lib/libsasl2.so.2.0.25 /usr/lib/libsasl2.so.2Build SendmailDownload sendmail 8.14.5Stop existing sendmail processes. # sudo svcadm disable svc:/network/smtp:sendmail# sudo svcadm disable svc:/network/sendmail-client:default# gzip -dc sendmail.8.14.5.tar.gz | tar xf - # cd sendmail-8.14.5Set up sendmail FEATURES (SSL, SASL, DB) # cat > devtools/Site/site.config.m4dnl ### Changes to disable the default NIS supportAPPENDDEF(`confENVDEF', `-UNIS')dnl ### Changes for STARTTLS supportAPPENDDEF(`confENVDEF',`-DSTARTTLS')APPENDDEF(`confLIBS', `-lssl -lcrypto')APPENDDEF(`confLIBDIRS', `-L/usr/local/lib -L/usr/lib -R/usr/lib')APPENDDEF(`confINCDIRS', `-I/usr/include/openssl')dnl ### SASL supportAPPENDDEF(`confENVDEF', `-DSASL=2')APPENDDEF(`conf_sendmail_LIBS', `-lsasl2')APPENDDEF(`confINCDIRS', `-I/usr/local/sasl -I/usr/include')dnl ### Berkley DB supportAPPENDDEF(`confENVDEF', `-DNEWDB')APPENDDEF(`confINCDIRS', `-I/usr/include')APPENDDEF(`confLIBDIRS', `-L/usr/lib')dnl ### TCP WrapperAPPENDDEF(`confENVDEF', `-DTCPWRAPPERS')APPENDDEF(`conf_sendmail_LIBS', `-lwrap')EOF# Build Sendmail # ./Build -c/* It should build but ignore the gtroff:error at the end */# sudo mv /usr/lib/sendmail /usr/lib/sendmail_orig# sudo cp obj.SunOS.5.11.i86pc/sendmail/sendmail /usr/lib/sendmail# sudo sudo chgrp smmsp /usr/lib/sendmail# sudo sudo chmod g+s,u-ws /usr/lib/sendmailNow a Vendor and SASL support to sendmail.cf file (I hope you know what you are doing here)Edit sendmil.mc add add the extra lines:define(`VENDOR_NAME', `Berkeley')define(`confAUTH_OPTIONS', `A')dnlTRUST_AUTH_MECH(`EXTERNAL DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnldefine(`confAUTH_MECHANISMS', `DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl# cd /etc/mail/cf/cf# cat sendmail.mcdivert(0)dnlVERSIONID(`sendmail.mc (Sun)')define(`VENDOR_NAME', `Berkeley')OSTYPE(`solaris11')dnlDOMAIN(`solaris-generic')dnldefine(`confAUTH_OPTIONS', `A')dnlTRUST_AUTH_MECH(`EXTERNAL DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnldefine(`confAUTH_MECHANISMS', `DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl define(`confFALLBACK_SMARTHOST', `mailhost$?m.$m$.')dnlMAILER(`local')dnlMAILER(`smtp')dnlLOCAL_NET_CONFIGR$* < @ $* .$m. > $* $#esmtp $@ $2.$m $: $1 < @ $2.$m. > $3## sudo make sendmail.cf# sudo cp sendmail.cf ../../sendmail.cf# sudo /usr/lib/sendmail -bt -d0 < /dev/nullVersion 8.14.5Compiled with: DNSMAP LOG MAP_REGEX MATCHGECOS MILTER MIME7TO8 MIME8TO7 NAMED_BIND NDBM NETINET NETINET6 NETUNIX NEWDB PIPELINING SASLv2 SCANF STARTTLS TCPWRAPPERS USERDB XDEBUGSetup sasl authentication serverNow lets set up SASL plugin via a Sendmail.conf# sudo cat > /usr/local/lib/sasl2/Sendmail.confpwcheck_method: saslauthdEOF#For authentication to work the saslauthd has to be started, so you can start in as a one off process (good for debugging) and at boot time.# sudo /usr/local/sbin/saslauthd -n 1 -V -d -a pamsaslauthd[398] :main : num_procs : 1saslauthd[398] :main : mech_option: NULLsaslauthd[398] :main : run_path : /var/run/saslauthdsaslauthd[398] :main : auth_mech : pamsaslauthd[398] :ipc_init : using accept lock file: /var/run/saslauthd/mux.acceptsaslauthd[398] :detach_tty : master pid is: 0saslauthd[398] :ipc_init : listening on socket: /var/run/saslauthd/muxsaslauthd[398] :main : using process modelsaslauthd[398] :get_accept_lock : acquired accept lockTo set it up ant boot time we should set up a services, but for now we will stick to init.d files.cat > /etc/init.d/saslauthd#!/usr/sbin/sh#NAME=saslauthdDAEMON="/usr/local/sbin/${NAME}"DESC="SASL Authentication Daemon"# -a Selects the authentication mechanism to use.# -n Number of worker processes to create.# -V Enable verbose logging# -d Debugging (don't detach from tty, implies -V)case "$1" in'start')${DAEMON} -n 1 -V -d -a pam;;'stop')pgrep saslauthd;;*)echo "Usage: $0 { start | stop }"exit 1;;esacEOF## sudo ln init.d/saslauthd rc0.d/K38saslauthd # sudo ln init.d/saslauthd rc1.d/K38saslauthd # sudo ln init.d/saslauthd rc2.d/S82saslauthd # sudo ln init.d/saslauthd rcS.d/K38saslauthdTime for some testing, via first the SASL daemon.# cyrus-sasl-2.1.25/saslauthd/testsaslauthd -u RealUser -p MyPassword0: OK "Success."We need the encrypted username and password to test sendmail:# perl -MMIME::Base64 -e 'print encode_base64("\000MyUser\000MyPassword")'AE15VXNlcgBNeVBhc3N3b3Jk Test the sendmail part:# sudo /usr/lib/sendmail -bv -O LogLevel=14 -bs -Am220 mailer5.dcs.bbk.ac.uk ESMTP Sendmail 8.14.5/8.14.5; Tue, 31 Jan 2012 09:52:24 GMTEHLO localhost250-mailer5.dcs.bbk.ac.uk Hello root@localhost, pleased to meet you250-ENHANCEDSTATUSCODES250-PIPELINING250-EXPN250-VERB250-8BITMIME250-SIZE250-DSN250-ETRN250-AUTH DIGEST-MD5 CRAM-MD5 LOGIN PLAIN250-DELIVERBY250 HELPAUTH PLAIN AE15VXNlcgBNeVBhc3N3b3Jk235 2.0.0 OK Authenticatedquit221 2.0.0 mailer5.dcs.bbk.ac.uk closing connectionThat is it.refernces:- Guía Instalación de Sendmail con SMTP-AUTH y (tema #2688)- Sendmail-SMTP-AUTH-TLS-Howto- OpenSolaris Sendmail Auth 2012-02-01T13:36:00Z Andrew Watkins noreply@blogger.com
Can Solaris 11 Sendmail / SASL handle SMTP AUTH
Planet Illumos - Tue, 02/14/2012 - 22:24
I thought I would finally get SMTP AUTH working on my Solaris 11 servers, so that I could then setup my phone to send e-mail correctly.So can you do it with default Solaris 11 software?If you can't wait the answer is NO! but here is what I tried just in case someone can solve the problem.First Solaris 11 comes with Sendmail and SASL, but worst luck Sendmail has not been built with SASL support (you see the following when you edit sendmail.cf. see later).# sudo /usr/lib/sendmail -d0 -bt < /dev/nullVersion 8.14.5+Sun Compiled with: DNSMAP LDAPMAP LOG MAP_REGEX MATCHGECOS MILTER MIME7TO8MIME8TO7 NAMED_BIND NDBM NETINET NETINET6 NETUNIX NEWDB NISPIPELINING SCANF STARTTLS TCPWRAPPERS USERDB USE_LDAP_INITXDEBUGWarning: Option: AuthMechanisms requires SASL support (-DSASL)Warning: Option: AuthOptions requires SASL support (-DSASL)Lets try compiling sendmail to see if Solaris SASL can work!Install the packages which we requite and download sendmail 8.14.5# sudo pkg install pkg:/developer/build/make system/header developer/gcc-3 text/gnu-grep database/berkeleydb-5# echo $PATH/usr/sbin:/usr/bin# export PATH=$PATH:/usr/gnu/bin:/usr/sfw/binStop existing sendmail processes. # sudo svcadm disable svc:/network/smtp:sendmail# sudo svcadm disable svc:/network/sendmail-client:default# gzip -dc sendmail.8.14.5.tar.gz | tar xf - # cd sendmail-8.14.5Set up sendmail FEATURES (SSL, SASL, DB) # cat > devtools/Site/site.config.m4dnl ### Changes for STARTTLS supportAPPENDDEF(`confENVDEF',`-DSTARTTLS')APPENDDEF(`confLIBS', `-lssl -lcrypto')APPENDDEF(`confLIBDIRS', `-L/usr/lib -R/usr/li')APPENDDEF(`confINCDIRS', `-I/usr/include/openssl')dnl ### SASL supportAPPENDDEF(`confENVDEF', `-DSASL')APPENDDEF(`conf_sendmail_LIBS', `-lsasl')APPENDDEF(`confINCDIRS', `-I/usr/include/sasl')dnl ### Berkley DB supportAPPENDDEF(`confENVDEF', `-DNEWDB')APPENDDEF(`confINCDIRS', `-I/usr/include')APPENDDEF(`confLIBDIRS', `-L/usr/lib')EOF Build Sendmail # ./Build -c/* It should build but ignore the gtroff:error at the end */# sudo mv /usr/lib/sendmail /usr/lib/sendmail_orig# sudo cp obj.SunOS.5.11.i86pc/sendmail/sendmail /usr/lib/sendmail# sudo sudo chgrp smmsp /usr/lib/sendmail# sudo sudo chmod g+s,u-ws /usr/lib/sendmailNow a Vendor and SASL support to sendmail.cf file (I hope you know what you are doing here)Edit sendmil.mc add add the extra lines:define(`VENDOR_NAME', `Berkeley')define(`confAUTH_OPTIONS', `A')dnlTRUST_AUTH_MECH(`EXTERNAL DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnldefine(`confAUTH_MECHANISMS', `DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl# cd /etc/mail/cf/cf# cat sendmail.mcdivert(0)dnlVERSIONID(`sendmail.mc (Sun)')define(`VENDOR_NAME', `Berkeley')OSTYPE(`solaris11')dnlDOMAIN(`solaris-generic')dnldefine(`confAUTH_OPTIONS', `A')dnlTRUST_AUTH_MECH(`EXTERNAL DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnldefine(`confAUTH_MECHANISMS', `DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl define(`confFALLBACK_SMARTHOST', `mailhost$?m.$m$.')dnlMAILER(`local')dnlMAILER(`smtp')dnlLOCAL_NET_CONFIGR$* < @ $* .$m. > $* $#esmtp $@ $2.$m $: $1 < @ $2.$m. > $3## sudo make sendmail.cf# sudo cp sendmail.cf ../../sendmail.cf# sudo /usr/lib/sendmail -bt -d0 < /dev/nullVersion 8.14.5Compiled with: DNSMAP LOG MAP_REGEX MATCHGECOS MILTER MIME7TO8 MIME8TO7NAMED_BIND NDBM NETINET NETINET6 NETUNIX NEWDB PIPELININGSASLv2 SCANF STARTTLS USERDB XDEBUGNow lets set up SASL plugin via a Sendmail.conf# cat > /etc/sasl/Sendmail.conf# pw_check: shadow I have tried the following in the above with no luck: shadow,pam,auxprop, but none seem to check password information.Testing, but first we need the encrypted username and password:# perl -MMIME::Base64 -e 'print encode_base64("\000MyUser\000MyPassword")'AE15VXNlcgBNeVBhc3N3b3Jk Test the sendmail part:# sudo /usr/lib/sendmail -bv -O LogLevel=14 -bs -Am220 mailer5.dcs.bbk.ac.uk ESMTP Sendmail 8.14.5/8.14.5; Tue, 31 Jan 2012 09:52:24 GMTEHLO localhost250-mailer5.dcs.bbk.ac.uk Hello root@localhost, pleased to meet you250-ENHANCEDSTATUSCODES250-PIPELINING250-EXPN250-VERB250-8BITMIME250-SIZE250-DSN250-ETRN250-AUTH DIGEST-MD5 CRAM-MD5 PLAIN250-DELIVERBY250 HELPAUTH PLAIN AE15VXNlcgBNeVBhc3N3b3Jk535 5.7.0 authentication failedJan 31 09:52:35 mailer5 sendmail[5928]: Password verification failedJan 31 09:52:35 mailer5 sendmail[5928]: q0V9qOd1005928: AUTH failure (PLAIN): user not found (-20) SASL(-13): user not found: Password verification failed, relay=root@localhostquit221 2.0.0 mailer5.dcs.bbk.ac.uk closing connectionSo no luck, if you can spot anything I am doing wrong please let me know.I will post shortly how to get over the above. refernces:- Guía Instalación de Sendmail con SMTP-AUTH y (tema #2688)- Sendmail-SMTP-AUTH-TLS-Howto- OpenSolaris Sendmail Auth 2012-01-31T10:26:00Z Andrew Watkins noreply@blogger.com
ZFS+10: illumos meetup
Planet Illumos - Tue, 02/14/2012 - 22:24
ZFS recently celebrated its informal 10th anniversary; to mark the occasion, Delphix hosted a ZFS-themed meetup for the illumos community (sponsored generously by Joyent). Many thanks to Deirdre Straughan, the new illumos community manager, for helping to organize and for filming the event. Three of my colleagues at Delphix presented work they’ve been doing in [...]
MWAC in Global Zone
Planet Illumos - Tue, 02/14/2012 - 22:24
Solaris 11 has a new cool feature called Immutable Zones. Darren Moffat presented new features in Solaris 11 Zones at the last LOSUG meeting in London. Immutable Zones basically allow for read-only or partially read-only Zones to be deployed. You can even combine it with ZFS encryption - see Darren's blog entry for more details. The underlying technology to immutable zones is called Mandatory Write Access Control (MWAC) and is implemented in kernel. So for each open, unlink, etc. syscall a VFS layer checks if MWAC is enabled for a given filesystem and a zone and if it is it will check white and black lists associated with a zone and potentially deny write access to a file (generating EROFS). The actual definitions for different default profiles are located in /usr/lib/brand/solaris/config.xml file. It is *very* simple to use the pre-defined profiles when creating a zone and it just works. Really cool. Thanks Darren for the great demo.Now MWAC only works with non-global zones... at least by default. There is no public interface exposed to manipulate MWAC rules directly or to enable it for a global zone but it doesn't mean one can't try to do it anyway. DTrace, objdump, mdb, etc. were very helpful here to see what's going on. The result of having couple of hours of fun is below.root@global # touch /test/file1root@global # rm -f /test/file1root@global # ./mwac -b "/test/file1"MWAC black list for the global zone installed.root@global # touch /test/file1touch: cannot create /test/file1: Read-only file systemroot@global # touch /test/file2 ; rm /test/file2root@global # Now lets disable MWAC again:root@global # mwac -uMWAC unlock succeeded.root@global # touch /test/file1 ; rm /test/file1root@global # You can even use patterns:root@global # mwac -b "/test/*"MWAC black list for the global zone installed.root@global # touch /test/a ; mkdir /test/btouch: cannot create /test/a: Read-only file systemmkdir: Failed to make directory "/test/b"; Read-only file systemroot@global #
ReFS
Planet Illumos - Tue, 02/14/2012 - 22:24
Next generation file system for Windows: ReFSIt looks pretty interesting and promising. Something like ZFS lite for Windows. "The key goals of ReFS are:Maintain a high degree of compatibility with a subset of NTFS features that are widely adopted while deprecating others that provide limited value at the cost of system complexity and footprint.Verify and auto-correct data. Data can get corrupted due to a number of reasons and therefore must be verified and, when possible, corrected automatically. Metadata must not be written in place to avoid the possibility of “torn writes,” which we will talk about in more detail below.Optimize for extreme scale. Use scalable structures for everything. Don’t assume that disk-checking algorithms, in particular, can scale to the size of the entire file system.Never take the file system offline. Assume that in the event of corruptions, it is advantageous to isolate the fault while allowing access to the rest of the volume. This is done while salvaging the maximum amount of data possible, all done live.Provide a full end-to-end resiliency architecture when used in conjunction with the Storage Spaces feature, which was co-designed and built in conjunction with ReFS.The key features of ReFS are as follows (note that some of these features are provided in conjunction with Storage Spaces).Metadata integrity with checksumsIntegrity streams providing optional user data integrityAllocate on write transactional model for robust disk updates (also known as copy on write)Large volume, file and directory sizesStorage pooling and virtualization makes file system creation and management easyData striping for performance (bandwidth can be managed) and redundancy for fault toleranceDisk scrubbing for protection against latent disk errorsResiliency to corruptions with "salvage" for maximum volume availability in all casesShared storage pools across machines for additional failure tolerance and load balancing"
Sending Email with Attachments from the Command Line
Planet Illumos - Tue, 02/14/2012 - 22:24
I have lots of awesome CLI based reporting tools. One was so awesome that other people in the company wanted to get it on a regular basis but they preferred to see it as CSV so it could be manipulated in Numbers or Excel. Modifying my report to output CSV was easy, I just added [...]
On SOPA and PIPA
Planet Illumos - Tue, 02/14/2012 - 22:24
I can't say anything you haven't heard my tech friends say already on the subject. I can, however, quote this, because it's both funny and true:"I think we need to drive a stake into this thing's heart, fill its mouth with garlic, cut off its head, expose it to sunlight and then throw the ash into a running body of water. It is vital that people not let up on the pressure merely because they appear to compromise."Thank you Perry, for eloquently stating what should be SOPA's and PIPA's fates.
What is Watson?
Planet Illumos - Tue, 02/14/2012 - 22:24
You probably heard about Watson from IBM. Michael Perrone gave a very entertaining presentation on Watson at LISA. Enjoy.
Solaris 11 Source Code
Planet Illumos - Tue, 02/14/2012 - 22:24
It seems that Oracle has "published" S11 source code after all...And a little bit more on the story.
Flame on: Graphing hot paths through X server code
Planet Illumos - Tue, 02/14/2012 - 22:24
Last week, Brendan Gregg, who wrote the book on DTrace, published a new tool for visualizing hot sections of code by sampling the process stack every few milliseconds and then graphing which stacks were seen the most during the run: Flame Graphs. This looked neat, but my experience has been that I get the most understanding out of things like this by trying them out myself, so I did. Fortunately, it's very easy to setup, as Brendan provided the tools as two standalone perl scripts you can download from the Flame Graph github repo. Then the next step is deciding what you want to run it on and capturing the data from a run of that. It just so turns out that a few days before the Flame Graph release, another mail had crossed my inbox that gave me something to look at. Several years ago, I added some Userland Statically Defined Tracing probes to the Xserver code base, creating an Xserver provider for DTrace. These got integrated first to the Solaris Xservers (both Xsun & Xorg) and then merged upstream to the X.Org community release for Xserver 1.4, where they're available for all platforms with DTrace support. Earlier this year, Adam Jackson enabled the dtrace hooks in the Fedora Xorg server package, using the SystemTap static probe support for DTrace probe compatibility. He then noticed a performance drop, which isn't supposed to happen, as DTrace probes are simply noops when not being actively traced, and submitted a fix for it upstream. This was due to two of the probes I'd defined - when the Xserver processes a request from a client, there's a request-start probe just before the request is processed, and a request-done probe right after the request is processed. If you just want to see what requests a client is making you can trace either one, but if you want to measure the time taken to run a request, or determine if something is happening while a request is being processed, you need both of the probes. When they first got integrated, the code was simple: #ifdef XSERVER_DTRACE XSERVER_REQUEST_START(GetRequestName(MAJOROP), MAJOROP, ((xReq *)client->requestBuffer)->length, client->index, client->requestBuffer); #endif // skipping over input checking and auditing, to the main event, // the call to the request handling proc for the specified opcode: result = (* client->requestVector[MAJOROP])(client); #ifdef XSERVER_DTRACE XSERVER_REQUEST_DONE(GetRequestName(MAJOROP), MAJOROP, client->sequence, client->index, result); #endif The compiler sees XSERVER_REQUEST_START and XSERVER_REQUEST_DONE as simple function calls, so it does whatever work is necessary to set up their arguments and then calls them. Later, during the linking process, the actual call instructions are replaced with noops and the addresses recorded so that when a dtrace user enables the probe the call can be activated at that time. In these cases, that's not so bad, just a bunch of register access and memory loads of things that are going to be needed nearby. The one outlier is GetRequestName(MAJOROP) which looks like a function call, but was really just a macro that used the opcode as the index an array of strings and returned the string name for the opcode so that DTrace probes could see the request names, especially for extensions which don't have static opcode mappings. For that the compiler would just load a register with the address of the base of the array and then add the offset of the entry specified by MAJOROP in that array. All was well and good for a bit, until a later project came along during the Xorg 1.5 development cycle to unify all the different lists of protocol object names in the Xserver, as there were different ones in use by the DTrace probes, the security extensions, and the resource management system. That replaced the simple array lookup macro with a function call. While the function doesn't do a lot more work, it does enough to be noticed, and thus the performance hit was taken in the hot path of request dispatching. Adam's patch to fix this simply uses is-enabled probes to only make those function calls when the probes are actually enabled. x11perf testing showed the win on a Athlon 64 3200+ test system running Solaris 11: Before: 250000000 trep @ 0.0001 msec ( 8500000.0/sec): X protocol NoOperation After: 300000000 trep @ 0.0001 msec (10400000.0/sec): X protocol NoOperation But numbers are boring, and this gave an excuse to try out Flame Graphs. To capture the data, I took advantage of synchronization features built into xinit and dtrace -c: # xinit /usr/sbin/dtrace -x ustackframes=100 \ -n 'profile-997 /execname == "Xorg"/ { @[ustack()] = count(); }' \ -o out.user_stacks.noop -c "x11perf -noop" \ -- -config xorg.conf.dummy To explain this command, I'll start at the end. For xinit, everything after the double dash is a set of arguments to the Xserver it starts, in this case, Xorg is told to look in the normal config paths for xorg.conf.dummy, which it would find is this simple config file in /etc/X11/xorg.conf.dummy setting the driver to the Xvfb-like “dummy” driver, which just uses RAM as a frame buffer to take graphics driver considerations out of this test: Section "Device" Identifier "Card0" Driver "dummy" EndSection Since I'm using a modern Xorg version, that's all the configuration needed, all the unspecified sections are autoconfigured. xinit starts the Xserver, waits for the Xserver to signal that it's finished its start up, and then runs the first half of the command line as a client with the DISPLAY set to the new Xserver. In this case it runs the dtrace command, which sets up the probes based on the examples in the Flame Graphs README, and then runs the command specified as the -c argument, the x11perf benchmark tool. When x11perf exits, dtrace stops the probes, generates its report, and then exits itself, which in turn causes xinit to shut down the X server and exit. The resulting Flame Graphs are, in their native SVG interactive form: Before After If your browser can't handle those as inline SVG's, or they're scrunched too small to read, try Before (SVG) or Before (PNG), and After (SVG) or After (PNG). You can see in the first one a bar showing a little over 10% of the time was in stacks involving LookupMajorName, which is completely gone in the second patch. Those who saw Adam's patch series come across the xorg-devel list last week may also notice the presence of XaceHook calls, which Adam optimized in another patch. Unfortunately, while I did build with that patch as well, we don't get the benefits of it since the XC-Security extension is on by default, and those fill in the hooks, so it can't just bypass them as it does when the hooks are empty. I also took measurements of what Xorg did as gdm started up and a test user logged in, which produced the much larger flame graph you can see in SVG or PNG. As you can see the recursive calls in the font catalogue scanning functions make for some really tall flame graphs. You can also see that, to no one's surprise, xf86SlowBCopy is slow, and a large portion of the time is spent “blitting” bits from one place to another. Some potential areas for improvement stand out - like the 5.7% of time spent rescanning the font path because the Solaris gnome session startup scripts make xset fp calls to add the fonts for the current locale to the legacy font path for old clients that still use it, and another nearly 5% handling the ListFonts and ListFontsWithInfo calls, which dtracing with the request-start probes turned out to be the Java GUI for the Solaris Visual Panels gnome-panel applet. Now because of the way the data for these is gathered, from looking at them alone you can't tell if a wide bar is one really long call to a function (as it is for the main() function bar in all these) or millions of individual calls (as it was for the ProcNoOperation calls in the x11perf -noop trace), but it does give a quick and easy way to pick out which functions the program is spending most of its time in, as a first pass for figuring out where to dig deeper for potential performance improvements. Brendan has made these scripts easy to use to generate these graphs, so I encourage you to try them out as well on some sample runs to get familiar with them, so that when you really need them, you know what cases they're good for and how to capture the data and generate the graphs for yourself. Trying really is the best method of learning. 2011-12-20T03:34:39Z alanc
From SunOS thru Solaris and OpenSolaris to illumos
Planet Illumos - Tue, 02/14/2012 - 22:24
Last week I attended LISA '11 conference and one of the great and fun presentations was Bryan Cantrill's talk titled "Fork Yeah! The Rise and Development of illumos".See the video and the slides.Also, if you are interested in the whole DevOps transformation watch Ben Rockwood's talk - see the video and the slides.There were more interesting talks - you can find them at http://www.usenix.org/events/lisa11/tech/
LISA Keynote 2011: The DevOps Transformation
Planet Illumos - Tue, 02/14/2012 - 22:24
Last week I was given the incredible opportunity to not only speak at LISA but to deliver the opening keynote. I hadn’t expected to even go, but when I learned the topic was DevOps I made a last minute plea on the eve of the submission deadline for a slot to deliver a talk I [...]
The case of the un-unmountable tmpfs
Planet Illumos - Tue, 02/14/2012 - 22:24
Every once in a rare while our development machines encounter an fatal error during boot because we couldn’t unmount tmpfs. This weekend I cracked the case, so I thought I’d share my uses of boot-time DTrace, and the musty corners of the operating systems that I encountered along the way. First I should explain a [...]
A Tale of Two Soccer Websites (A Security Story)
Planet Illumos - Tue, 02/14/2012 - 22:24
(Pardon the latency on this post. I had it in the Drafts section for a while.)When a website requires a password for registration, said site SHOULD NOT EVER mail you back the password in the clear in an e-mail. Let me repeat that... SHOULD... NOT... EVER.One of my daughters plays soccer, and has for two towns. My whole family enjoy seeing the Boston Breakers play soccer too. Both my daughter's town website (outsourced to Blue Sombrero) and the Boston Breaker's ticketing website (run by PMI ticketing using TicketSocket's technology) made the aforementioned mistake. Both of them, quickly addressed the issue with direct and up-front e-mails. I believe Blue Sombrero addressed the problem a bit quicker, but that's because of a combination of smaller organizations and the Breakers' mistake happening on a weekend.The Blue Sombrero handling of my daughter's old town website mistake was quick, and without incident. Hats off (no pun intended) to the Blue Sombrero folks, who I hope have implemented the no-mailing-passwords policy throughout their entire customer base.One bad thing someone in the Breakers organization did was remove my original complaining posts on Facebook. I suspect this was merely the case of panic and not active malice. The General Manager of the Breakers, Andy Crossley, sent me a mail on Saturday to see what was going on. Once he understood the problem, he got the relevant technical folks involved, and they solved things.While I'm glad to see quick turnaround on these flaws, the one piece of advice I will reiterate is NEVER SEND OUT CLEARTEXT PASSWORDS. Thank you.
@SF 2011
Planet Illumos - Tue, 02/14/2012 - 22:24
Last year my visit to San Francisco was really fast, and i was really involved with the Solaris Express Program (Solaris 11). This year i should be at SF between October/4 and October/11 (my birthday is October/9, so good time to celebrate ;-). So i hope to have time to talk (with beers) with old [...] 2011-09-26T17:41:12Z Marcelo Leal
ZFS Internals (part #11)
Planet Illumos - Tue, 02/14/2012 - 22:24
PLEASE BE AWARE THAT ANY INFORMATION YOU MAY FIND HERE MAY BE INACCURATE, AND COULD INCLUDE TECHNICAL INACCURACIES, TYPOGRAPHICAL ERRORS, AND EVEN SPELLING ERRORS. From the MANUAL page: The zdb command is used by support engineers to diagnose failures and gather statistics. Since the ZFS file system is always consistent on disk and is self-repairing, [...] 2011-08-29T17:50:15Z Marcelo Leal
Nice colors for a tea and a book…
Planet Illumos - Tue, 02/14/2012 - 22:24
Do you remember? The good days of OpenSolaris… ;-) 2011-06-10T02:06:38Z Marcelo Leal
S11 X11: ye olde window system in today's new operating system
Planet Illumos - Tue, 02/14/2012 - 22:24
Today's the big release for Oracle Solaris 11, after 7 years of development. For me, the Solaris 11 release comes a little more than 11 years after I joined the X11 engineering team at what was then Sun, and finishes off some projects that were started all the way back then. For instance, when I joined the X team, Sun was finishing off the removal of the old OpenWindows desktop, and we kept getting questions asking about the rest of the stuff being shipped in /usr/openwin, the directory that held both the OpenLook applications and the X Window System software. I wrote up an ARC case at the time to move the X software to /usr/X11, but there were various issues and higher priority work, so we didn't end up starting that move until near the end of the Solaris 10 development cycle several years later. Solaris 10 thus had a mix of the recently added Xorg server and related code delivered in /usr/X11, while most of the existing bits from Sun's proprietary fork of X11R6 were still in /usr/openwin. During Solaris 11 development, we finished that move, and then jumped again, moving the programs directly into /usr/bin, following the general Solaris 11 strategy of using /usr/bin for most of the programs shipped with the OS, and using other directories, such as /usr/gnu/bin, /usr/xpg4/bin, /usr/sunos/bin, and /usr/ucb for conflicting alternate implementations of the programs shipped in /usr/bin, no longer as a way to segregate out various subsystems to allow the OS to better fit onto the 105Mb hard disks that shipped with Sun workstations back when /usr/openwin was created. However, if for some reason you wanted to build your own set of X binaries, you could put them in /usr/X11R7 (as I do for testing builds of the upstream git master repos), and then put that first in your $PATH, so nothing is really lost here. The other major project that was started during Solaris 10 development and finished for Solaris 11 was replacing that old proprietary fork of X11R6, including the Xsun server, with the modernized, modularized, open source X11R7.* code base from the new X.Org, including the Xorg server. The final result, included in this Solaris 11 release, is based mostly on the X11R7.6 release, including recent additions such as the XCB API I blogged about last year, though we did include newer versions of modules that had upstream releases since the X11R7.6 katamari, such as Xorg server version 1.10.3. That said, we do still apply some local patches, configuration options, and other changes, for things from just fitting into the Solaris man page style or adding support for Trusted Extensions labeled desktops. You can see all of those changes in our source repository, which is searchable and browsable via OpenGrok on src.opensolaris.org (or via hgweb on community mirrors such as openindiana.org) and available for anonymous hg cloning as well. That xnv-clone tree is now frozen, a permanent snapshot of the Solaris 11 sources, while we've created a new x-s11-update-clone tree for the Solaris 11 update releases now being developed to follow on from here. Naturally, when your OS has 7 years between major release cycles, the hardware environment you run on greatly changes in the meantime as well, and as the layer that handles the graphics hardware, there have been changes due to that. Most of the SPARC graphics devices that were supported in Solaris 10 aren't any more, because the platforms they ran in are no longer supported - we still ship a couple SPARC drivers that are supported, the efb driver for the Sun XVR-50, XVR-100, and XVR-300 cards based on the ATI Radeon chipsets, and the astfb driver for the AST2100 remote Keyboard/Video/Mouse/Storage (rKVMS) chipset in the server ILOM devices. On the x86 side, the EOL of 32-bit platforms let us clear out a lot of the older x86 video device drivers for chipsets and cards you wouldn't find in x64 systems - of course, there's still many supported there, due to the wider variety of graphics hardware found in the x64 world, and even some recent updates, such as the addition of Kernel Mode Setting (KMS) support for Intel graphics up through the Sandy Bridge generation. For those who followed the development as it happened, either via watching our open source code releases or using one of the many development builds and interim releases such as the various Solaris Express trains, much of this is old news to you. For those who didn't, or who want a refresher on the details, you can see last year's summary in my X11 changes in the 2010.11 release blog post. Once again, the detailed change logs for the X11 packages are available, though unfortunately, all the links in them to the bug reports are now broken, so browsing the hg history log is probably more informative. Since that update, which covered up to the build 151 released as 2010.11, we've continued development and polishing to get this Solaris 11 release finished up. We added a couple more components, including the previously mentioned xcb libraries, the FreeGLUT library, and the Xdmx Distributed Multihead X server. We cleaned up documentation, including the addition of some docs for the Xserver DTrace provider in /usr/share/doc/Xserver/. The packaging was improved, clearing up errors and optimizing the builds to reduce unnecessary updates. A few old and rarely used components were dropped, including the rstart program for starting up X clients remotely (ssh X forwarding replaces this in a more secure fashion) and the xrx plugin for embedding X applications in a web browser page (which hasn't been kept up to date with the rapidly evolving browser environment). Because Solaris 11 only supports 64-bit systems, and most of the upstream X code was already 64-bit clean, the X servers and most of the X applications are now shipped as 64-bit builds, though the libraries of course are delivered in both 32-bit and 64-bit versions for binary compatibility with applications of each flavor. The Solaris auditing system can now record each attempt by a client to connect to the Xorg server and whether or not it succeeded, for sites which need that level of detail. In total, we recorded 1512 change request id's during Solaris 11 development, from the time we forked the “Nevada” gate from the Solaris 10 release until the final code freeze for todays release - some were one line bug fixes, some were man page updates, some were minor RFE's and some were major projects, but in the end, the result is both very different (and hopefully much better) than what we started with, and yet, still contains the core X11 code base with 24 years of backwards compatibility in the core protocols and APIs. 2011-11-09T16:57:07Z alanc
Using Graphite to Graph DTrace Metrics: Part II
Planet Illumos - Tue, 02/14/2012 - 22:24
In a previous entry I described Graphite and gave an overly simplistic example of integrating it with DTrace… lets get a little more serious and see what fun we can have. For a years a problem nagged at me. I wanted to get really fine grained latency information from an NFS server to track user [...]