Justin's IT and Security Pages

Here is a current shot of the Flower console:

I’ve made an EC2 AMI of the Analysis Server available on Amazon’s AWS. If you’d like to try the system out and already have EC2 configured, follow these instructions:

1. Configure a security group that allows ports 22/tcp, 8080/tcp, and 9995/udp
2. Create an EC2 instance of ami-3857bf51; the command I use is (my security group is called “appserver”):

ec2-run-instances -g appserver ami-3857bf51

3. Once loaded, log in via SSH and start the Flower server (all passwords are set to “flower” by default):

asadmin start-domain flower

4. Use http://app.jdthomas.net to log in to your server; in the server field, enter “http://[EC2 IP Address]:8080″ and both the username and password are “flower” – just hit [Enter] on any field to submit the form (there is no button)

Of course, app.jdthomas.net will need access to your EC2 system on port 8080/tcp (it uses JAX-WS to access and present your data to you), so if you lock things down be sure to account for that. HTTPS works, but you’ll need to configure and use a valid certificate on your EC2 system.

You can now configure your NetFlow exporters to send to the IP address of your EC2 instance on UDP port 9995 (version 9 only). I use softflowd on a couple of Debian systems with this command:

softflowd -i eth0 -n [EC2 IP Address]:9995 -p /var/run/sfd.pid.ec2 -c /var/run/sfd.ctl.ec2 -v 9

Configure your server using the “Manage” menu in the web application. Be sure to add networks (use CIDR addressing like 192.168.1.0/24) and users. Specified networks are required for the map, but area charts and flow tables will work without any networks specified.

E-mail me at justin@jdthomas.net with any questions! There are lots and lots of bugs and missing features, but I’ll get to them over the coming months.

UPDATE: I made a few changes tonight to improve the Visualization server configuration; the URL is now http://app.jdthomas.net (this will redirect you to an SSL site of the same base but with a longer path). I’ve made that change in the text above as well.

Here’s a recent shot of the Flower console.

In addition to the updated interface (I finally accepted AJAX into my life and it opened a lot of possibilities), there are a ton of changes under the hood. I’ve deprecated the Flower Capture Engine in favor of collecting NetFlow (v9) data from standard generators (Cisco routers, softflowd, etc.) Storage has been migrated from a relational database to a BerkeleyDB backend; this move greatly simplified the code and increased performance significantly.

I’ll be looking for a few beta testers who are willing to fling some NetFlow data across the Internet; contact me at justin@jdthomas.net if you’re interested. In the future, you’ll be able to deploy a collector on site that will encrypt the data before delivering it to the analysis server, but that part isn’t quite ready for public consumption yet.

Things are moving on this item. I consider the code for this “alpha” quality at best. If you’re brave and want to give it a try, keep reading:

I still haven’t created a VM image for this yet, but you can download the Java web application (I use Glassfish as the container) and UNIX/Linux (I use Debian) sensor sources from this web site:

https://sourceforge.net/projects/flower-nfa/

All code is available in the Subversion repository.

The current status is that the network map and tabular chart functions work well along with the “constraint” (i.e., specifying traffic patterns to view) mechanism. Both IPv4 and IPv6 traffic can be profiled and queried.

On the client end, Firefox 3 and Safari 4 both work quite well. Safari (really, Webkit) has a minor issue with the title graphic – it doesn’t scale as the window grows and shrinks. That doesn’t impact the functionality at all.

Firefox 3.5 has introduced some pretty significant bugs in SVG rendering that make the scale functionality of the map work poorly; I’ve encountered numerous browser hangs and crashes when I scale the image past about 2.0x and try to move it. It looks to me like maybe they’ve done some work with the text rendering that has introduced this new instability. Hopefully that will be resolved quickly. I’ll try to figure a concise way to file a bug report about it.

Chrome has the same issues as Safari. Chrome also renders the SVG text differently than Safari or Firefox; not a big deal but it does impact readability a little bit.

Opera works, but is “tweaky”. Hard to really describe exactly how.

Internet Explorer does not work at all, of course (no SVG capability). Adobe’s plug-in might work, but I’ve never tried it.

The “flower_config.xml” file in the web application root directory contains the specifications for the networks to profile and some protocol information to normalize non-TCP flows. That must be modified to match your environment.

User name and password for the application are hard-coded as “flower/flower”. I’ll add a real authentication mechanism as things progress towards a beta release.

I use a URL base of “/flower”, but theoretically you should be able to choose whatever you want. (Strike that, the sensor is hard-coded to use “http://. . .:8080/flower’ – I will fix that, though).

Available constraints are: “tcp, udp, esp, ipv4, ipv6, sip , dip , port , protocol ” Additional constraints are easy to add as I find the need.

The analysis engine runs on Debian Linux (and probably most other Linux distributions). It requires libcurl and libpcap. Compilation is a simple: “./autogen.sh;./configure;make” The resulting executable is src/sensor and is executed with the following options:

-b Executes the sensor in the background
-s <address> Analysis server IP address to report to
-d <device> Ethernet device to listen on (i.e., eth0)

If not run as background, you will be presented with a prompt: “FCE>”. Here you can query the running analysis engine. Just type “help” to see the available commands.

This is a topic that I have struggled with for several months. Until today, I had only been able to convince my OpenSolaris 2009.06 system to share files via SMB as a workgroup; every attempt to join my Windows 2008 Active Directory domain was greeted with a “no domain controllers found” error – even when Kerberos was working flawlessly and LDAP queries were returning everything as they should.

After reading many forums and conversations about what might be causing this error, I finally decided that it must be an idiosyncrasy with OpenSolaris attempting to operate in a multiple-DC environment. I bit the bullet and demoted 2 of my 3 test DCs.

To my chagrin, the OSOL box still refused to join the domain with a “no domain controllers found” error.

I performed a snoop of the traffic and realized that the error message was not entirely truthful. The DC was being found, but the authentication was failing. This differed from my previous attempts which were thwarted when pulling up the “_ldap._tcp.dc._msdcs.domain.name” DNS record and receiving multiple servers in the reply.

I had read on multiple forums that the lmauth level might cause problems with Windows 2008 domains. This command solved that problem:

sharectl set -p lmauth_level=2 smb

I was elated that I could authenticate from my windows machine transparently . . . until I rebooted. Upon reboot, the SMB server service failed. Period. No interesting logs. It just wouldn’t start.

So, for about 30 minutes I gave up. I had almost decided that I’d just run it in workgroup mode and be happy, but I thought I’d do one more Google search. What I discovered was that the SMB server problems were far more common than the other problems I had been running into – in fact, it appeared that snv_114 included some code changes to solve a number of SMB problems. The difficulty was that OpenSolaris was (and is) stuck on snv_111b.

Another alternative exists. Solaris Express Community Edition (SXCE) is currently at snv_115. That distribution does not have some of the niceties of OpenSolaris (package manager, etc.), but appears to be a bit more advanced. I decided to give it a shot.

As it turns out, SXCE rocks. Everything works (almost) right out of the box. I could even install it from the CLI – no messing around with X or VNC just to get the OS installed. It even gives me the option to install a “Core” system, without all of the nonsense I don’t really need. This is what Solaris used to be, and what it should be again.

One problem that came up when I rebooted turned out to be due to unresolvable SID mappings. The symptom was that, again, my domain users were unable to authenticate – log messages in /var/adm/messages reflected the failures. This command solved that issue:


svccfg -s idmap setprop config/unresolvable_sid_mapping = boolean: true
svcadm refresh idmap


I’ll keep an eye on OpenSolaris, but for now I think SXCE will do nicely. I’ll write up a quick blog post on how to get SXCE to play nicely on XenServer in a bit; the process is similar to OpenSolaris, but the CLI install options make things a bit easier.

So to recap: If you want SMB domain authentication with Windows 2008, use SXCE with a single AD domain controller and be sure to execute the sharectl and idmap commands above. Beyond those caveats, all of the instructions peppering OSOL blog posts out there will do fine to help you get things up and running.

Also remember, snoop is your friend.

I recently went through the motions to install Nexenta CP 2 RC3 on XenServer 5, following the directions that I had previously posted for RC1. I’m happy to confirm that the process for RC1 seems to work fine for RC3.

As an aside, a poster to that original article (Bill McGonigle) included some instructions for using lofiadm to alter the miniroot archive from the Nexenta installation DVD image to use with Xen and XenServer (I used a separate OpenSolaris system for that step). Take a look at the comments attached to the previous article for details.

I’ve been working on building out some critical parts of an application that has been a work in progress for some time. Here’s a screen shot of the latest updates showing the traffic on my messy home network.

More to come on this soon!

I put together a XenServer template for Nexenta. If you’re interested, please give it a try and post your results in the comments:

NexentaCP 2 RC1 Template

Here are the instructions from the README.txt file in that package:

Steps to install this package:

1. Copy the “nexentacp2_rc1.tar.gz” kernel/ramdisk archive to the root directory on the XenServer using scp (or, since you’ll likely be doing this from Windows, you can just use WinSCP):

scp ./nexentacp2_rc1.tar.gz root@xenserver:/

2. At the console of the XenServer (you can use XenCenter or just connect via SSH), decompress the kernel/ramdisk archive (be sure to use the “P” option to maintain the directory structure):

[root@xen /]# tar xvPf ./nexentacp2_rc1.tar.gz

The kernel should now be in /opt/xen/kernels and the ramdisk should be in /opt/xen/ramdisks on your XenServer. There will also be a failsafe ramdisk in the ramdisks folder should you ever need it.

3. Import the XVA into XenCenter. The image size is about 5GB; you’ll need an appropriate amount of space available to execute the import.

Thanks to rootard for posting a mirror of the rather large (~1GB) ZIP file.

UPDATED: Created a slimmer template with a helper script included (see README.txt). Thanks to eXeC001er for helping me to test and validate the original package!

Just a quick follow-up to my previous post on problems with snv_110 and Xen. snv_111 showed up in IPS recently and I immediately updated one of my OpenSolaris test systems from snv_109 to the new code. The problem with hald/Xen in snv_110 looks to have been solved and everything seems to be working as intended!

Although Nexenta uses the OpenSolaris kernel, there are a few unique steps that you’ll need to take in order to get Nexenta up and running paravirtualized (PV) on XenServer 5. These steps are a result of the lack of certain files in the default ramdisk on the Nexenta installation CD.

Also, the CDROM device seems to be handled differently in Nexenta than it is in OpenSolaris. As a result, these instructions will require you to fully install Nexenta in HVM mode and then flip the right bits to convert it to PV.

A few notes: I use /opt/kernels on my XenServer to store the kernels and ramdisks I use for my PV systems. You can substitute whatever you like for that directory; technically I think the “correct” place would be somewhere in “/usr/local”. Likewise, the names that I give the ramdisks and kernels are completely subjective; feel free to devise your own scheme.

1. Using the “Other Media” option in XenCenter, install NexentaCP from the installation CD like you’d install a Windows system.
2. Configure the installation appropriately at the presented prompts. Note that due to running in HVM mode, this initial installation will take a significant amount of time. Fear not, things will move much more quickly by the time you’re done.
3. Copy /platform/i86pc/miniroot from the Nexenta installation CD to a system where you can work with it. I used an OpenSolaris system for the next steps.
4. Rename miniroot to miniroot.gz.
5. Decompress the miniroot.gz archive: gunzip miniroot.gz
6. Mount the miniroot archive as a loopback device:
   mount -o loop [/path/to/]miniroot /[mountpoint]
7. Change directories into the /platform directory on the mounted filesystem
8. Copy the entire /platform/i86xpv directory from the HVM Nexenta system via scp over to the mounted filesystem. You might need to do this from the HVM Nexenta system (unless you svcadm enable ssh on the HVM Nexenta system).
9. Move out of the miniroot filesystem on your working machine (the OpenSolaris box for me) and unmount the archive:
   unmount /[mountpoint]
10. Recompress the miniroot archive: gzip miniroot
11. Copy the newly modified miniroot archive over to an appropriate directory on the XenServer:
    scp ./miniroot.gz
root@xenserver:/opt/kernels/ramdisk_nexenta_install
12. Log in to the Nexenta HVM system and copy the i86xpv kernel over to the XenServer:
    scp /platform/i86xpv/kernel/unix
root@xenserver:/opt/kernels/kernel_nexenta
13. Log in to the XenServer and configure the Nexenta VM to use the kernel and ramdisk you’ve moved over:
    1. xe vm-param-set uuid=[VM UUID]
PV-kernel=/opt/kernels/kernel_nexenta
    2. xe vm-param-set uuid=[VM UUID] PV-ramdisk=
/opt/kernels/ramdisk_nexenta_install
    3. xe vm-param-set uuid=[VM UUID]
PV-args='/platform/i86xpv/kernel/unix
-B console=ttya -m milestone=0 -v'
    4. xe vm-param-set uuid=[VM UUID]
HVM-boot-policy=
14. Reboot the newly paravirtualized Nexenta VM. It will fail to boot – that’s okay.
15. When prompted to log in for maintenance, enter root with a blank password. If you have trouble getting the VM to respond to your keystrokes in XenCenter, try restarting XenCenter; every time I’ve done this, XenCenter has failed at this point and had to be restarted.
16. Import the syspool zpool: zpool import -f syspool
17. Configure the PV-args on the XenServer to specify the bootfs:
    xe vm-param-set uuid=[VM UUID] PV-args=
'/platform/i86xpv/kernel/unix -B console=ttya,
zfs-bootfs=syspool/rootfs-nmu-000,
bootpath="/xpvd/xdf@51712:a"'
18. Reboot your Nexenta system. At this point, it should boot up and let you log in normally.
19. Once booted, plumb your virtualized network interfaces: ifconfig xnf0 plumb
20. Rename /etc/hostname.rtls0 and /etc/hostname6.rtls0 to /etc/hostname.xnf0 and /etc/hostname6.xnf0
21. Reboot.
22. Copy /platform/i86pc/boot_archive from the booted Nexenta system over to the XenServer as something like /opt/kernels/ramdisk_nexenta.
23. Reconfigure the VM parameters to point at the new boot archive:
    xe vm-param-set uuid=[VM UUID]
PV-ramdisk=/opt/kernels/ramdisk_nexenta
24. Reboot.

Congratulations! You now have a paravirtualized Nexenta core system. Upon booting, the VM screen should look something like:

v3.2.1 chgset '58bf50a2c754.3c18e9e0f827 (3.2.1 5.0.0.235.17085)'
SunOS Release 5.11 Version NexentaOS_20081207 32-bit
Loading Nexenta...
NOTICE: xdf@51712: failed to read feature-barrier
Hostname: nexenta-test
Reading ZFS config: done.
Mounting ZFS filesystems: (2/2)

NexentaCore 2.0 RC1 (Hardy 8.04/b104+)

nexenta-test console login:

Take particular note: when you update a package that includes a kernel module, be sure to update the boot_archive (bootadm update-archive) and copy that updated archive over to the XenServer as /opt/kernels/ramdisk_nexenta BEFORE rebooting the Nexenta VM. I ran into a problem where the system could not load the console after doing an aptitude safe-upgrade and rebooting without updating the ramdisk.

Also, after installing I had to run an apt-get -f install to finish the installation of libtimedate-perl before doing an aptitude safe-upgrade. That package doesn’t appear to be installed correctly by the installer.

As always, please let me know if you have any comments or suggestions!

After my post last night, I did a little more digging; all that was required was to import the syspool from the HVM install (zpool import -f syspool) and reset the PV-args.

It looks like I need to re-configure the interfaces (the rtls0 interface would not plumb, presumably because the PV process changed the name), but I think we’re there. I’ll clean up the process and post comprehensive instructions later tonight.