Justin's IT and Security Pages

Archive for the ‘programming’ Category

CSS Tables with Fixed Headers

leave a comment

I’ve been using a table in one of my web applications for a while that’s bothered me. The table displays NetFlow data and may contain many thousands of lines with fields of varying lengths (e.g., shorter fields for IPv4 addresses and longer fields for IPv6 addresses). To allow for maximum flexibility, I’ve defined the table using percentages so that it can be expanded to full screen or to occupy a smaller window.

The aspect that bothered me the most was the header. To get the labels to line up correctly, the header needed to be part of the larger table to accommodate the flexible field widths; if I wanted the labels to line up, it could not be defined as an outer table with an inner div that allowed for scrolling as many sites suggest. This meant that the header would scroll off the page when the data was accessed. However, I finally figured out a way to overcome that difficulty.

I use a lot of JavaScript in this application, so relying on that to resize fields doesn’t give me much heartache. I figured out that I can query the inner elements for their offsetWidth after they’ve been added to the table and then dynamically style the header field widths to match. Below is my example.

Given this structure:

<table class="flows">

  <thead id="flow-header">
    <tr>
      <th id="start">Start Time</th>
      <th id="protocol">Type</th>
      <th id="source">Source Address</th>
      <th id="sport">Port</th>
      <th id="destination">Destination Address</th>
      <th id="dport">Port</th>
      <th id="flags">Flags</th>
      <th id="size">Size</th>
    </tr>
  </thead>

  <tfoot id="flow-footer">
    <tr>
      <td>Start Time</td>
      <td>Type</td>
      <td>Source Address</td>
      <td>Port</td>
      <td>Destination Address</td>
      <td>Port</td>
      <td>Flags</td>
      <td>Size</td>
    </tr>
  </tfoot>

  <tbody>
    <tr>
      <td colspan="8">
        <div id="flow-table">
          <table id="flow-data"></table>
        </div>
      </td>
    </tr>
  </tbody>

</table>

…I can use this JavaScript snippet to resize the th fields (and by extension the footer td fields) to match the data that is pulled in via AJAX to the “flow-data” table:

$('start').style.width = $('flow-data').childNodes[0].childNodes[0].offsetWidth + "px";
$('protocol').style.width = $('flow-data').childNodes[0].childNodes[1].offsetWidth + "px";
$('source').style.width = $('flow-data').childNodes[0].childNodes[2].offsetWidth + "px";
$('sport').style.width = $('flow-data').childNodes[0].childNodes[3].offsetWidth + "px";
$('destination').style.width = $('flow-data').childNodes[0].childNodes[4].offsetWidth + "px";
$('dport').style.width = $('flow-data').childNodes[0].childNodes[5].offsetWidth + "px";
$('flags').style.width = $('flow-data').childNodes[0].childNodes[6].offsetWidth + "px";
$('size').style.width = $('flow-data').childNodes[0].childNodes[7].offsetWidth + "px";

Obviously, I use prototype.js to make my life a little easier, but that’s the extent of the JavaScript frameworks that I employ.

The result is a table wherein the header fields are dynamically resized to match the content that is pulled in to the DOM (gradually – as the scrollbar moves down the table).

*I’ll probably get rid of all those ID tags, since I should be able to refer to them relative from their parent elements.

Written by justin

August 1st, 2011 at 9:34 pm

Posted in programming

Tagged with

SVG Matrix Transformations and JavaScript

4 comments

No matter how much you might resist using matrix transformations with SVG documents, if you intend to modify an image dynamically (and cumulatively), matrixes are your only viable option.

Many sites tell you that you should use matrixes, ostensibly for speed purposes. In my opinion, speed is not the issue. The issue is the complexity associated with applying multiple transformations to an element; you just can’t do it with simple transformations (e.g., rotate, translate, skew and/or scale).

Here are a few notes about the stumbling blocks that I encountered in my journey towards using matrix transformations:

  1. Most online guides seem to assume that you will be working with a static image; they tell you how to convert simple transformations to matrixes as a one-time operation, but give you no (straightforward) information on how to subsequently alter those transformations dynamically. The JavaScript method, element.getCTM() is your key to handling this situation. By using this method (short for get Current Transformation Matrix), you can obtain a matrix that includes all of the transformations currently applied to your graphic element. That matrix can then be used to generate updated matrixes after applying dynamically updated transformations.
  2. The sylvester.js JavaScript library is a great resource to handle your matrix math needs. You’ll use the method matrix.x() to multiply the current matrix with the matrix representing the transforms you want to apply to obtain your newly combined matrix to apply to your DOM element.
  3. Some guides talk about changing the matrix.e and matrix.f variables directly to apply translation transformations. When dealing with multiple transforms, that will just cause you a world of grief (e.g., rotation transforms update the e and f variables in complex ways that are difficult to calculate without matrix math).
  4. Rotation transformations use sine and cosine methods extensively. At first glance, one would think that the JavaScript methods Math.sin() and Math.cosin() would work nicely. They do, but keep in mind that those methods deal in radians, not degrees. If you want to rotate something in (e.g., 45) degrees, you’ll need to convert that value to radians (e.g., 0.785398163) before using those methods.
  5. You should be able to combine a translation transformation with a rotation transformation in a single matrix to choose the center of rotation, but I haven’t been able to get that to work. Instead, I perform a pre-shift to move my desired rotation point to the origin and then a post-shift to move it back after the rotation. That seems to work reliably and allows me to rotate the graphic where my pointer is hovering.

Here is an example of how I have implemented these concepts in my Flower Network Flow Analysis Visualization server (I also have some prototype.js markup in here and this.nonce refers to a random, one-time string I use to distinguish between multiple, similar, generated SVGs existing in a single DOM):

var content = $('content_' + this.nonce);
var matrix = content.getCTM();

var map = $('map');

var leftVal = map.offsetLeft;
var topVal = map.offsetTop;
var parent = map.offsetParent;

while(parent != null) {
	leftVal += parent.offsetLeft;
	topVal += parent.offsetTop;
	parent = parent.offsetParent;
}

var pointerX = event.clientX - leftVal;
var pointerY = event.clientY - topVal;

var radians = rotation * (Math.PI/180);
var cos = Math.cos(radians);
var sin = Math.sin(radians);

var current = $M([[ matrix.a, matrix.c, matrix.e ], [ matrix.b, matrix.d, matrix.f ], [0, 0, 1]]);
var preshift = $M([[ 1, 0, -pointerX], [0, 1, -pointerY], [0, 0, 1]]);
var rotated = $M([[cos, -sin, 0], [sin, cos, 0], [0, 0, 1]]);
var postshift = $M([[ 1, 0, pointerX], [0, 1, pointerY], [0, 0, 1]]);

var updated = postshift.x(rotated.x(preshift.x(current)));

content.setAttribute("transform", "matrix(" +
	updated.e(1, 1) + " " + updated.e(2, 1) + " " +
	updated.e(1, 2) + " " + updated.e(2, 2) + " " +
	updated.e(1, 3) + " " + updated.e(2, 3) + ")");

The basic operations above are:

  1. Obtain the current transformation matrix. (lines 1 and 2)
  2. Determine how far the pointer is from the edges of the map DIV using offsetLeft and offsetTop. (lines 4-17)
  3. Pre-calculate the sine and cosine values of the desired rotation (convert to radians as an intermediate step). (lines 19-21)
  4. Convert the SVG matrix to a sylvester.js matrix ($M). (line 23)
  5. Build the pre-shift transformation matrix. Translation matrixes are constructed thusly, with X and Y being the number of pixels the graphic should be shifted in the X and Y directions, respectively (line 24):
      [ 1, 0, X ]
      [ 0, 1, Y ]
      [ 0, 0, 1 ]
  6. Build the rotation transformation matrix. Rotation matrixes are constructed thusly, with R being the rotation value (line 25):
      [  cos(R), sin(R),   0 ]
      [ -sin(R), cos(R),   0 ]
      [       0,      0,   1 ]
  7. Build the post-shift transformation matrix. (line 26)
  8. Calculate the updated matrix; note that the order of operations is important. I’m not precisely sure about the rules that apply here (I kind of guessed until I got the order right, to be perfectly honest). The really important thing to remember is that matrix multiplication is apparently sensitive to order (unlike simple multiplication). (line 28)
  9. Apply the updated matrix to your graphic element. Note that in my example above I’m using the sylvester.js (row, column) notation. The actual matrix() transform only uses the first six values of the full matrix – the last row of 0, 0, 1 never changes and should not be specified. (line 30-33)

By way of example, and if you have a newer version of Firefox, Chrome, Safari, Epiphany (or shockingly, even IE9!), visit a mock-up of a generated network map here. For anyone else, here is a still screenshot of a map that’s been twisted, translated and scaled.

 

Network Map Mockup

Network Map Mockup

 

If you do visit that page, try using your mouse wheel to rotate the map or the graphical slider to zoom in and out. You can also just drag the map around to reposition it. Clicking on a connection will open a small detail box and clicking on a node will narrow the display to only connections involving that node. Clicking on subsequent nodes will add those nodes’ connections. Double clicking in the white space will cause all connections to be visible again. Hovering over a connection or node will show you that connection or node’s details (at the bottom of the graphic). The information in there is mostly nonsense – I went through and “sanitized” the addresses – although protocol, port and volume information are real.

I hope the above information helps someone else! I know it would have saved me a lot of time to have a working example of JavaScript code that updates a transformation matrix dynamically based on DOM events.

Written by justin

February 19th, 2011 at 9:15 pm

Posted in programming

Snort IDS Events and Flower

leave a comment

Over the last couple of weekends I’ve added the ability to capture and report on alerts generated by Snort IDS sensors. The additional code consists of: modifications to the Analysis Server to store and retrieve IDS related data, modifications to the Visualization Server to present that IDS data in a meaningful manner, and a new Python-based module that is installed on the IDS sensor itself.

That last component opens a local UNIX domain socket on the sensor. Snort is then configured to log alerts to that socket. When data is received on the open socket, the Python code parses the alert and opens a web services connection to the Analysis Server to deliver the results.

This is what the interface currently looks like with those modifications:

IDS Alert Console

IDS Alert Console

I only recently deployed the Snort sensor and it has not been tuned yet; that’s why we see so many “(portscan)” alerts in the console above. I’m viewing the last 24 hours worth of alerts in this shot and have zeroed in on one of the less frequent alerts. All of the alerts shown are false positives, but their generation is good for my testing.

I haven’t added the Python code to GitHub, but will be doing so shortly. I also plan to post another entry on this blog about configuring XenServer with Open vSwitch to better accommodate IDS functionality. I recently installed that software and configured a mirror port to monitor all local network activity on my VM server for the Flower Analysis server. The difference (from just using “promiscuous mode” on an interface) in visibility and efficiency is striking.

That vSwitch software comes with a bit of a learning curve, but the functionality it enables is very cool.

Written by justin

November 20th, 2010 at 11:01 pm

Flower NFA Update

leave a comment

I’ve made some significant updates both to the Flower Visualization Server and to the supporting Analysis Server:

  • Visualization Server
    1. I implemented changes necessary to support the additional resolutions noted in the Analysis Server section. This is transparent to the end-user (the proper resolution is automatically selected).
    2. New dialog boxes were added to support AD configuration.
      Active Directory Configuration

      Active Directory Configuration

    3. Many, minor aesthetic changes were made to the Network Map output.
      Network Map

      Network Map

    4. Fonts (@font-face) are used more consistently throughout the workings of the console.
    5. The most significant recent changes to the Visualization Server are in the area chart output:
      Volume Chart

      Volume Chart


      I changed the chart from arithmetic to logarithmic because I found that my UDP data was being pushed down to 1 pixel as it was tremendously outweighed by the TCP data. Horizontal scale lines were added to illustrate the fact that the bottom of the graph represents an amount of data many orders of magnitude less than the top of the graph. I also wrote code to insert vertical lines at the beginning of each hour, day and month to better make sense of the data. Here is a 30-day graph shown within it’s containing window (data older than a few days is not pictured because I had to reset the database when I implemented the resolution concept):
      Volume Window

      30 Days

  • Analysis Server
    1. In line with the changes I noted in my last update to how statistical flow data is stored, I’ve introduced multiple levels of summarization to greatly decrease the time it takes to complete queries for longer time durations. The levels are configurable, but presently data is stored in 10 second, 1000 second, and 10000 second resolutions.

      To put that in to perspective, at a resolution of 10 seconds, a query for 30 days worth of data would need to scroll through 259,200 records. That same query would need to traverse only 2,592 records at a 1,000 second resolution and only 260 records at a 10,000 second resolution.

    2. I updated the chart data generation code to take advantage of the new resolution levels. Network maps represent data within a time period with little regard for how that data is distributed, so we can use a very low resolution and access the data very quickly.

      Area charts are more sensitive to how data is distributed throughout a time period and I’ve updated the code to dynamically select a resolution based on the width (in pixels) of the ultimate output and the time period selected. For example, a chart that is 600 pixels wide wouldn’t be well supported by a 30 day query split into 260 intervals. The Analysis server would select a 1,000 second resolution (2,592 intervals) to provide an adequate level of granularity with an optimal query structure.

    3. Microsoft Active Directory (AD) may now be used as an authentication and authorization source with minimal configuration. I’ve implemented code that interfaces with AD servers using LDAP/S (or optionally, and highly discouraged, over LDAP). AD servers are identified automatically by leveraging SRV records provided by the DNS server used by the Analysis Server.

      To enable this capability, an administrator need only specify the fully-qualified domain name of the forest root (e.g., “internal.company.com”) and the group(s) to authenticate against. Each group can be specified as “privileged” to permit management of the Flower systems themselves.

      No AD user data is stored within the Flower systems and all communication occurs over SSL (unencrypted authentication may be selected, but that option is not exposed yet and would be highly dangerous if not handled properly). Importing the SSL certificate from AD into the GlassFish server is a little tricky, and I’ll write up a wiki entry for that soon. Java is very particular about working only with trusted certificates. It is, of course, essential that it be so.

  • Overall Updates
    • The whole system may be configured to use SSL (from Visualization to Analysis and from Analysis to Active Directory). I have had great success using StartSSL’s free certificates.
    • Both the Visualization and Analysis servers may (should!) be installed behind an Apache server. I prefer this because it permits me to use Apache for SSL and mod_security. I’ve found that mod_proxy works best for the Visualization server and mod_jk works best for the Analysis server (mod_proxy with Analysis causes the WSDL files to be generated with “localhost” as the destination and mod_jk seemed unstable with the Visualization server).

    To try all of this out for yourself, download, compile and install the Analysis Server from the GitHub repository and head over to http://app.jdthomas.net and log in to your new Flower Network Flow Analysis system.

Written by justin

October 7th, 2010 at 10:00 pm

Posted in flower,programming

Might need an adjustment . . .

leave a comment

I think it might be time to add a sensitivity parameter to my network mapping code:

Network Map on Nmap

That’s the result of me running an nmap scan on my internal network to track down the IP address of an old network switch that I recently connected (which I don’t have the proper gendered serial connector to connect in to at the moment).

I suppose the upside is that it’s easy to see when someone is scanning my network.

Written by justin

August 16th, 2010 at 11:19 pm

Posted in flower,programming

Git Repository Available

leave a comment

I’ve published the Analysis Server code out to GitHub.com. Instructions for building and deploying the server from source (using Ant – NetBeans is not required) are included on the Wiki.

This is a screenshot of the Visualization interface as currently available at https://app.jdthomas.net:

Flower Visualization Console

Flower Visualization Console

I’ve made some fundamental changes to how data is stored within the Analysis Server (see StatisticsManager.java on GitHub for details). Instead of building the charts off of raw NetFlow data, those flows are now normalized according to time increment (set at a 5 second interval currently). This allows me to query based on a time specified time frame without worrying about how to handle the start and stop times that might fall outside of that range. It also sets up a scenario where we can derive some interesting statistical information from the data; more on that to come soon, hopefully.

I’ll update the EC2 AMI soon and remove the old image in favor of the new. I’m planning to also add Apache and mod_security to that server to protect the web service better for anyone who chooses to use that service.

Written by justin

August 13th, 2010 at 5:28 am

Posted in flower,programming

Flow Patterns

leave a comment

I’ve been noticing some interesting patterns using my Flower Analysis tool.

For whatever reason, I’ve noticed that one of my servers is continually chatting (in 3.2 – 4MB increments) with a whole lot of servers at Google. These chats generally occur over port 80/tcp. I first noticed the traffic in an area chart that showed a spike pattern in a very consistent rhythm. I tracked down the destination addresses of the flows and determined that they belong to networks owned by Google. I then added those networks to my map.

I’ll be doing more research on this; I’m at a loss as to why this one server would be sending (or receiving) so much information to (or from) Google.

I suspect espionage.

Written by justin

July 15th, 2010 at 6:58 am

Posted in flower,programming

Flower EC2 AMI Available

leave a comment

Here is a current shot of the Flower console:

I’ve made an EC2 AMI of the Analysis Server available on Amazon’s AWS. If you’d like to try the system out and already have EC2 configured, follow these instructions:

1. Configure a security group that allows ports 22/tcp, 8080/tcp, and 9995/udp
2. Create an EC2 instance of ami-3857bf51; the command I use is (my security group is called “appserver”):

ec2-run-instances -g appserver ami-3857bf51

3. Once loaded, log in via SSH and start the Flower server (all passwords are set to “flower” by default):

asadmin start-domain flower

4. Use http://app.jdthomas.net to log in to your server; in the server field, enter “http://[EC2 IP Address]:8080″ and both the username and password are “flower” – just hit [Enter] on any field to submit the form (there is no button)

Of course, app.jdthomas.net will need access to your EC2 system on port 8080/tcp (it uses JAX-WS to access and present your data to you), so if you lock things down be sure to account for that. HTTPS works, but you’ll need to configure and use a valid certificate on your EC2 system.

You can now configure your NetFlow exporters to send to the IP address of your EC2 instance on UDP port 9995 (version 9 only). I use softflowd on a couple of Debian systems with this command:

softflowd -i eth0 -n [EC2 IP Address]:9995 -p /var/run/sfd.pid.ec2 -c /var/run/sfd.ctl.ec2 -v 9

Configure your server using the “Manage” menu in the web application. Be sure to add networks (use CIDR addressing like 192.168.1.0/24) and users. Specified networks are required for the map, but area charts and flow tables will work without any networks specified.

E-mail me at justin@jdthomas.net with any questions! There are lots and lots of bugs and missing features, but I’ll get to them over the coming months.

UPDATE: I made a few changes tonight to improve the Visualization server configuration; the URL is now http://app.jdthomas.net (this will redirect you to an SSL site of the same base but with a longer path). I’ve made that change in the text above as well.

Written by justin

July 8th, 2010 at 3:48 am

Posted in flower,programming

Flower Network Flow Analysis

2 comments

Here’s a recent shot of the Flower console.

Flower Console

In addition to the updated interface (I finally accepted AJAX into my life and it opened a lot of possibilities), there are a ton of changes under the hood. I’ve deprecated the Flower Capture Engine in favor of collecting NetFlow (v9) data from standard generators (Cisco routers, softflowd, etc.) Storage has been migrated from a relational database to a BerkeleyDB backend; this move greatly simplified the code and increased performance significantly.

I’ll be looking for a few beta testers who are willing to fling some NetFlow data across the Internet; contact me at justin@jdthomas.net if you’re interested. In the future, you’ll be able to deploy a collector on site that will encrypt the data before delivering it to the analysis server, but that part isn’t quite ready for public consumption yet.

Written by justin

June 10th, 2010 at 6:05 am

Posted in flower,programming