We’ve Moved!

Visit our new blog location on www.exelisvis.com!

Linux support in IDL 8.3

In IDL 8.3, scheduled for this fall, we will be updating our minimum build support on Linux to Red Hat Enterprise Linux 6. Minimum parameters are kernel 2.6.32 and glibc 2.12.

As a part of this update, we’ll be discontinuing support for 32-bit Linux. IDL version 8.2.3 or earlier will continue to run on 32-bit Linux machines, but you won’t be able to upgrade these machines to IDL 8.3.

My intent is to prepare you by announcing this change as early as possible.

A WRITE_VIDEO example

The routines QUERY_VIDEO, READ_VIDEO and WRITE_VIDEO, introduced in IDL 8.2.3, act as procedural wrappers around the IDLffVideoRead and IDLffVideoWrite classes. These routines are intended to provide a simplified interface for working with video data, analogous to the QUERY_IMAGE, READ_IMAGE and WRITE_IMAGE routines for working with images. They should also evoke the look and feel of XINTERANIMATE and the old MPEG routines (MPEG_OPEN, MPEG_PUT, etc.).

Here’s a simple example of making a movie with WRITE_VIDEO.

Start by initializing a video file with a call to WRITE_VIDEO, returning a handle:

outfile = 'write_video_ex.avi'
video_dims = [640, 512]
write_video, outfile, handle=h, video_dimensions=video_dims, video_fps=24

The handle h is used as a reference to the open file.

Next, display a contour plot:

c = contour(dist(40), c_value=indgen(6)*5, rgb_table=72, $
   /fill, planar=0, dimensions=video_dims, $
   title='IDL 8.2.3 WRITE_VIDEO Example')

Now rotate the contour plot 181 times and take a screen grab on each iteration. Write the grab to the video file opened above, using the handle as a reference.

for i=0, 180 do begin
   c.rotate, 2, /zaxis
   sgrab = c.copywindow()
   write_video, outfile, sgrab, handle=h
endfor

Although the handle is specified, the outfile parameter is still needed in the call to WRITE_VIDEO.

Last, close the video file:

write_video, /close, handle=h

The file write_video_ex.avi should be written to your current IDL directory.

The movie is best viewed on repeat in your favorite movie player. (Be careful, though, it’s hypnotic.)

VISualize 2013: wrap-up and presentations

From Matt Hallas, VISualize 2013 organizer:

Hello VISualize 2013 attendees and speakers! Thank you all very much for another extremely successful ENVI and IDL user group. This event would not have been possible without the participation of our attendees and speakers and the support of the World Wildlife Fund.

We have made almost all of the presentations from VISualize available in PDF format, so you can see any you missed or share these data and image analysis solutions with colleagues. To access the presentations, simply follow this link or click the image above.

The presentations at VISualize help to tell the story of remote sensing and GIS from data collect to analysis. The range of topics detailing satellite launch and sensor requirements, community data sharing ventures, analysis of data from all types of sensors including LiDAR and HSI, as well as analysis of the impacts of land conservation programs paint a nearly complete picture of what we all focus on throughout the US and the world.

We are already planning VISualize 2014, so be sure to keep this event in mind as you work on various projects over the next twelve months. Once again, thank you all so much for making VISualize 2013 a great success, and we hope to see you again next year!

How many frames are in a movie?

In IDL 8.2.3, we introduced video read capabilities, though the IDLffVideoRead class and the READ_VIDEO function, to complement the video write capabilities available since IDL 8.1. I’m still new to using video as a data format, so I thought I’d post an example of something interesting that I learned recently.

We’ve included a video of a coronal mass ejection viewed from NASA’s SDO and SOHO spacecraft in the IDL distribution:

IDL> video_file = file_which('CME.mp4')

How many frames are in this video file? The answer isn’t as simple I’d expected.

Start with QUERY_VIDEO, which can return a structure of information about a video file:

IDL> !null = query_video(video_file, video_info)
IDL> print, video_info.num_frames
         574

There are 574 frames in the file. But wait, let’s try to read the entire file into IDL with READ_VIDEO:

IDL> movie = read_video(video_file, /all)
IDL> help, movie
MOVIE           BYTE      = Array[3, 512, 288, 564]

READ_VIDEO returns frames as pixel-interleaved RGB images. So I guess there are only 564 frames in the file?

Let’s turn to the lower-level API exposed in IDLffVideoRead to check whether it provides different information. The GetStreams method gives information about the single video stream in the file:

IDL> v = idlffvideoread(video_file)
IDL> print, (v.getstreams()).count
         574

If you look at the source code for QUERY_VIDEO, you’ll see that it uses this technique for returning the frame count. But what about iterating through the file, reading frame by frame, until the end is reached?  This code block:

   i = 0
   repeat begin
      data = v.getnext(type=t)
      ++i
   endrep until t eq -1

does so. The result:

IDL> print, --i
         564

OK, why are there two different values for the number of frames in the video file? I asked an engineer on the IDL team, Andrew Magill, who is far more knowledgeable about video than I am, about this. He offered a pair of possibilities:

It’s possible FFmpeg doesn’t actually know ahead of time how many frames there are. The number out of ::GetStreams might be an estimate based on video length, framerate, file size, etc. Or maybe there are actually 574 frames, but the last 9 can’t be decoded.

Andrew also gave some technical details that I haven’t included, and suggested that these may not be the only possibilities. Further, I thought his summary was enlightening:

Unfortunately, video technology is full of these little technical gotchas, and seems to be full of questions that can only be answered with “well, it depends”.  FFmpeg can seem really inconsistent sometimes, but I think they’ve done a heroic job of making all these different standards work almost exactly the same.

I hope that through Andrew’s work, and the power of FFmpeg, we can make video processing a straightforward task in IDL.

I’ll post other examples of working with video as I learn more about it!

Reminder: VISualize 2013

A reminder that VISualize 2013 starts tomorrow! If you’re in the area, please stop in to see interesting talks on using ENVI and IDL in the areas of climate change and environmental monitoring. Click on the banner for more information.

IDL 8.2.3 released!

IDL 8.2.3 is now available for download from our website, exelisvis.com. It represents the latest work of the IDL development team, including new functionality in the areas of visualization and file access, as well as over 100 bugs fixed.

File Access

• New routines to provide read/write support for ZIP, GZIP and TAR archive formats:
  • FILE_GZIP / FILE_GUNZIP
  • FILE_ZIP (which I used to package the demo programs) / FILE_UNZIP
  • FILE_TAR / FILE_UNTAR
  • ZLIB_COMPRESS / ZLIB_UNCOMPRESS (in memory)
• New IDLffVideoRead class provides read capabilities for MP4, AVI, etc., movie files
• New READ_VIDEO, WRITE_VIDEO, QUERY_VIDEO procedural wrappers to IDLffVideoRead/IDLffVideoWrite

Visualization

• New BUBBLEPLOT and VOLUME functions
• (New) Graphics now provides output to Scalable Vector Graphics (SVG) files
• Headless mode in (New) Graphics (i.e., setting BUFFER) doesn’t require an X server
• Improvements to speed of overplotting in (New) Graphics
• IMAGE, CONTOUR and SURFACE functions now handle irregularly gridded data
• TEXT function can display an overline above a group of characters
• IDLgrAxis and the AXIS function have a SUBGRIDSTYLE property

Language

• New TERMINAL_SIZE function allows a user to get the number of columns and rows in IDL Workbench Console or IDL Command Line
• The LIST::ToArray method now has a DIMENSION keyword
• Added the imaginary number “i” to !CONST

Analysis

• INTERPOLATE now has a DOUBLE keyword to force output type to double precision

Workbench

• Eclipse upgrade to 3.7.2
• SVN connectors update: SVN Kit 1.7.8, which is compatible with SVN 1.7

Seminar: Using IDL 8 In Earth Systems Science

Chris Torrence (IDL Project Lead) and I are visiting the NCAR Mesa Lab today (Tuesday, June 4) to give the aforenamed talk at 3 pm in the Director’s conference room. If you’re at the lab, please join us!

Beware the behavior of SMOOTH

Suppose I have an array that has an outlier. A really big outlier:

IDL> a = [1.0, 1.0, 2.0, 3.0, 4.0, 1.0d18, 4.0, 3.0, 2.0, 1.0, 1.0]

I’d like to smooth this double-precision array with a running mean (or boxcar or tophat, depending on where you learned this technique) filter, such as provided by the IDL SMOOTH function. For efficiency, SMOOTH divides the kernel width into a running total of the differences between neighboring values. Here’s the result of applying SMOOTH to A with a filter width of 3:

IDL> print, smooth(a, 3)
       1.0000000       1.3333333       2.0000000       3.0000000  3.3333333e+017
  3.3333333e+017  3.3333333e+017      0.00000000      -1.0000000      -1.6666667
       1.0000000

Whoa. The input array is symmetric, so why isn’t the output? Also—and this is worrisome—the input array is composed of positive numbers, so how can the mean of any subset of these numbers be negative?

The answer lies in the way floating point numbers are represented on computers. To see why, we can use the information returned from the MACHAR function:

IDL> m = machar(/double)
IDL> help, m
** Structure DMACHAR, 13 tags, length=72, data length=68:
IBETA           LONG                 2
IT              LONG                53
IRND            LONG                 5
NGRD            LONG                 0
MACHEP          LONG               -52
NEGEP           LONG               -53
IEXP            LONG                11
MINEXP          LONG             -1022
MAXEXP          LONG              1024
EPS             DOUBLE      2.2204460e-016
EPSNEG          DOUBLE      1.1102230e-016
XMIN            DOUBLE      2.2250739e-308
XMAX            DOUBLE      1.7976931e+308

Using the IBETA field, which gives the base used to construct numbers (2, no surprise), and the IT field, which gives the number of base-2 digits used in the mantissa of a number, the maximum resolvable distance between two double precision numbers must be given by:

IDL> mrd = double(m.ibeta)^m.it
IDL> print, mrd
  9.0071993e+015

So, when SMOOTH tries to difference two numbers whose distance is greater than MRD, bad things can happen because of loss of precision. Let’s apply this information to the example above. Define values slightly above and below the threshold set by MRD:

IDL> below = double(m.ibeta)^(m.it-1)
IDL> above = double(m.ibeta)^(m.it+1)
IDL> print, below, above
  4.5035996e+015  1.8014399e+016

and substitute them into the array used above:

IDL> b = [1.0, 1.0, 2.0, 3.0, 4.0, below, 4.0, 3.0, 2.0, 1.0, 1.0]
IDL> c = [1.0, 1.0, 2.0, 3.0, 4.0, above, 4.0, 3.0, 2.0, 1.0, 1.0]

Now apply SMOOTH to these arrays and evaluate the results:

IDL> print, smooth(b, 3)
       1.0000000       1.3333333       2.0000000       3.0000000  1.5011999e+015
  1.5011999e+015  1.5011999e+015       3.0000000       2.0000000       1.3333333
       1.0000000
IDL> print, smooth(c, 3)
       1.0000000       1.3333333       2.0000000       3.0000000  6.0047995e+015
  6.0047995e+015  6.0047995e+015       3.3333333       2.3333333       1.6666667
       1.0000000

Note that SMOOTH works well when applied to B, but not to C—the results aren’t symmetric.

A better discussion of this behavior is given in the section Note on Smoothing Over Large Data Ranges in the IDL Help page for SMOOTH, along with a workaround for this situation.

Update (2013-07-01): I neglected to mention that the seed for this post, as well as the note in the Help, came from discussions a few years ago with Carmen Lucas at DRDC Atlantic. Thanks, Carmen, for pointing out this unexpected behavior to me.

Where am I?

Well, more accurately, “where on the filesystem is the source for the currently scoped routine?”, but that’s not a catchy title. In IDL 8.2.2, the ROUTINE_FILEPATH function was modified to return this information when called without any parameters. Here’s a simple program demonstrating this behavior:

pro routine_filepath_no_params
   compile_opt idl2

   print, 'I''m called from a program in the file: ' + routine_filepath()
end

When I run this program, I get:

IDL> routine_filepath_no_params
I'm called from a program in the file: /home/mpiper/VIS/IDL/8/22/demos/src/routine_filepath_no_params.pro

Jim Pendleton, for example, has found this helpful in determining whether a program is being run from a SAVE file or a PRO file. I’ve used it to save a visualization to a PNG file with the same base name as the PRO file in which it’s generated.

(This is, I think, my last 8.2.2 post before 8.2.3 comes out next week!)