IKAROS 1.0.0

My Notes on IKAROS 1.0.0

C. G. Prince
6/26/07; 7/5/07; 7/26/07

These are my notes on getting my iSenseStream application working with IKAROS 1.0.0.

1. Problem with dynamic linking libjpeg

In my first attempt to run IKAROS 1.0.0, I obtained the following error:

Crspybits:~/Desktop/IKAROS.1.0.0/Examples chris$ IKAROS
dyld: Symbol not found: __cg_jpeg_resync_to_restart
Referenced from: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ImageIO.framework/Versions/A/ImageIO
Expected in: /usr/local/lib/libjpeg.62.dylib

Trace/BPT trap
Crspybits:~/Desktop/IKAROS.1.0.0/Examples chris$ ls
Media group.ikc morphology.ikc
double_randomizer.ikc images.ikc sequence.ikc
edge_detectors.ikc including_file.ikc views.ikc
gridworld_demo.ikc legacy.xml web_ui_objects.ikc

Crspybits:~/Desktop/IKAROS.1.0.0/Examples chris$ IKAROS edge_detectors.ikc
dyld: Symbol not found: __cg_jpeg_resync_to_restart
Referenced from: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ImageIO.framework/Versions/A/ImageIO
Expected in: /usr/local/lib/libjpeg.62.dylib

Trace/BPT trap

It seems this problem was caused by having multiple installations of libjpeg on my system. By commenting out a line in my .bashrc, I resolved this:

#export DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH:/usr/local/lib

It seems it was a library conflict. I found a hint that the OS X ImageIO framework ships with the libjpeg/libpng libraries now (http://wiki.urbanek.info/index.cgi?TigeR).

2. Problem with initial running of Mixelgram example

I have linked in my Mixelgram and RMSAudio modules into IKAROS 1.0.0 now, and just tried to run it from the command line. I get the following:

IKAROS: WARNING: Could not create module: Class "RMSAudio" does not exist.
IKAROS: WARNING: Could not create module: Class "Mixelgram" does not exist.
IKAROS: WARNING: Could not connect: Module "mixelgram" does not exists.

I'm not sure why.

a) Possibly, all modules need the .ikc description files that are now required in version 1.0.0. That does not appear to be the case. I just added a .ikc file for the Mixelgram module, and I still get the above warning for the Mixelgram module.

b) Do I have IKAROS properly compiled and linked?

To try to resolve this issue, I did a "Clean" in Xcode and am now trying to rebuild IKAROS 1.0.0. I now have obtained 474 errors, which are basic compilation problems. For example: "/usr/include/c++/4.0.0/cstring:79: error: '::memcpy' has not been declared". These errors are coming from the GNU file "cstring".

I am using Xcode 2.4. I'm going to try using Xcode 2.4.1 to see if I can resolve this problem. Hmmm. Nope. I installed Xcode 2.4.1, and this does not resolve my problem.

The problem occurs with the following #include in IKAROS.1.0.0/Source/Kernel/main.cc:

#include "UserModules/iSenseStream/RMSAudio/RMSAudio.h"

Of course, this is one of my #include files. It seems that the problem must come from RMS.h within RMSAudio.h. RMS.h includes System.h. It seems clear that System.h is on the path to causing this problem. When I use g++ from the command line to compile System.cpp (in iSenseStream/CommonFiles/Marsyas), which also includes System.h, I do not get this problem. Also, within Xcode, when I compile just this file (System.cpp) I do not get any compilation errors. It would seem it is a context or interaction issue. Somehow, in the context of IKAROS.1.0.0/Source/Kernel/main.cc, the System.h file is causing problems.

Continuing to explore this issue, in IKAROS.1.0.0/Source/Kernel/main.cc, the include sequence:

#include "System.h"
#include "Modules/Modules.h"
#include "UserModules/UserModules.h"

does not cause a problem. Compilation succeeds. However, the sequence:

#include "Modules/Modules.h"
#include "System.h"
#include "UserModules/UserModules.h"

does cause the 474 syntax errors. The sequence:

#include "Modules/Modules.h"
#include "UserModules/UserModules.h"

also does not cause any syntax errors. So, it seems clear that somehow the compilation problems arise from an interaction between Modules.h and System.h. Continuing on the debugging path, within Modules.h, if I include System.h before InputQTAudio.h, I do not get the errors. If I put it after InputQTAudio.h, I do get the errors. So, the problem appears to be an interaction between InputQTAudio.h and System.h. Further continuing the debugging path, the issue seems to arise with the code

namespace qt {
#include <QuickTime/QuickTime.h>
}

within InputQTAudio.h. If System.h is included before this namespace code, then no compilation issue occurs. However, when System.h is included after this namespace code, the 474 compilation errors occur.

This problem can be resolved if the include sequence within IKAROS.1.0.0/Source/Kernel/main.cc is changed from:

#include "Modules/Modules.h"
#include "UserModules/UserModules.h"

to:

#include "UserModules/UserModules.h"
#include "Modules/Modules.h"

This is certainly in the category of "hack", but so far I can't find another fix to the problem.

I now no longer get the warnings:

IKAROS: WARNING: Could not create module: Class "RMSAudio" does not exist.
IKAROS: WARNING: Could not create module: Class "Mixelgram" does not exist.
IKAROS: WARNING: Could not connect: Module "mixelgram" does not exists.

but, another issue has come up (problem 4 below).

3. Another error has reared its head. Now when I try to build with Xcode, I get a dialog box that says:

Error During Event Broadcast

*** -[NSCFDictionary setObject:forKey:]: attempt to insert nil key

This would appear to be an Xcode error and nothing to do with my use of it.

I just tried rebooting my system, and building now does not cause this error!

4. Running IKAROS 1.0.0 from a Terminal on my example, I encounter a problem:

To set up, I first do:

Crspybits:~/Desktop/IKAROS.1.0.0/Source/UserModules/iSenseStream/Mixelgram/Example chris$ ls
Mixelgram_test.xml tape1-clip1-cropped-first60s.dv
Crspybits:~/Desktop/IKAROS.1.0.0/Source/UserModules/iSenseStream/Mixelgram/Example chris$ IKAROS -w8000 Mixelgram_test.xml
IKAROS: Reading XML file "Mixelgram_test.xml".
IKAROS: WARNING: Did not find <group> element in IKC/XML file "Mixelgram_test.xml". Will use legacy mode.
IKAROS: WARNING: Module "rms" (RMSAudio) uses DEPRECATED contructor Module::Module(char *).
IKAROS: WARNING: Module "mixelgram" (Mixelgram) uses DEPRECATED contructor Module::Module(char *).
IKAROS: WARNING: InputQTMovie is not thread safe
IKAROS: WARNING: InputQTAudio is not thread safe
Mixelgram::Init: m_audioWidth (number of audio features)= 1
Correlator::Correlator: m_audio_n (audio)= 1, m_visual_m (visual)= 1
IKAROS: Starting Ikaros WebUI server.
IKAROS: Connect from a browser on this computer with the URL "http://127.0.0.1:8000/".
IKAROS: Use the URL "http://<servername>:8000/" from other computers.

Then, I launch Firefox, type "http://127.0.0.1:8000" into the browser, and click on "Movie View", and the following additional message comes up from IKAROS within the Terminal:

Cannot find "viewer.svg"; webui_dir="/Users/chris/Desktop/IKAROS.1.0.0/Source/UserModules/iSenseStream/Mixelgram/Example/../Source/WebUI/UI/"

If I locate my example code within the Source directory, I no longer have this problem:

Crspybits:~/Desktop/IKAROS.1.0.0/Source chris$ pwd
/Users/chris/Desktop/IKAROS.1.0.0/Source
Crspybits:~/Desktop/IKAROS.1.0.0/Source chris$ IKAROS -w8000 Mixelgram_test.xml
IKAROS: Reading XML file "Mixelgram_test.xml".
IKAROS: WARNING: Did not find <group> element in IKC/XML file "Mixelgram_test.xml". Will use legacy mode.
IKAROS: WARNING: Module "rms" (RMSAudio) uses DEPRECATED contructor Module::Module(char *).
IKAROS: WARNING: Module "mixelgram" (Mixelgram) uses DEPRECATED contructor Module::Module(char *).
IKAROS: WARNING: InputQTMovie is not thread safe
IKAROS: WARNING: InputQTAudio is not thread safe
Mixelgram::Init: m_audioWidth (number of audio features)= 1
Correlator::Correlator: m_audio_n (audio)= 1, m_visual_m (visual)= 1
IKAROS: Starting Ikaros WebUI server.
IKAROS: Connect from a browser on this computer with the URL "http://127.0.0.1:8000/".
IKAROS: Use the URL "http://<servername>:8000/" from other computers.
InputQTAudio::Tick: numSamples= 1601, audioFlags= 0
RMSAudio::Tick: num audio samples == 0 ...
InputQTAudio::Tick: numSamples= 1602, audioFlags= 0
RMSAudio::Tick: num audio samples == 3202 ...

Is an inappropriate strategy being used within IKAROS to locate the WebUI directory? Looking into WebUI.cc, it would appear that the variable k->ikaros_dir is not getting set correctly somewhere. If "ikaros_dir" is actually supposed to be the directory for IKAROS, it's not getting set that way. Rather, its getting set as the path to the current working directory. It appears that this problem is arising because of the #define IKAROSPATH within IKAROS_System.h. It seems that a -DIKAROSPATH needs to define the base location of IKAROS when the system is compiled. (Would it be better to have this as an environment variable?). I'll just work with this for now. Within Xcode, I have now put

-DIKAROSPATH='"/Users/chris/Desktop/IKAROS.1.0.0/"'

within "Other C++ Flags", within the menu Project/Edit Active Target 'IKAROS'. Note the use of outer single quotes and inner double quotes. This seems necessary to get the double quotes into the string for the C++ compiler. Also note that the forward slash before the ending quotes is necessary.

5. Somehow when I compile IKAROS using Xcode, I don't get all of the dependencies tracked correctly. For example, if I change some of .h files, and then try a build, it does not detect that they .h files were changed, and doesn't recompile. I'm forced to do a "Clean" and rebuild everything to ensure that all necessary recompilation is carried out.

6. The standard InputQTMovie module is still using a crude grayscale conversion. I'm going to continue using my InputQTMovie2 module. I just ran a brief test with InputQTMovie2, and it seems to be working fine.

Note: There is an onling bug with InputQTMovie2 that will likely prohibit it's more general use. When InputQTMovie2 is not running in the GUI foreground, it fails. There appears to be some kind of system context dependency here. I've not tracked this problem down. :(.

7. In doing a test with OutputQTAudioVisual, I am having a problem. Both the visual and audio tracks have problems. First, I'm digging into the documentation (hmmm. the OutputQTAudioVisual.html file that was present in verison 0.8.1 was not carried along to the 1.0.0 version) for OutputQTAudioVisual. I am now trying the previous version of OutputQTAudioVisual (that was present in IKAROS 0.8.1) to see if there was some error in changes to the new version of IKAROS. This does not resolve the problem. The previous version of OutputQTAudioVisual provides the same erroneous results. What else could be causing this problem?

A) My testing example (e.g., I'm using a DV format input file).

I just tried another video format, and this produces the same results. It seems clear the problem was not specific to my testing example.

B) Changes in the Quicktime system interfaces.

If I try my old IKAROS (0.8.1), this could provide some insight into the problem. If I recompile my old IKAROS, and it runs the example, this would suggest that it is not the Quicktime system interfaces, but rather the change to IKAROS 1.0.0. Ok, so I just tried this. The old version of IKAROS (0.8.1) no longer runs my example. I get the same results as with IKAROS 1.0.0. This suggests that something has changed in the Quicktime interfaces and the OutputQTAudioVisual module is not (yet) incorporating that change.

The problem with the visual track is in part a setting for max pixel value. Setting that to 255 causes the visual output to look appropriate. However, I'm still getting what appears to be a very slow frame rate on the output video file. It would appear that we are not advancing appropriately to the next frame in the input video file.

Hmmm. I had been using the InputQTMovie module, but now I've switched to using the InputQTMovie2 module. Now, I'm getting the input data appropriately-- i.e., I'm getting all frames of the input movie. Is something broken with the InputQTMovie module?

Now, onto debugging the audio track issue. This issue could arise due to the InputQTAudio module or the OutputQTAudioVisual module. I just used a testing module of mine (AudioVisualExample) to provide a visualization of the audio output of the InputQTAudio module. It seems clear that the audio output of this module is reasonable. It thus also seems clear that the problem I am having has to do with the OutputQTAudioVisual.

The problem with the OutputQTAudioVisual could be coming in two places. First, I could be having a problem with getting the data from the InputQTAudio module into the OutputQTAudioVisual module (less likely) or I could be having a problem getting the data out of the OutputQTAudioVisual module into the Quicktime file (more likely).

OK. So, I've resolved this now. It does have to do with my change to an Intel Mac. In OutputQTAudioVisual.cc there was a line:

asbd.mFormatFlags = kAudioFormatFlagIsFloat | kAudioFormatFlagIsBigEndian;

If you change this to

asbd.mFormatFlags = kAudioFormatFlagIsFloat;

the code works again. It appears there is an endianness difference acrosss PPC and Intel. It would seem there is a need for a compile time flag in IKAROS to distinguish between Mac OS PPC and Mac OS Intel. For present I've put a

#define INTEL_MAC

into OutputQTAudioVisual.h.

C) Something to do with the change to IKAROS 1.0.0.

D) Something to do with my change to an Intel Mac. (YES!)

8. Now, I'm back to testing my IKAROS application (iSenseStream), and I'm having a problem with the OutputFile module. An expected output ("EDGE_DETECT_SUM") does not exist. Comparing the old code for the OutputFile module to the new code for this module, there have been numerous changes to the module. The new version of the module has been coded in terms of explicit XML operations. It appears this has interfered with the operation of my old XML file. For my XML code:

  <module
    class = "OutputFile"
    name = "output"
    filename = "t1c1crpd.mov.output.txt">
    <column name = "EDGE_DETECT_SUM" />
  </module>
  
The new OutputModule is not detecting the column. I think I've figured out the problem. It appears that the line of code:

XMLNode * par = xml->parent;

in OutputFile should instead be:

XMLNode * par = xml;

This leaves "par" referencing the current node and not the parent of the node. The corrected file is here.

9. Continuing on my debugging trail with iSenseStream, I still have a problem. After some intialization, I am getting:

IKAROS: outputAV-mixelgram::Tick (OutputQTAudioVisual) Start
Floating point exception

and then, of course, IKAROS fails. Throwing some debugging code into OutputQTAudioVisual, the code is failing in m_soundTrack->AddSamples. Digging further, SoundTrack::AddSamples is failing in the call to AddMediaSample2. Perhaps this is not surprising. In AddSamples, I'm getting:

SoundTrack::AddSamples: numAudioFrames= 0
SoundTrack::AddSamples: sndDataSize= 0

From my documentation within OutputQTAudioVisual:

/* CGP; 7/5/07. On some Tick's I'm getting 0 audio samples. The place where I'm getting this right now is during initialization.
This is causing a floating point exception in AddSamples and specifically within AddMediaSample2. Possibly something has changed
in AddMediaSample2 within Quicktime, which disallows adding 0 audio samples? I'm going to work around this by not adding either
visual nor audio frame if I get no audio for a Tick.
*/

This has now resolved my present problems, and I now have a successful run of iSenseStream working. My new version of OutputQTAudioVisual is here (but see next issue).

10. I had some further problems with OutputQTAudioVisual. In particular, I had set up XML parameters to disable audio and/or visual tracks (e.g., audio_track_flag = "0"). This is not present in the IKAROS 1.0.0 OutputQTAudioVisual module.

It appears an "old" version of OutputQTAudioVisual was used in IKAROS 1.0.0. Here's my current version of OutputQTAudioVisual.