Uwar tools: Difference between revisions

From University of Washington - Ubicomp Research Page
Jump to navigationJump to search
No edit summary
mNo edit summary
 
(19 intermediate revisions by 2 users not shown)
Line 1: Line 1:
== Overview ==
== Overview ==
This document lists the various tools for working with uwar files and related data. An example .uwar file is hosted on this machine that you can use to test with. All outputs on this page are from this file: [{{SERVER}}/files/uwar/example.uwar example.uwar].
This document lists the various tools for working with uwar files and related data. An example .uwar file is hosted on this machine that you can use to test with. Most of the outputs on this page are from this file: [{{SERVER}}/files/uwar/example.uwar example.uwar].




== parse_uwar_dir ==
* <tt>msp/uwar_stats/parse_uwar_dir.cpp</tt>
* [{{SERVER}}/files/uwar/tools_linx86/parse_uwar_dir linux download (x86)]
* [{{SERVER}}/files/uwar/tools_win32/parse_uwar_dir.exe windows download]
* [{{SERVER}}/files/uwar/tools_win32/libsndfile.dll windows download (required libsndfile.dll)]


== central ==
parse_uwar_dir replaces uwarToMatlab and converts data into binary data blobs called sensorName.bin and metaData files called sensorName.meta. The meta data files are of the format:
* <tt>msp/src/ipc/central.c</tt>
<pre>
* [{{SERVER}}/files/uwar/imote2_sdcard/central linux download (xscale)]
"NumSamples",47013
* [{{SERVER}}/files/uwar/tools_linx86/central linux download (x86)]
"SampleRate",512.000000
* windows download
"Name","accelMag"
 
"DataType",1
central is the server for the IPC (Inter Process Communications) package developed by Reid Simmons. IPC provides a simple means of communicating between applications on the same computer and other machines via TCP/IP. Central is used by the msb-server and several other applications to share information. More information about IPC can be found at the projects [http://www.ri.cmu.edu/projects/project_394.html homepage].
"firstTimestamp",0.000000
"lastTimestamp",91.820313
</pre>
Where DataType is defined as:
<pre>
                case 1
                    DataType_string = 'single';
                    DataType_size  = 4;
                case 2
                    DataType_string = 'double';
                    DataType_size  = 8;
                case 3
                    DataType_string = 'int8';
                    DataType_size  = 1;
                case 4
                    DataType_string = 'uint8';
                    DataType_size  = 1;
                case 5
                    DataType_string = 'int16';
                    DataType_size  = 2;
                case 6
                    DataType_string = 'uint16';
                    DataType_size  = 2;
                case 7
                    DataType_string = 'int32';
                    DataType_size  = 4;
                case 8
                    DataType_string = 'uint32';
                    DataType_size  = 4;
                case 9
                    DataType_string = 'int64';
                    DataType_size  = 8;
                case 10
                    DataType_string = 'uint64';
                    DataType_size  = 8;
</pre>


=== usage ===
=== usage ===
<ul>
<ul>
==== Typical Usage ====
<ul>
Spawn central and send it to the background
<pre>
<pre>
central -u &
parse_uwar_dir [-options] <input file(s)>
</pre>
==== Available Commands ====
<pre>
parse_uwar_dir [-options] <input file(s)>
  [-outputdir N]
      Create all output in directory N rather than the inferred directory
      (Default behavior is to figure out the path from the first input file)
  [-maxGap_inSeconds N]
      Maximum gap between .uwar files (in seconds, default 2*60)
  [-ignoreRollover]
      Ignore roll over events that appear to be errors and continue
  [-NoWriteImageSubDir]
      Don't write images to 'images' sub directory in the target dir (default on)
  [-writeTraceFiles]
      Creates <filename>.processing files to UWAR which are open
      this provides some feedback should the program crash (default off)
  [-writeTraceLog]
      Creates conversion.log csv file to output directory, detailing which files
      have been processed, and where their output went (default off)
 
  [-scandir N]
      Scan the directory for .uwar files to process, arrange them into sessions and convert the data
</pre>
</pre>
Note, that only 1 copy of central can be run at one time. If you want to run multiple central's (see the -p argument below) or access central running on another machine you will need to export the CENTRALHOST variable:
<bash># If you want to use another port
export CENTRALHOST=localhost:1234
# If you want to connect to another machine
export CENTRALHOST=othermachine
</bash>
</ul>
</ul>
 
=== example output ===
==== Available Commands ====
<ul>
<ul>
Various debugging/logging capabilities are also available if necessary. Remember to have a process cat the .log file that central creates.
<pre>
<pre>
The following are central commands:
parse_uwar_dir example.uwar
help: print this message
# INFO: Data will be created in directory: .\
display: display the active and pending messages
 
status: display the known modules and their status
# INFO: ------------------
memory: display total memory usage
# INFO: Scanning files to remove obviously bad files
close <module>: close a connection to a module
# INFO: File [ 1]            Opened, "example.uwar"
  unlock <resource>: unlock a locked resource


The following command line options can also be used as commands:
# INFO: ------------------
-v: display server version information
# INFO: Processing Session  1 of  1
-l<option>: logging onto terminal. Options are:
# INFO: Creating output directory:
  m (message traffic)
# INFO: ".\part_00"
  s (status of IPC)
# INFO: Session  1, Processing File  1 of  1 (total)
  t (time messages are received by central)
# WARNING: Packet  14707, Filling  16 missing samples for field msb2/audio
  d (data associated with messages)
# WARNING: Packet  17420, Filling  16 missing samples for field msb2/audio
  i (ignore logging certain internal messages)
# WARNING: Packet  19526, Filling  16 missing samples for field msb2/audio
  h (handle time summary of incoming messages)
# WARNING: Packet  20792, Filling  16 missing samples for field msb2/audio
  r (log the reference ID as well as the message name)
# WARNING: Packet  30340, Filling  16 missing samples for field msb2/audio
  p (log the reference ID of the message’s parent)
# WARNING: Packet  37818, Filling  17 missing samples for field msb2/audio
  x (no logging)
# INFO: Session  1 of 1 completed
# INFO: ------------------
</pre>
</ul>


-l (no options) is equivalent to -lmstdh; the default is –lmsi


-L<option>: logging into file.
== uwar_tag_dump ==
  Options are the same as above, with the addition of
* <tt>msp/uwar_stats/uwar_tag_dump.cpp</tt>
  F (don’t flush file after each line)
* [{{SERVER}}/files/uwar/tools_linx86/uwar_tag_dump linux download (x86)]
  n (don’t prompt user for comments)
* [{{SERVER}}/files/uwar/tools_win32/uwar_tag_dump.exe windows download]
  The default is –Lx
* [{{SERVER}}/files/uwar/tools_win32/libsndfile.dll windows download (required libsndfile.dll)]


-f<filename>: filename to use for logging; If not specified, name is automatically generated.
uwar_tag_dump allows you to dump text tags from a TAG stream to output text file.


-p<port>: connect to central server on this port number.
=== usage ===
-c: Use direct (not via central) connections when possible
<ul>
-I<msgName>: Ignore logging this message (can occur multiple times).
<pre>
-I<filename>: File with names of messages to ignore logging
uwar_tagdump [-options] <input file(s)> -out <output file>
-s: silent running; don’t print anything to terminal.
</pre>
-u: don’t run the user (tty) interface.
==== Available Commands ====
-r: try resending non-completed messages when modules crash and then reconnect.
<pre>
uwar_tagdump [-options] <input file(s)> -out <output file>
  [-maxGap_inSeconds N]
      Maximum gap between .uwar files (in seconds, default 2*60)
 
  [-scandir N]
      Scan the directory for .uwar files to process, arrange them into sessions and convert the data
The output format is:
UWAR Timestamp, MSB2 Timestamp, TAG string
</pre>
</pre>
</ul>
</ul>
=== example output ===
=== example output ===
<ul>
<ul>
central typically does not output anything to the console unless an error occurs.
<pre>
</ul>
uwar_tagdump example.uwar -out test.txt
# INFO: Output TAG file will be: test.txt
 
# INFO: ------------------
# INFO: Scanning files to remove obviously bad files
# INFO: File [  1]            Opened, "example.uwar"
# INFO: Reading TAGs from UWAR Files...
# INFO: Reading File  1 of  1, example.uwar
# INFO: Done reading tags
</pre>
</ul>
</ul>




== msb-server ==  
== uwar_modify ==  
* <tt>msp/src/msb/msb-main.c</tt>
* <tt>msp/uwar_stats/uwar_modify.cpp</tt>
* [{{SERVER}}/files/uwar/imote2_sdcard/msb-server linux download (xscale)]
* [{{SERVER}}/files/uwar/tools_linx86/uwar_modify linux download (x86)]
* [{{SERVER}}/files/uwar/tools_linx86/msb-server linux download (x86)]
* [{{SERVER}}/files/uwar/tools_win32/uwar_modify.exe windows download]
* currently not available for windows
* [{{SERVER}}/files/uwar/tools_win32/libsndfile.dll windows download (required libsndfile.dll)]


The msb-server is a program which interfaces with the MSB via the serial port and handles logging of uwar files and transmission of the sensor data over IPC and other shared memory mechanisms. msb-server was written by Dirk Hähnel of Intel Research Seattle.
uwar_modify will change the metaValues in a UWAR header, the input file is used as the output.


=== usage ===
=== usage ===
<ul>
==== Typical Usage ====
<ul>
<ul>
<pre>
<pre>
msb-server -ini msb.ini
uwar-modify -streamID N -metaID N -metaValue N <input/output file>
</pre>
</pre>
where msb.ini will typically contain:
<pre>
#Desktops should use the USB device
#SERIAL_DEVICE /dev/ttyUSB1
#the iMote2 uses ttyS0 only
SERIAL_DEVICE /dev/ttyS0
BAUD_RATE       921600
HARDWARE_FLOW_CONTROL 0
SOFTWARE_FLOW_CONTROL 0
USE_RTC                      1
SCHEDULE_INTERRUPT_COUNT      5120
SCHEDULE_MIC                8192
SCHEDULE_ACCELEROMETER        5120
SCHEDULE_AMB_LIGHT            2560
SCHEDULE_BAROMETER              71
SCHEDULE_DALS                  30
SCHEDULE_HUMIDITY              10
SCHEDULE_POWER                  0
MIC_GAIN                        64
LOG_FILE test.uwar
LOG_START 1
</pre>
</ul>
==== Available Commands ====
==== Available Commands ====
<ul>
In addition the msb-server also supports the following switch commands:
<pre>
<pre>
-ipc-publish-data    publish the converted data via IPC
uwar-modify [-options]  <input file>
-ipc-publish-raw    publish the raw sensor data via IPC
  uwar-modify will change the metaValues in a UWAR header, the input file is used as the output
-ipc-publish-stream stream serial data via ipc
  [-streamID N]
-shm-publish-stream stream serial data via shared memory
      Specify the stream number (which start at 0) you want to change
-ipc-publish-mic    publish microphone packets
      If the stream number is not found an error will occur
-sync-rtc
  [-metaID N]
-show-rtc
       Meta ID we wish to change (IDs start at 0)
-set-rtc
      If the metaID number is not found an error will occur
-use-rtc
   [-metaValue N]
-log       specify a file to log data to, see -max-size and -max-time for log rotation settings
      New value to be written to the metaID (currently only type 'i' is supported)
-log-type  specify the type of data to log:
  uwar      serial data is stored as packets in a uwar file
  stream   serial data is written directly to a file
-max-size  maximum size of the uwar file before rotating the log file (in kilobytes)
-max-time  maximum time between file rotations
</pre>
</pre>
</ul>
</ul>
=== example output ===
=== example output ===
<ul>
<ul>
Typically the msb-server will output something similar to the following. Where the final recieved stats will continue until the program is terminated or an error occurs.
To change the schedule sample rate in a .uwar file we can use uwar-modify.
<pre>
<pre>
./msb-server -ini ./msb.ini -ipc-publish-data
uwar_modify -streamID -1 -metaID 11 -metaValue 71 "example.uwar
# INFO: ********************************
# INFO: Stream ID to be modified: -1
# INFO: ***********   MSB  ************
# INFO: Meta ID to be modified:  11
# INFO: ********************************
# INFO: Meta to be modified to:   71
# INFO: read ini file ./msb.ini
# INFO: Header values before changes:
# INFO: initialize data structures
# INFO: File: "example.uwar"
# INFO: allocate data messages
# INFO: UWAR File Contains  5 streams
# INFO: IPC for Mobile Sensing Platform Toolkit (Ver. 1.0.0)
# INFO: Symbol ID = auto-generated ID number used inside each data packet
# INFO: set device:
# INFO:       ID = UWAR_STREAM ID for identifying stream types
# INFO:   port   = /dev/ttyUSB1
# INFO:
# INFO:   baud   = 921600
# INFO: Stream   0: Symbol ID= 0 (ID = 0x2063696D:'mic ') [MIC]
# INFO:   params = 8N1
# INFO: Stream   1: Symbol ID=  1 (ID = 0x3262736D:'msb2') [MSB2]
# INFO: send reset command
# INFO: Stream  2: Symbol ID= 2 (ID = 0x20676174:'tag ') [TAG]
# INFO: ---------------------
# INFO: Stream  3: Symbol ID=  3 (ID = 0x6B636C63:'clck') [CLOCK]
# INFO: send cease command ... done
# INFO: Stream  4: Symbol ID= 4 (ID = 0x20737067:'gps ') [GPS]
# INFO: send rtc query command to MSB
# INFO:
# INFO: current msb time = 2065-25-45 45:85:85!
# INFO: UWAR File Meta Data:
# INFO: change system time to RTC!
# INFO: uwar header meta  0/ 17: (ID = 0x726E6462:'bdnr') type='i' = 11
# INFO: set system time failed!
# INFO: uwar header meta  1/ 17: (ID = 0x72657668:'hver') type='i' = 2
# INFO: send calibration query command to MSB
# INFO: uwar header meta  2/ 17: (ID = 0x616D7773:'swma') type='i' = 1
# INFO: ---- calibration ----
# INFO: uwar header meta  3/ 17: (ID = 0x696D7773:'swmi') type='i' = 1
# INFO: board nr            = b
# INFO: uwar header meta  4/ 17: (ID = 0x31306477:'wd01') type='i' = 48224
# INFO: hardware version nr = 2
# INFO: uwar header meta  5/ 17: (ID = 0x32306477:'wd02') type='i' = 47962
# INFO: major soft version  = 1
# INFO: uwar header meta  6/ 17: (ID = 0x33306477:'wd03') type='i' = 28705
# INFO: minor soft version  = 1
# INFO: uwar header meta  7/ 17: (ID = 0x34306477:'wd04') type='i' = 44956
# INFO: calibration word 1  = 48224
# INFO: uwar header meta  8/ 17: (ID = 0x72746E69:'intr') type='i' = 5120
# INFO: calibration word 2  = 47962
# INFO: uwar header meta  9/ 17: (ID = 0x2063696D:'mic ') type='i' = 8192
# INFO: calibration word 3  = 28705
# INFO: uwar header meta 10/ 17: (ID = 0x20636361:'acc ') type='i' = 5120
# INFO: calibration word 4  = 44956
# INFO: uwar header meta 11/ 17: (ID = 0x20626D61:'amb ') type='i' = 2560
# INFO: calibration val c1  = 24112
# INFO: uwar header meta 12/ 17: (ID = 0x20726162:'bar ') type='i' = 71
# INFO: calibration val c2  = 2140
# INFO: uwar header meta 13/ 17: (ID = 0x706D6F63:'comp') type='i' = 0
# INFO: calibration val c3 = 702
# INFO: uwar header meta  14/ 17: (ID = 0x736C6164:'dals') type='i' = 30
# INFO: calibration val c4 = 448
# INFO: uwar header meta 15/ 17: (ID = 0x206D7568:'hum ') type='i' = 10
# INFO: calibration val c5 = 749
# INFO: uwar header meta  16/ 17: (ID = 0x72776F70:'powr') type='i' = 0
# INFO: calibration val c6 = 26
# INFO: Stream  0: Symbol ID= 0 (ID = 0x2063696D:'mic ') [MIC]
# INFO: send schedule query command to MSB
# INFO: Stream  1: Symbol ID= 1 (ID = 0x3262736D:'msb2') [MSB2]
# INFO: --- USER REQUEST ---
# INFO: Stream  2: Symbol ID= 2 (ID = 0x20676174:'tag ') [TAG]
# INFO: ------ schedule -----
# INFO: Stream  3: Symbol ID= 3 (ID = 0x6B636C63:'clck') [CLOCK]
# INFO: interrupt count    = 5120
# INFO: Stream  4: Symbol ID= 4 (ID = 0x20737067:'gps ') [GPS]
# INFO: mic hz              = 8192
# INFO: File: "example.uwar"
# INFO: accelerometer hz    = 512.0
# INFO: UWAR File Contains  5 streams
# INFO: amb light hz        = 256.0
# INFO: Symbol ID = auto-generated ID number used inside each data packet
# INFO: barometer hz        = 7.1
# INFO:       ID = UWAR_STREAM ID for identifying stream types
# INFO: compass hz          = 0.0
# INFO:
# INFO: dals hz            = 3.0
# INFO: Stream  0: Symbol ID= 0 (ID = 0x2063696D:'mic ') [MIC]
# INFO: humidity hz        = 1.0
# INFO: Stream  1: Symbol ID= 1 (ID = 0x3262736D:'msb2') [MSB2]
# INFO: power              = 0
# INFO: Stream  2: Symbol ID= 2 (ID = 0x20676174:'tag ') [TAG]
# INFO: microphone gain    = 255
# INFO: Stream  3: Symbol ID= 3 (ID = 0x6B636C63:'clck') [CLOCK]
# INFO: send schedule query command to MSB
# INFO: Stream  4: Symbol ID=  4 (ID = 0x20737067:'gps ') [GPS]
# INFO: --- NEW SETTINGS  ---
# INFO:
# INFO: ------ schedule -----
# INFO: UWAR File Meta Data:
# INFO: interrupt count    = 5120
# INFO: uwar header meta  0/ 17: (ID = 0x726E6462:'bdnr') type='i' = 11
# INFO: mic hz              = 8192
# INFO: uwar header meta  1/ 17: (ID = 0x72657668:'hver') type='i' = 2
# INFO: accelerometer hz    = 512.0
# INFO: uwar header meta  2/ 17: (ID = 0x616D7773:'swma') type='i' = 1
# INFO: amb light hz        = 256.0
# INFO: uwar header meta  3/ 17: (ID = 0x696D7773:'swmi') type='i' = 1
# INFO: barometer hz        = 7.1
# INFO: uwar header meta  4/ 17: (ID = 0x31306477:'wd01') type='i' = 48224
# INFO: compass hz          = 0.0
# INFO: uwar header meta  5/ 17: (ID = 0x32306477:'wd02') type='i' = 47962
# INFO: dals hz            = 3.0
# INFO: uwar header meta  6/ 17: (ID = 0x33306477:'wd03') type='i' = 28705
# INFO: humidity hz        = 1.0
# INFO: uwar header meta  7/ 17: (ID = 0x34306477:'wd04') type='i' = 44956
# INFO: power              = 0
# INFO: uwar header meta  8/ 17: (ID = 0x72746E69:'intr') type='i' = 5120
# INFO: microphone gain    = 255
# INFO: uwar header meta   9/ 17: (ID = 0x2063696D:'mic ') type='i' = 8192
# INFO: ------ schedule -----
# INFO: uwar header meta  10/ 17: (ID = 0x20636361:'acc ') type='i' = 5120
# INFO: interrupt count    = 5120
# INFO: uwar header meta  11/ 17: (ID = 0x20626D61:'amb ') type='i' = 71
# INFO: mic hz              = 8192
# INFO: uwar header meta  12/ 17: (ID = 0x20726162:'bar ') type='i' = 71
# INFO: accelerometer hz    = 512.0
# INFO: uwar header meta  13/ 17: (ID = 0x706D6F63:'comp') type='i' = 0
# INFO: amb light hz        = 256.0
# INFO: uwar header meta  14/ 17: (ID = 0x736C6164:'dals') type='i' = 30
# INFO: barometer hz        = 7.1
# INFO: uwar header meta  15/ 17: (ID = 0x206D7568:'hum ') type='i' = 10
# INFO: compass hz          = 0.0
# INFO: uwar header meta 16/ 17: (ID = 0x72776F70:'powr') type='i' = 0
# INFO: dals hz            = 3.0
# INFO: Stream  0: Symbol ID=  0 (ID = 0x2063696D:'mic ') [MIC]
# INFO: humidity hz        = 1.0
# INFO: Stream  1: Symbol ID=  1 (ID = 0x3262736D:'msb2') [MSB2]
# INFO: power              = 0
# INFO: Stream  2: Symbol ID=  2 (ID = 0x20676174:'tag ') [TAG]
# INFO: microphone gain    = 255
# INFO: Stream  3: Symbol ID=  3 (ID = 0x6B636C63:'clck') [CLOCK]
# INFO: -------   log  -------
# INFO: Stream  4: Symbol ID=  4 (ID = 0x20737067:'gps ') [GPS]
# INFO: open log-file file test-0002.uwar [test-0002.uwar]
# INFO: write header to log file ... => 967903080
done
# INFO: log-max-size: 4 Mbytes
# INFO: log-max-time: 5.00 minutes
# INFO: ------- start -------
# INFO: send init command to MSB
# INFO:  2.40 kbits/sec (0 crc of 9 packets)
# INFO: 234.95 kbits/sec (0 crc of 169 packets)
# INFO: 221.89 kbits/sec (0 crc of 321 packets)
# INFO: unknown packet type 0xb4
# INFO: 221.89 kbits/sec (0 crc of 473 packets)
</pre>
</pre>
</ul>
</ul>
</ul>


Line 273: Line 295:
</pre>
</pre>
</ul>
</ul>


== uwar-info ==
== uwar-info ==
Line 377: Line 398:
# INFO: read UWAR v2 header
# INFO: read UWAR v2 header
# INFO: read UWAR v2 data
# INFO: read UWAR v2 data
</pre>
</ul>
== uwar-clean ==
* <tt>msp/src/fileio/uwar-clean.c</tt> (not yet committed to CVS)
* [{{SERVER}}/files/uwar/tools_linx86/uwar-clean linux download (x86)]
* [{{SERVER}}/files/uwar/tools_win32/uwar_clean.exe windows download]
uwar_clean removes corrupted packets from a .uwar file and stores the cleaned stream in a new file. Note, this program is based off of uwar-info so any file uwar-info can process it should be able to handle. It does NOT clean bad data inside a valid uwar packet.
=== usage ===
<ul>
<pre>
uwar_clean <input_uwar_filename> <output_cleaned_uwar_filename>
</pre>
Note: Be careful not to specify the same input/output name as this will immediately erase your file. Remember if you say something like 'uwar_clean myVeryImportantData.uwar ./myVeryImportantData.uwar' you will likely lose your data!
</ul>
=== example output ===
<ul>
<pre>
uwar_clean.exe log-1969-11-31-16.00.dat log-1969-11-31-16.00_clean.dat
# INFO: open file log-1969-11-31-16.00.dat
# INFO: read UWAR v2 header
# INFO: all meta  0: id=(726e6462:bdnr) type='i' = 61463
# INFO: all meta  1: id=(72657668:hver) type='i' = 2
# INFO: all meta  2: id=(616d7773:swma) type='i' = 1
# INFO: all meta  3: id=(696d7773:swmi) type='i' = 1
# INFO: all meta  4: id=(31306477:wd01) type='i' = 39064
# INFO: all meta  5: id=(32306477:wd02) type='i' = 31260
# INFO: all meta  6: id=(33306477:wd03) type='i' = 36959
# INFO: all meta  7: id=(34306477:wd04) type='i' = 42677
# INFO: all meta  8: id=(72746e69:intr) type='i' = 5120
# INFO: all meta  9: id=(2063696d:mic ) type='i' = 16384
# INFO: all meta 10: id=(20636361:acc ) type='i' = 5120
# INFO: all meta 11: id=(20626d61:amb ) type='i' = 2560
# INFO: all meta 12: id=(20726162:bar ) type='i' = 71
# INFO: all meta 13: id=(706d6f63:comp) type='i' = 0
# INFO: all meta 14: id=(736c6164:dals) type='i' = 30
# INFO: all meta 15: id=(206d7568:hum ) type='i' = 10
# INFO: all meta 16: id=(72776f70:powr) type='i' = 0
# INFO: stream 0: id=0 (2063696d:mic ) -> ref(0)
# INFO: stream 1: id=1 (3262736d:msb2) -> ref(1)
# INFO: stream 2: id=2 (20676174:tag ) -> ref(2)
# INFO: stream 3: id=3 (6b636c63:clck) -> ref(3)
# INFO: stream 4: id=4 (20737067:gps ) -> ref(4)
# INFO: read UWAR data
# INFO: ---- calibration ----
# INFO: board nr            = f017 (61463)
# INFO: hardware version nr = 2
# INFO: major soft version  = 1
# INFO: minor soft version  = 1
# INFO: calibration word 1  = 39064
# INFO: calibration word 2  = 31260
# INFO: calibration word 3  = 36959
# INFO: calibration word 4  = 42677
# INFO: calibration val c1  = 19532
# INFO: calibration val c2  = 2037
# INFO: calibration val c3  = 666
# INFO: calibration val c4  = 577
# INFO: calibration val c5  = 488
# INFO: calibration val c6  = 28
# INFO: ------ schedule -----
# INFO: interrupt count    = 5120
# INFO: mic hz              = 16384
# INFO: accelerometer hz    = 512.0
# INFO: amb light hz        = 256.0
# INFO: barometer hz        = 7.1
# INFO: compass hz          = 0.0
# INFO: dals hz            = 3.0
# INFO: humidity hz        = 1.0
# INFO: power              = 0
# INFO: read UWAR v2 data
# INFO: read 463037 packets
# INFO: time from:  31.11.1969 16:00:28.949999
# INFO: time to:    31.11.1969 16:08:01.166999
INFO: statistics packets
      231555  msb2 (missed 5 packets (0.0022%))
      231478  mic  (missed 81 packets (0.0350%))
          4  corrupt
      463037  total
</pre>
The cleaned file should now show no corrupt packets:
<pre>
uwar_info.exe log-1969-11-31-16.00_clean.dat
# INFO: open file log-1969-11-31-16.00_clean.dat
# INFO: read UWAR v2 header
# INFO: all meta  0: id=(726e6462:bdnr) type='i' = 61463
# INFO: all meta  1: id=(72657668:hver) type='i' = 2
# INFO: all meta  2: id=(616d7773:swma) type='i' = 1
# INFO: all meta  3: id=(696d7773:swmi) type='i' = 1
# INFO: all meta  4: id=(31306477:wd01) type='i' = 39064
# INFO: all meta  5: id=(32306477:wd02) type='i' = 31260
# INFO: all meta  6: id=(33306477:wd03) type='i' = 36959
# INFO: all meta  7: id=(34306477:wd04) type='i' = 42677
# INFO: all meta  8: id=(72746e69:intr) type='i' = 5120
# INFO: all meta  9: id=(2063696d:mic ) type='i' = 16384
# INFO: all meta 10: id=(20636361:acc ) type='i' = 5120
# INFO: all meta 11: id=(20626d61:amb ) type='i' = 2560
# INFO: all meta 12: id=(20726162:bar ) type='i' = 71
# INFO: all meta 13: id=(706d6f63:comp) type='i' = 0
# INFO: all meta 14: id=(736c6164:dals) type='i' = 30
# INFO: all meta 15: id=(206d7568:hum ) type='i' = 10
# INFO: all meta 16: id=(72776f70:powr) type='i' = 0
# INFO: stream 0: id=0 (2063696d:mic ) -> ref(0)
# INFO: stream 1: id=1 (3262736d:msb2) -> ref(1)
# INFO: stream 2: id=2 (20676174:tag ) -> ref(2)
# INFO: stream 3: id=3 (6b636c63:clck) -> ref(3)
# INFO: stream 4: id=4 (20737067:gps ) -> ref(4)
# INFO: read UWAR data
# INFO: ---- calibration ----
# INFO: board nr            = f017 (61463)
# INFO: hardware version nr = 2
# INFO: major soft version  = 1
# INFO: minor soft version  = 1
# INFO: calibration word 1  = 39064
# INFO: calibration word 2  = 31260
# INFO: calibration word 3  = 36959
# INFO: calibration word 4  = 42677
# INFO: calibration val c1  = 19532
# INFO: calibration val c2  = 2037
# INFO: calibration val c3  = 666
# INFO: calibration val c4  = 577
# INFO: calibration val c5  = 488
# INFO: calibration val c6  = 28
# INFO: ------ schedule -----
# INFO: interrupt count    = 5120
# INFO: mic hz              = 16384
# INFO: accelerometer hz    = 512.0
# INFO: amb light hz        = 256.0
# INFO: barometer hz        = 7.1
# INFO: compass hz          = 0.0
# INFO: dals hz            = 3.0
# INFO: humidity hz        = 1.0
# INFO: power              = 0
# INFO: read UWAR v2 data
# INFO: read 463033 packets
# INFO: time from:  31.11.1969 16:00:28.949999
# INFO: time to:    31.11.1969 16:08:01.167000
INFO: statistics packets
      231555  msb2 (missed 5 packets (0.0022%))
      231478  mic  (missed 81 packets (0.0350%))
      463033  total
</pre>
</ul>
== uwarToMatlab [replaced by parse_uwar_dir and completeDataFn] ==
* <tt>msp/uwar-stats/uwarToMatlab.cpp</tt>
* not available for xscale
* [{{SERVER}}/files/uwar/tools_linx86/uwarToMatlab.tgz linux download (x86), ~8.2MB]
* [{{SERVER}}/files/uwar/tools_win32/uwarToMatlab.zip windows download (.zip), ~6MB]
This program can be used to process and combine multiple .uwar files into more usable output formats. You should only pass in .uwar files that belong to a single recording run from a single device (you should only try to combine files that are uninterrupted recordings). The program will sort your input files for you; however, if it is unable to find an ordering without significant gaps it will reject the sequence. You can use the '-maxGap_sec' argument to increase the allowable gap.<br>
<b><i>Note these files contain the paths of the source files used to create them (e.g. 'C:\someData\log.uwar') so avoid using anything embarrassing or anything that causes any privacy concerns.</i></b>
<ul>
;completeSensorAudio.wav: is a wav file created from the sensor board audio file with any missing samples filled in
;dataOut.mat: is a Matlab file containing a struct with the sensor data in a timestamp, data format
;completeData.mat: is a Matlab file containing a struct with the sensor data placed into fixed sample rate arrays. For example a sensor with a 100Hz sample would have 1 second of data represented by a 100 element array, and a corresponding time array with the start/end time: [0 1.0] (seconds).
;sirfBinaryOutput.bin: is a binary file containing the serial stream from the sirf GPS unit
</ul>
=== Matlab notes ===
Since uwarToMatlab uses Matlab to output its .mat files you need several .dlls to run the program. If you have the Matlab Component Runtime installed you don't need the associated .dlls in the .zip file (as they will have been added to your path during the MCR install process). If you are using Linux you should have the MCR installed or you can use the libraries included in the .tgz. Note that you may need to set your LD_LIBRARY_PATH to include the directory with the extracted .so files.
=== usage ===
<ul>
==== Typical Usage ====
To combine two .uwar files and dump them into another target directory:
<pre>
uwarToMatlab log-0001.uwar log-0002.uwar -outputdir "C:\temp\target"
</pre>
Remember, uwarToMatlab will only try to combine the .uwar files you give it. If it believes the files shouldn't go together it will exit, if you are missing any files it will either quit or there will be large gaps in the data, and if you run it on multiple .uwar files with the same output directory it will erase the files it creates, not append to them.
==== Available Commands ====
<pre>
uwarToMatlab [-options] <input file(s)>
  [-outputdir N]
      Create all output in directory N rather than the inferred directory
      (Default behavior is to figure out the path from the first input file)
  [-NoDataOut]
      Do not create a dataOut mat file
  [-NoCompleteData]
      Do not create a completeData mat file
  [-NoWavFile]
      Do not write a completeSensorAudio.wav file
  [-buffSize N]
      Read buffer size in bytes (default 200k)
  [-maxGap_sec N]
      Maximum gap between .uwar files (in seconds, default 50*60)
  [-disableChecks]
      Disable header comparisons for uwar files, may result in corrupt files
  [-ignoreRollover]
      Ignore roll over events that appear to be errors and continue
  [-printProgress]
      Print progress of the parser every printProgressMax packets (default 10,000)
  [-printProgressMax N]
      Print progress of the parser every N packets (default 10,000)
  [-blacklist file.txt]
      List of uwar packet numbers to ignore
  [-write_timestampFile]
      Write the uwar packet #, time_counter, and interrupt_counter to a file 'timestamp.txt'
  [-integerOutput]
      Write the integer output from the msb2 parser instead of the double output
  [-write_summaryFile]
      Write some summary information to an <input file>.stats file
</pre>
</ul>
=== example output ===
<ul>
There are a couple of notes about the output of the program:
* Text output:
** #INFO lines are supplemental information
** #WARNING lines contain non-critical errors
** #ERROR lines contain information about fatal processing errors.
* Warnings about missing samples are ok so long as the table at the end shows relatively little lost data. uwarToMatlab currently doesn't have any threshold for rejecting files containing too many missing samples.
* When a file error is encountered the program enters a "UWAR Error mode" where it searches the .uwar datastream for a new packet. This should typically only require a 1 byte search, resulting in 0 bytes skipped. However, it is possible for a file to be severly corrupted and this search may skip a large number of unrecoverable samples.
* The table printed at the end of the run contains summary information about the samples found, be sure to check that the file contains the appropriate samples and is for the most part correct.
<pre>
C:\Test Data\home>uwarToMatlab log-0001.uwar log-0002.uwar -outputdir "C:\Test Data\home"
# INFO: Data will be created in directory: C:\Test Data\home\
# INFO: open file [ 1] 'log-0001.uwar'
# INFO: open file [ 2] 'log-0002.uwar'
# INFO: MSB2 sensor schedule:
# INFO:  msb/audio            163840
# INFO:  msb/bar_press            71
# INFO:  msb/compass                0
# INFO:  msb/hf_vis_light        1280
# INFO:  msb/humidity              10
# INFO:  msb/ir_light              30
# INFO:  msb/amb_light            30
# INFO:  msb/temp                  71
# INFO:  msb/temp_sensiron        10
# INFO:  msb/vis_light            30
# INFO:  msb/x_accel            5120
# INFO:  msb/y_accel            5120
# INFO:  msb/z_accel            5120
# INFO:  msb/mag_accel          5120
# INFO:  interrupt count        5120
# INFO: MSB2 Calibration data:
# INFO:  msb2 bar C1            22336
# INFO:  msb2 bar C2            1830
# INFO:  msb2 bar C3              712
# INFO:  msb2 bar C4              570
# INFO:  msb2 bar C5              562
# INFO:  msb2 bar C6              25
# INFO:  msb2 board number    0x0000f103
# INFO:  msb2 hardware ver    0x00000003
# INFO:  msb2 major ver      0x00000001
# INFO:  msb2 minor ver      0x00000005
# INFO:  msb2 micr gain      0x00000000
# INFO: LSB sensor schedule:
# INFO:  lsb/x_gyro              640
# INFO:  lsb/y_gyro              640
# INFO:  lsb/z_gyro              640
# INFO:  lsb/x_gyroTemp          640
# INFO:  lsb/y_gyroTemp          640
# INFO:  lsb/z_gyroTemp          640
# INFO:  lsb/x_mag                320
# INFO:  lsb/y_mag                320
# INFO:  lsb/z_mag                320
# INFO:  interrupt count        5120
# INFO: LSB Calibration data:
# INFO:  lsb word 1            44672
# INFO:  lsb word 2            35993
# INFO:  lsb word 3            36508
# INFO:  lsb word 4            45606
# INFO:  lsb board number    0x00000001
# INFO:  lsb hardware ver    0x00000003
# INFO:  lsb major ver        0x00000001
# INFO:  lsb minor ver        0x00000005
# INFO: [ 1] starts at Wed Dec 31 16:04:46 1969
# INFO: [ 2] starts at Wed Dec 31 16:09:06 1969
# INFO: Sorted order:
# INFO: [ 1] starts at Wed Dec 31 16:04:46 1969
# INFO: [ 2] starts at Wed Dec 31 16:09:06 1969  (  0 hours  4 mins 20.00 seconds from previous file)
# INFO: Sequence appears to be valid
# INFO: Parsing files to determine Matlab array dimensions and to find any errors:
# INFO:  File [ 1]
# INFO:  File [ 2]
# WARNING: file index 12121092, MSB2 Packet CRC invalid
# WARNING: file index 12124690, UWAR Packet's ID is invalid
# WARNING: UWAR Error mode, looking for new uwar packet, start pos: 12124691
# WARNING: UWAR Error mode, found new uwar packet, at pos: 12124691 (0 bytes skipped)
# INFO: Scanning complete
# INFO: File [ 1], MSB2 spans  134923 to  267990    259.90 seconds (  0 hours  4 mins 19.90 seconds)
# INFO: File [ 2], MSB2 spans  267990 to  401053    259.89 seconds (  0 hours  4 mins 19.89 seconds)
# INFO: File [ 1], LSB  spans  133189 to  266255    259.89 seconds (  0 hours  4 mins 19.89 seconds)
# INFO: File [ 2], LSB  spans  266256 to  399317    259.88 seconds (  0 hours  4 mins 19.88 seconds)
# INFO:    266130 MSB2 interrupt counts
# INFO:    266127 LSB  interrupt counts
# INFO: MSB2    519.79 seconds (  0 hours  8 mins 39.79 seconds)
# INFO: LSB    519.78 seconds (  0 hours  8 mins 39.78 seconds)
# INFO:      Sensor Name  SampleRate ExpectedSamples  FoundSamples    Diff  Percent Difference
# INFO:        msb/audio  16384.00Hz        8516160      8513377  -2783  ( 99.9673%)
# INFO:    msb/bar_press      7.10Hz            3690          3690      +0  (100.0000%)
# INFO:      msb/compass      0.00Hz              0            0      +0  (  0.0000%)
# INFO:  msb/hf_vis_light    128.00Hz          66531        66526      -5  ( 99.9925%)
# INFO:      msb/humidity      1.00Hz            519          519      +0  (100.0000%)
# INFO:      msb/ir_light      3.00Hz            1559          1559      +0  (100.0000%)
# INFO:    msb/amb_light      3.00Hz            1559          1559      +0  (100.0000%)
# INFO:          msb/temp      7.10Hz            3691          3691      +0  (100.0000%)
# INFO: msb/temp_sensiron      1.00Hz            519          519      +0  (100.0000%)
# INFO:    msb/vis_light      3.00Hz            1559          1559      +0  (100.0000%)
# INFO:      msb/x_accel    512.00Hz          266130        266101    -29  ( 99.9891%)
# INFO:      msb/y_accel    512.00Hz          266130        266101    -29  ( 99.9891%)
# INFO:      msb/z_accel    512.00Hz          266130        266101    -29  ( 99.9891%)
# INFO:        lsb/x_gyro    64.00Hz          33264        33262      -2  ( 99.9940%)
# INFO:        lsb/y_gyro    64.00Hz          33264        33262      -2  ( 99.9940%)
# INFO:        lsb/z_gyro    64.00Hz          33264        33262      -2  ( 99.9940%)
# INFO:    lsb/x_gyroTemp    64.00Hz          33264        33262      -2  ( 99.9940%)
# INFO:    lsb/y_gyroTemp    64.00Hz          33264        33262      -2  ( 99.9940%)
# INFO:    lsb/z_gyroTemp    64.00Hz          33264        33262      -2  ( 99.9940%)
# INFO:        lsb/x_mag    32.00Hz          16632        16632      +0  (100.0000%)
# INFO:        lsb/y_mag    32.00Hz          16631        16631      +0  (100.0000%)
# INFO:        lsb/z_mag    32.00Hz          16631        16631      +0  (100.0000%)
# INFO: Creating output files
# INFO: Data will be written to: 'C:\Test Data\home\'
# INFO: Opening .wav file:            'C:\Test Data\home\completeSensorAudio.wav'
# INFO: Opening dataOut.mat file:      'C:\Test Data\home\dataOut.mat'
# INFO: Opening completeData.mat file: 'C:\Test Data\home\completeData.mat'
# WARNING: Magnetometer samples are unequal, enforcing minimum size of 16631
# INFO:  File [ 1]
# WARNING: Packet  3155, WAV, Filling  31 missing samples at pos  43424
# WARNING: Packet  3160, WAV, Filling  2 missing samples at pos  43551
# WARNING: Packet  20310, WAV, Filling  32 missing samples at pos  279359
# WARNING: Packet  20312, WAV, Filling  2 missing samples at pos  279423
# WARNING: Packet  40668, WAV, Filling  32 missing samples at pos  559360
# WARNING: Packet  46767, WAV, Filling  33 missing samples at pos  643263
# WARNING: Packet  54531, WAV, Filling  33 missing samples at pos  750079
# WARNING: Packet  76395, WAV, Filling  33 missing samples at pos 1050719
# WARNING: Packet  86143, WAV, Filling  33 missing samples at pos 1184767
# WARNING: Packet  91594, WAV, Filling  34 missing samples at pos 1259807
# WARNING: Packet  91603, WAV, Filling  31 missing samples at pos 1259968
# WARNING: Packet  91605, WAV, Filling  2 missing samples at pos 1260031
# WARNING: Packet  93977, WAV, Filling  33 missing samples at pos 1292575
# WARNING: Packet 100873, WAV, Filling  33 missing samples at pos 1387423
# WARNING: Packet 100875, WAV, Filling  2 missing samples at pos 1387487
# WARNING: Packet 103887, WAV, Filling  32 missing samples at pos 1428895
# WARNING: Packet 103891, WAV, Filling  2 missing samples at pos 1428991
# WARNING: Packet 119670, WAV, Filling  33 missing samples at pos 1645983
# WARNING: Packet 120266, WAV, Filling  33 missing samples at pos 1654175
# WARNING: Packet 125648, WAV, Filling  32 missing samples at pos 1728223
# WARNING: Packet 125651, WAV, Filling  2 missing samples at pos 1728287
# WARNING: Packet 134129, WAV, Filling  32 missing samples at pos 1844895
# WARNING: Packet 134131, WAV, Filling  2 missing samples at pos 1844959
# WARNING: Packet 141766, WAV, Filling  33 missing samples at pos 1949919
# WARNING: Packet 144164, WAV, Filling  33 missing samples at pos 1982847
# WARNING: Packet 144172, WAV, Filling  32 missing samples at pos 1983007
# WARNING: Packet 144176, WAV, Filling  2 missing samples at pos 1983103
# WARNING: Packet 153217, WAV, Filling  32 missing samples at pos 2107519
# WARNING: Packet 153221, WAV, Filling  2 missing samples at pos 2107615
# WARNING: Packet 155041, WAV, Filling  33 missing samples at pos 2132607
# WARNING: Packet 178163, WAV, Filling  33 missing samples at pos 2450624
# WARNING: Packet 195222, WAV, Filling  32 missing samples at pos 2685248
# WARNING: Packet 195236, WAV, Filling  2 missing samples at pos 2685439
# WARNING: Packet 214551, WAV, Filling  33 missing samples at pos 2951008
# WARNING: Packet 232503, WAV, Filling  33 missing samples at pos 3197919
# WARNING: Packet 244593, WAV, Filling  33 missing samples at pos 3364223
# WARNING: Packet 256193, WAV, Filling  33 missing samples at pos 3523808
# WARNING: Packet 256206, WAV, Filling  32 missing samples at pos 3523968
# WARNING: Packet 300972, WAV, Filling  33 missing samples at pos 4139679
# WARNING: Packet 300974, WAV, Filling  2 missing samples at pos 4139743
# INFO:  File [ 2]
# WARNING: Packet  5220, WAV, Filling  31 missing samples at pos 4329920
# WARNING: Packet  5223, WAV, Filling  2 missing samples at pos 4329983
# WARNING: Packet  12590, WAV, Filling  33 missing samples at pos 4431327
# WARNING: Packet  26143, WAV, Filling  34 missing samples at pos 4617791
# WARNING: Packet  37454, WAV, Filling  33 missing samples at pos 4773279
# WARNING: Packet  63151, WAV, Filling  33 missing samples at pos 5126687
# WARNING: Packet  67405, WAV, Filling  33 missing samples at pos 5185183
# WARNING: Packet  80853, WAV, Filling  33 missing samples at pos 5370175
# WARNING: Packet  86136, WAV, Filling  33 missing samples at pos 5442751
# WARNING: Packet  93745, WAV, Filling  33 missing samples at pos 5547455
# WARNING: Packet 102842, WAV, Filling  34 missing samples at pos 5672607
# WARNING: Packet 110762, WAV, Filling  34 missing samples at pos 5781567
# WARNING: Packet 111883, WAV, Filling  33 missing samples at pos 5796991
# WARNING: Packet 124512, WAV, Filling  31 missing samples at pos 5970656
# WARNING: Packet 124518, WAV, Filling  2 missing samples at pos 5970783
# WARNING: Packet 129266, WAV, Filling  33 missing samples at pos 6036031
# WARNING: Packet 148985, WAV, Filling  33 missing samples at pos 6307199
# WARNING: Packet 159037, WAV, Filling  32 missing samples at pos 6445472
# WARNING: Packet 168461, WAV, Filling  34 missing samples at pos 6575071
# WARNING: Packet 178658, WAV, Filling  33 missing samples at pos 6715327
# WARNING: Packet 196278, WAV, Filling  32 missing samples at pos 6957664
# WARNING: Packet 202828, WAV, Filling  31 missing samples at pos 7047776
# WARNING: Packet 202831, WAV, Filling  2 missing samples at pos 7047839
# WARNING: Packet 203008, WAV, Filling  33 missing samples at pos 7050239
# WARNING: Packet 228148, WAV, Filling  33 missing samples at pos 7395935
# WARNING: file index 12121092, MSB2 Packet CRC invalid
# WARNING: file index 12124690, UWAR Packet's ID is invalid
# WARNING: UWAR Error mode, looking for new uwar packet, start pos: 12124691
# WARNING: UWAR Error mode, found new uwar packet, at pos: 12124691 (0 bytes skipped)
# WARNING: Packet 244197, WAV, Filling 961 missing samples at pos 7616767
# WARNING: Packet 244198, Filling  29 missing samples for field accel
# WARNING: Packet 244198, Filling  6 missing samples for field visFreqLight
# WARNING: Packet 244198, Filling  29 missing samples for field accelY
# WARNING: Packet 244198, Filling  29 missing samples for field accelZ
# WARNING: Packet 244201, WAV, Filling  2 missing samples at pos 7617791
# WARNING: Packet 244212, Filling  3 missing samples for field gyro
# WARNING: Packet 244212, Filling  3 missing samples for field gyroTemp
# WARNING: Packet 244212, Filling  3 missing samples for field gyroY
# WARNING: Packet 244212, Filling  3 missing samples for field gyroZ
# WARNING: Packet 244212, Filling  3 missing samples for field gyroTempY
# WARNING: Packet 244212, Filling  3 missing samples for field gyroTempZ
# WARNING: Packet 254806, WAV, Filling  33 missing samples at pos 7763616
# WARNING: Packet 258487, WAV, Filling  34 missing samples at pos 7814239
# WARNING: Packet 264336, WAV, Filling  33 missing samples at pos 7894687
# WARNING: Packet 281685, WAV, Filling  32 missing samples at pos 8133247
# WARNING: Packet 281688, WAV, Filling  2 missing samples at pos 8133311
# WARNING: Packet 297231, WAV, Filling  32 missing samples at pos 8347072
# WARNING: Packet 304508, WAV, Filling  32 missing samples at pos 8447167
# WARNING: Packet 304512, WAV, Filling  2 missing samples at pos 8447263
# WARNING: completeData- Packet 309523, magnetometer artificial ceiling reached, sample dropped, for sensor mag
# INFO: Summary:
# INFO: File [ 1], MSB2 spans  134923 to  267990    259.90 seconds (  0 hours  4 mins 19.90 seconds)
# INFO: File [ 2], MSB2 spans  267990 to  401053    259.89 seconds (  0 hours  4 mins 19.89 seconds)
# INFO: File [ 1], LSB  spans  133189 to  266255    259.89 seconds (  0 hours  4 mins 19.89 seconds)
# INFO: File [ 2], LSB  spans  266256 to  399317    259.89 seconds (  0 hours  4 mins 19.89 seconds)
# INFO:    266130 MSB2 interrupt counts
# INFO:    266127 LSB  interrupt counts
# INFO: MSB2    519.79 seconds (  0 hours  8 mins 39.79 seconds)
# INFO: LSB    519.78 seconds (  0 hours  8 mins 39.78 seconds)
# INFO:      Sensor Name  SampleRate ExpectedSamples  FoundSamples    Diff  Percent Difference
# INFO:        msb/audio  16384.00Hz        8516160      8513377  -2783  ( 99.9673%)
# INFO:    msb/bar_press      7.10Hz            3690          3690      +0  (100.0000%)
# INFO:      msb/compass      0.00Hz              0            0      +0  (  0.0000%)
# INFO:  msb/hf_vis_light    128.00Hz          66531        66526      -5  ( 99.9925%)
# INFO:      msb/humidity      1.00Hz            519          519      +0  (100.0000%)
# INFO:      msb/ir_light      3.00Hz            1559          1559      +0  (100.0000%)
# INFO:    msb/amb_light      3.00Hz            1559          1559      +0  (100.0000%)
# INFO:          msb/temp      7.10Hz            3691          3691      +0  (100.0000%)
# INFO: msb/temp_sensiron      1.00Hz            519          519      +0  (100.0000%)
# INFO:    msb/vis_light      3.00Hz            1559          1559      +0  (100.0000%)
# INFO:      msb/x_accel    512.00Hz          266130        266101    -29  ( 99.9891%)
# INFO:      msb/y_accel    512.00Hz          266130        266101    -29  ( 99.9891%)
# INFO:      msb/z_accel    512.00Hz          266130        266101    -29  ( 99.9891%)
# INFO:        lsb/x_gyro    64.00Hz          33264        33262      -2  ( 99.9940%)
# INFO:        lsb/y_gyro    64.00Hz          33264        33262      -2  ( 99.9940%)
# INFO:        lsb/z_gyro    64.00Hz          33264        33262      -2  ( 99.9940%)
# INFO:    lsb/x_gyroTemp    64.00Hz          33264        33262      -2  ( 99.9940%)
# INFO:    lsb/y_gyroTemp    64.00Hz          33264        33262      -2  ( 99.9940%)
# INFO:    lsb/z_gyroTemp    64.00Hz          33264        33262      -2  ( 99.9940%)
# INFO:        lsb/x_mag    32.00Hz          16632        16632      +0  (100.0000%)
# INFO:        lsb/y_mag    32.00Hz          16631        16631      +0  (100.0000%)
# INFO:        lsb/z_mag    32.00Hz          16631        16631      +0  (100.0000%)
# INFO: Closing completeSensorAudio.wav
# INFO: Closing dataOut.mat
# INFO: Closing completeData.mat
C:\Test Data\home>
</pre>
</pre>
</ul>
</ul>

Latest revision as of 23:40, 24 January 2009

Overview

This document lists the various tools for working with uwar files and related data. An example .uwar file is hosted on this machine that you can use to test with. Most of the outputs on this page are from this file: example.uwar.


parse_uwar_dir

parse_uwar_dir replaces uwarToMatlab and converts data into binary data blobs called sensorName.bin and metaData files called sensorName.meta. The meta data files are of the format: 

"NumSamples",47013
"SampleRate",512.000000
"Name","accelMag"
"DataType",1
"firstTimestamp",0.000000
"lastTimestamp",91.820313

Where DataType is defined as: 

                case 1
                    DataType_string = 'single';
                    DataType_size   = 4;
                case 2
                    DataType_string = 'double';
                    DataType_size   = 8;
                case 3
                    DataType_string = 'int8';
                    DataType_size   = 1;
                case 4
                    DataType_string = 'uint8';
                    DataType_size   = 1;
                case 5
                    DataType_string = 'int16';
                    DataType_size   = 2;
                case 6
                    DataType_string = 'uint16';
                    DataType_size   = 2;
                case 7
                    DataType_string = 'int32';
                    DataType_size   = 4;
                case 8
                    DataType_string = 'uint32';
                    DataType_size   = 4;
                case 9
                    DataType_string = 'int64';
                    DataType_size   = 8;
                case 10
                    DataType_string = 'uint64';
                    DataType_size   = 8;

usage

    parse_uwar_dir [-options] <input file(s)>

    Available Commands

    parse_uwar_dir [-options] <input file(s)>
   [-outputdir N]
       Create all output in directory N rather than the inferred directory
       (Default behavior is to figure out the path from the first input file)
   [-maxGap_inSeconds N]
       Maximum gap between .uwar files (in seconds, default 2*60)
   [-ignoreRollover]
       Ignore roll over events that appear to be errors and continue
   [-NoWriteImageSubDir]
       Don't write images to 'images' sub directory in the target dir (default on)
   [-writeTraceFiles]
       Creates <filename>.processing files to UWAR which are open
       this provides some feedback should the program crash (default off)
   [-writeTraceLog]
       Creates conversion.log csv file to output directory, detailing which files
       have been processed, and where their output went (default off)

   [-scandir N]
       Scan the directory for .uwar files to process, arrange them into sessions and convert the data

example output

    parse_uwar_dir example.uwar
# INFO: Data will be created in directory: .\

# INFO: ------------------
# INFO: Scanning files to remove obviously bad files
# INFO: File [  1]            Opened, "example.uwar"

# INFO: ------------------
# INFO: Processing Session  1 of  1
# INFO: Creating output directory:
# INFO: ".\part_00"
# INFO: Session  1, Processing File   1 of   1 (total)
# WARNING: Packet  14707, Filling  16 missing samples for field msb2/audio
# WARNING: Packet  17420, Filling  16 missing samples for field msb2/audio
# WARNING: Packet  19526, Filling  16 missing samples for field msb2/audio
# WARNING: Packet  20792, Filling  16 missing samples for field msb2/audio
# WARNING: Packet  30340, Filling  16 missing samples for field msb2/audio
# WARNING: Packet  37818, Filling  17 missing samples for field msb2/audio
# INFO: Session  1 of  1 completed
# INFO: ------------------


uwar_tag_dump

uwar_tag_dump allows you to dump text tags from a TAG stream to output text file.

usage

    uwar_tagdump [-options] <input file(s)> -out <output file>

    Available Commands

    uwar_tagdump [-options] <input file(s)> -out <output file>
   [-maxGap_inSeconds N]
       Maximum gap between .uwar files (in seconds, default 2*60)

   [-scandir N]
       Scan the directory for .uwar files to process, arrange them into sessions and convert the data
The output format is:
 UWAR Timestamp, MSB2 Timestamp, TAG string

example output

    uwar_tagdump example.uwar -out test.txt
# INFO: Output TAG file will be: test.txt

# INFO: ------------------
# INFO: Scanning files to remove obviously bad files
# INFO: File [  1]            Opened, "example.uwar"
# INFO: Reading TAGs from UWAR Files...
# INFO: Reading File   1 of   1, example.uwar
# INFO: Done reading tags


uwar_modify

uwar_modify will change the metaValues in a UWAR header, the input file is used as the output.

usage

    uwar-modify -streamID N -metaID N -metaValue N <input/output file>

    Available Commands

    uwar-modify [-options]  <input file>
   uwar-modify will change the metaValues in a UWAR header, the input file is used as the output
   [-streamID N]
       Specify the stream number (which start at 0) you want to change
       If the stream number is not found an error will occur
   [-metaID N]
       Meta ID we wish to change (IDs start at 0)
       If the metaID number is not found an error will occur
   [-metaValue N]
       New value to be written to the metaID (currently only type 'i' is supported)

example output

    To change the schedule sample rate in a .uwar file we can use uwar-modify. 
    uwar_modify -streamID -1 -metaID 11 -metaValue 71 "example.uwar
# INFO: Stream ID to be modified: -1
# INFO: Meta ID to be modified:   11
# INFO: Meta to be modified to:   71
# INFO: Header values before changes:
# INFO: File: "example.uwar"
# INFO: UWAR File Contains   5 streams
# INFO: Symbol ID = auto-generated ID number used inside each data packet
# INFO:        ID = UWAR_STREAM ID for identifying stream types
# INFO:
# INFO: Stream   0: Symbol ID=  0 (ID = 0x2063696D:'mic ') [MIC]
# INFO: Stream   1: Symbol ID=  1 (ID = 0x3262736D:'msb2') [MSB2]
# INFO: Stream   2: Symbol ID=  2 (ID = 0x20676174:'tag ') [TAG]
# INFO: Stream   3: Symbol ID=  3 (ID = 0x6B636C63:'clck') [CLOCK]
# INFO: Stream   4: Symbol ID=  4 (ID = 0x20737067:'gps ') [GPS]
# INFO:
# INFO: UWAR File Meta Data:
# INFO: uwar header meta   0/ 17: (ID = 0x726E6462:'bdnr') type='i' = 11
# INFO: uwar header meta   1/ 17: (ID = 0x72657668:'hver') type='i' = 2
# INFO: uwar header meta   2/ 17: (ID = 0x616D7773:'swma') type='i' = 1
# INFO: uwar header meta   3/ 17: (ID = 0x696D7773:'swmi') type='i' = 1
# INFO: uwar header meta   4/ 17: (ID = 0x31306477:'wd01') type='i' = 48224
# INFO: uwar header meta   5/ 17: (ID = 0x32306477:'wd02') type='i' = 47962
# INFO: uwar header meta   6/ 17: (ID = 0x33306477:'wd03') type='i' = 28705
# INFO: uwar header meta   7/ 17: (ID = 0x34306477:'wd04') type='i' = 44956
# INFO: uwar header meta   8/ 17: (ID = 0x72746E69:'intr') type='i' = 5120
# INFO: uwar header meta   9/ 17: (ID = 0x2063696D:'mic ') type='i' = 8192
# INFO: uwar header meta  10/ 17: (ID = 0x20636361:'acc ') type='i' = 5120
# INFO: uwar header meta  11/ 17: (ID = 0x20626D61:'amb ') type='i' = 2560
# INFO: uwar header meta  12/ 17: (ID = 0x20726162:'bar ') type='i' = 71
# INFO: uwar header meta  13/ 17: (ID = 0x706D6F63:'comp') type='i' = 0
# INFO: uwar header meta  14/ 17: (ID = 0x736C6164:'dals') type='i' = 30
# INFO: uwar header meta  15/ 17: (ID = 0x206D7568:'hum ') type='i' = 10
# INFO: uwar header meta  16/ 17: (ID = 0x72776F70:'powr') type='i' = 0
# INFO: Stream   0: Symbol ID=  0 (ID = 0x2063696D:'mic ') [MIC]
# INFO: Stream   1: Symbol ID=  1 (ID = 0x3262736D:'msb2') [MSB2]
# INFO: Stream   2: Symbol ID=  2 (ID = 0x20676174:'tag ') [TAG]
# INFO: Stream   3: Symbol ID=  3 (ID = 0x6B636C63:'clck') [CLOCK]
# INFO: Stream   4: Symbol ID=  4 (ID = 0x20737067:'gps ') [GPS]
# INFO: File: "example.uwar"
# INFO: UWAR File Contains   5 streams
# INFO: Symbol ID = auto-generated ID number used inside each data packet
# INFO:        ID = UWAR_STREAM ID for identifying stream types
# INFO:
# INFO: Stream   0: Symbol ID=  0 (ID = 0x2063696D:'mic ') [MIC]
# INFO: Stream   1: Symbol ID=  1 (ID = 0x3262736D:'msb2') [MSB2]
# INFO: Stream   2: Symbol ID=  2 (ID = 0x20676174:'tag ') [TAG]
# INFO: Stream   3: Symbol ID=  3 (ID = 0x6B636C63:'clck') [CLOCK]
# INFO: Stream   4: Symbol ID=  4 (ID = 0x20737067:'gps ') [GPS]
# INFO:
# INFO: UWAR File Meta Data:
# INFO: uwar header meta   0/ 17: (ID = 0x726E6462:'bdnr') type='i' = 11
# INFO: uwar header meta   1/ 17: (ID = 0x72657668:'hver') type='i' = 2
# INFO: uwar header meta   2/ 17: (ID = 0x616D7773:'swma') type='i' = 1
# INFO: uwar header meta   3/ 17: (ID = 0x696D7773:'swmi') type='i' = 1
# INFO: uwar header meta   4/ 17: (ID = 0x31306477:'wd01') type='i' = 48224
# INFO: uwar header meta   5/ 17: (ID = 0x32306477:'wd02') type='i' = 47962
# INFO: uwar header meta   6/ 17: (ID = 0x33306477:'wd03') type='i' = 28705
# INFO: uwar header meta   7/ 17: (ID = 0x34306477:'wd04') type='i' = 44956
# INFO: uwar header meta   8/ 17: (ID = 0x72746E69:'intr') type='i' = 5120
# INFO: uwar header meta   9/ 17: (ID = 0x2063696D:'mic ') type='i' = 8192
# INFO: uwar header meta  10/ 17: (ID = 0x20636361:'acc ') type='i' = 5120
# INFO: uwar header meta  11/ 17: (ID = 0x20626D61:'amb ') type='i' = 71
# INFO: uwar header meta  12/ 17: (ID = 0x20726162:'bar ') type='i' = 71
# INFO: uwar header meta  13/ 17: (ID = 0x706D6F63:'comp') type='i' = 0
# INFO: uwar header meta  14/ 17: (ID = 0x736C6164:'dals') type='i' = 30
# INFO: uwar header meta  15/ 17: (ID = 0x206D7568:'hum ') type='i' = 10
# INFO: uwar header meta  16/ 17: (ID = 0x72776F70:'powr') type='i' = 0
# INFO: Stream   0: Symbol ID=  0 (ID = 0x2063696D:'mic ') [MIC]
# INFO: Stream   1: Symbol ID=  1 (ID = 0x3262736D:'msb2') [MSB2]
# INFO: Stream   2: Symbol ID=  2 (ID = 0x20676174:'tag ') [TAG]
# INFO: Stream   3: Symbol ID=  3 (ID = 0x6B636C63:'clck') [CLOCK]
# INFO: Stream   4: Symbol ID=  4 (ID = 0x20737067:'gps ') [GPS]


uwar_cat

uwar_cat is a program to extract data from uwar files and dump them in a human readable format. The output format is something of the form: 

SENSORA      TIMESTAMP DATA ...
SENSORB      TIMESTAMP DATA ...
SENSORC      TIMESTAMP DATA ...

If you want to retrieve data from a specific sensor you can use the grep command.
Windows users: You can download a win32 version of grep from the gnuwin32 webpage. You will also need to grab some additional libraries or you can try this zipped version which should contain all the necessary libraries. Simply extract the zip to your Windows\system32 directory.

usage

    uwar_cat <filename>

example output

    uwar_cat example.uwar
# INFO: open file example.uwar
# INFO: read UWAR v2 header
# INFO: read UWAR v2 data
ACCEL        1160595736.915 -0.028381 0.013855 0.980713
ACCEL        1160595736.915 -0.027588 0.013733 0.981079
AMB-LIGHT    1160595736.915 16 9.602142
DALS         1160595736.915 671.500000 73.500000 219.288132
ACCEL        1160595736.915 -0.027222 0.014099 0.980042
ACCEL        1160595736.915 -0.027039 0.013428 0.981628
AMB-LIGHT    1160595736.915 16 9.661065
ACCEL        1160595736.915 -0.027893 0.014221 0.981079
ACCEL        1160595736.915 -0.027100 0.013916 0.981812
...

uwar-info

uwar_info will display the meta data stored in the uwar file. This information can be useful in determining what streams are inside a uwar file and what settings might be in use, for example you can retrieve the schedule and the calibration data from the sensor boards with this command.

usage

    uwar_info <filename>

example output

    uwar-info example.uwar 
# INFO: open file example.uwar
# INFO: read UWAR v2 header
# INFO: all meta  0: id=(726e6462:bdnr) type='i' = 11
# INFO: all meta  1: id=(72657668:hver) type='i' = 2
# INFO: all meta  2: id=(616d7773:swma) type='i' = 1
# INFO: all meta  3: id=(696d7773:swmi) type='i' = 1
# INFO: all meta  4: id=(31306477:wd01) type='i' = 48224
# INFO: all meta  5: id=(32306477:wd02) type='i' = 47962
# INFO: all meta  6: id=(33306477:wd03) type='i' = 28705
# INFO: all meta  7: id=(34306477:wd04) type='i' = 44956
# INFO: all meta  8: id=(72746e69:intr) type='i' = 5120
# INFO: all meta  9: id=(2063696d:mic ) type='i' = 8192
# INFO: all meta 10: id=(20636361:acc ) type='i' = 5120
# INFO: all meta 11: id=(20626d61:amb ) type='i' = 2560
# INFO: all meta 12: id=(20726162:bar ) type='i' = 71
# INFO: all meta 13: id=(706d6f63:comp) type='i' = 0
# INFO: all meta 14: id=(736c6164:dals) type='i' = 30
# INFO: all meta 15: id=(206d7568:hum ) type='i' = 10
# INFO: all meta 16: id=(72776f70:powr) type='i' = 0
# INFO: stream 0: id=0 (2063696d:mic ) -> ref(0)
# INFO: stream 1: id=1 (3262736d:msb2) -> ref(1)
# INFO: stream 2: id=2 (20676174:tag ) -> ref(2)
# INFO: stream 3: id=3 (6b636c63:clck) -> ref(3)
# INFO: stream 4: id=4 (20737067:gps ) -> ref(4)
# INFO: read UWAR data
# INFO: ---- calibration ----
# INFO: board nr            = b (11)
# INFO: hardware version nr = 2
# INFO: major soft version  = 1
# INFO: minor soft version  = 1
# INFO: calibration word 1  = 48224
# INFO: calibration word 2  = 47962
# INFO: calibration word 3  = 28705
# INFO: calibration word 4  = 44956
# INFO: calibration val c1  = 24112
# INFO: calibration val c2  = 2140
# INFO: calibration val c3  = 702
# INFO: calibration val c4  = 448
# INFO: calibration val c5  = 749
# INFO: calibration val c6  = 26
# INFO: ------ schedule -----
# INFO: interrupt count     = 5120
# INFO: mic hz              = 8192
# INFO: accelerometer hz    = 512.0
# INFO: amb light hz        = 256.0
# INFO: barometer hz        = 7.1
# INFO: compass hz          = 0.0
# INFO: dals hz             = 3.0
# INFO: humidity hz         = 1.0
# INFO: power               = 0
# INFO: read UWAR v2 data
# INFO: read 53899 packets           

# INFO: time from:   11.09.2006 12:42:16.914999
# INFO: time to:     11.09.2006 12:43:09.532000

INFO: statistics packets
       26953   msb2 (missed 0 packets (0.0000%))
       26946   mic  (missed 6 packets (0.0223%))
       53899   total


uwar_msb_dump

uwar_msb_dump is a program to dump the raw serial packets stored inside the uwar file to a single binary output file. This is useful for debugging problems with the MSB itself and analyzing the packets received by the msb-server.

usage

    uwar_msb_dump <input_uwar_filename> <output_binary_file>

example output

    uwar_msb_dump example.uwar example.bin
# INFO: open file example.uwar
# INFO: Output file example.bin
# INFO: read UWAR v2 header
# INFO: read UWAR v2 data


uwar-clean

uwar_clean removes corrupted packets from a .uwar file and stores the cleaned stream in a new file. Note, this program is based off of uwar-info so any file uwar-info can process it should be able to handle. It does NOT clean bad data inside a valid uwar packet.

usage

    uwar_clean <input_uwar_filename> <output_cleaned_uwar_filename>

    Note: Be careful not to specify the same input/output name as this will immediately erase your file. Remember if you say something like 'uwar_clean myVeryImportantData.uwar ./myVeryImportantData.uwar' you will likely lose your data!

example output

    uwar_clean.exe log-1969-11-31-16.00.dat log-1969-11-31-16.00_clean.dat

# INFO: open file log-1969-11-31-16.00.dat
# INFO: read UWAR v2 header
# INFO: all meta  0: id=(726e6462:bdnr) type='i' = 61463
# INFO: all meta  1: id=(72657668:hver) type='i' = 2
# INFO: all meta  2: id=(616d7773:swma) type='i' = 1
# INFO: all meta  3: id=(696d7773:swmi) type='i' = 1
# INFO: all meta  4: id=(31306477:wd01) type='i' = 39064
# INFO: all meta  5: id=(32306477:wd02) type='i' = 31260
# INFO: all meta  6: id=(33306477:wd03) type='i' = 36959
# INFO: all meta  7: id=(34306477:wd04) type='i' = 42677
# INFO: all meta  8: id=(72746e69:intr) type='i' = 5120
# INFO: all meta  9: id=(2063696d:mic ) type='i' = 16384
# INFO: all meta 10: id=(20636361:acc ) type='i' = 5120
# INFO: all meta 11: id=(20626d61:amb ) type='i' = 2560
# INFO: all meta 12: id=(20726162:bar ) type='i' = 71
# INFO: all meta 13: id=(706d6f63:comp) type='i' = 0
# INFO: all meta 14: id=(736c6164:dals) type='i' = 30
# INFO: all meta 15: id=(206d7568:hum ) type='i' = 10
# INFO: all meta 16: id=(72776f70:powr) type='i' = 0
# INFO: stream 0: id=0 (2063696d:mic ) -> ref(0)
# INFO: stream 1: id=1 (3262736d:msb2) -> ref(1)
# INFO: stream 2: id=2 (20676174:tag ) -> ref(2)
# INFO: stream 3: id=3 (6b636c63:clck) -> ref(3)
# INFO: stream 4: id=4 (20737067:gps ) -> ref(4)
# INFO: read UWAR data
# INFO: ---- calibration ----
# INFO: board nr            = f017 (61463)
# INFO: hardware version nr = 2
# INFO: major soft version  = 1
# INFO: minor soft version  = 1
# INFO: calibration word 1  = 39064
# INFO: calibration word 2  = 31260
# INFO: calibration word 3  = 36959
# INFO: calibration word 4  = 42677
# INFO: calibration val c1  = 19532
# INFO: calibration val c2  = 2037
# INFO: calibration val c3  = 666
# INFO: calibration val c4  = 577
# INFO: calibration val c5  = 488
# INFO: calibration val c6  = 28
# INFO: ------ schedule -----
# INFO: interrupt count     = 5120
# INFO: mic hz              = 16384
# INFO: accelerometer hz    = 512.0
# INFO: amb light hz        = 256.0
# INFO: barometer hz        = 7.1
# INFO: compass hz          = 0.0
# INFO: dals hz             = 3.0
# INFO: humidity hz         = 1.0
# INFO: power               = 0
# INFO: read UWAR v2 data
# INFO: read 463037 packets

# INFO: time from:   31.11.1969 16:00:28.949999
# INFO: time to:     31.11.1969 16:08:01.166999

INFO: statistics packets
      231555   msb2 (missed 5 packets (0.0022%))
      231478   mic  (missed 81 packets (0.0350%))
           4   corrupt
      463037   total

    The cleaned file should now show no corrupt packets: 

    uwar_info.exe log-1969-11-31-16.00_clean.dat

# INFO: open file log-1969-11-31-16.00_clean.dat
# INFO: read UWAR v2 header
# INFO: all meta  0: id=(726e6462:bdnr) type='i' = 61463
# INFO: all meta  1: id=(72657668:hver) type='i' = 2
# INFO: all meta  2: id=(616d7773:swma) type='i' = 1
# INFO: all meta  3: id=(696d7773:swmi) type='i' = 1
# INFO: all meta  4: id=(31306477:wd01) type='i' = 39064
# INFO: all meta  5: id=(32306477:wd02) type='i' = 31260
# INFO: all meta  6: id=(33306477:wd03) type='i' = 36959
# INFO: all meta  7: id=(34306477:wd04) type='i' = 42677
# INFO: all meta  8: id=(72746e69:intr) type='i' = 5120
# INFO: all meta  9: id=(2063696d:mic ) type='i' = 16384
# INFO: all meta 10: id=(20636361:acc ) type='i' = 5120
# INFO: all meta 11: id=(20626d61:amb ) type='i' = 2560
# INFO: all meta 12: id=(20726162:bar ) type='i' = 71
# INFO: all meta 13: id=(706d6f63:comp) type='i' = 0
# INFO: all meta 14: id=(736c6164:dals) type='i' = 30
# INFO: all meta 15: id=(206d7568:hum ) type='i' = 10
# INFO: all meta 16: id=(72776f70:powr) type='i' = 0
# INFO: stream 0: id=0 (2063696d:mic ) -> ref(0)
# INFO: stream 1: id=1 (3262736d:msb2) -> ref(1)
# INFO: stream 2: id=2 (20676174:tag ) -> ref(2)
# INFO: stream 3: id=3 (6b636c63:clck) -> ref(3)
# INFO: stream 4: id=4 (20737067:gps ) -> ref(4)
# INFO: read UWAR data
# INFO: ---- calibration ----
# INFO: board nr            = f017 (61463)
# INFO: hardware version nr = 2
# INFO: major soft version  = 1
# INFO: minor soft version  = 1
# INFO: calibration word 1  = 39064
# INFO: calibration word 2  = 31260
# INFO: calibration word 3  = 36959
# INFO: calibration word 4  = 42677
# INFO: calibration val c1  = 19532
# INFO: calibration val c2  = 2037
# INFO: calibration val c3  = 666
# INFO: calibration val c4  = 577
# INFO: calibration val c5  = 488
# INFO: calibration val c6  = 28
# INFO: ------ schedule -----
# INFO: interrupt count     = 5120
# INFO: mic hz              = 16384
# INFO: accelerometer hz    = 512.0
# INFO: amb light hz        = 256.0
# INFO: barometer hz        = 7.1
# INFO: compass hz          = 0.0
# INFO: dals hz             = 3.0
# INFO: humidity hz         = 1.0
# INFO: power               = 0
# INFO: read UWAR v2 data
# INFO: read 463033 packets

# INFO: time from:   31.11.1969 16:00:28.949999
# INFO: time to:     31.11.1969 16:08:01.167000

INFO: statistics packets
      231555   msb2 (missed 5 packets (0.0022%))
      231478   mic  (missed 81 packets (0.0350%))
      463033   total



uwarToMatlab [replaced by parse_uwar_dir and completeDataFn]

This program can be used to process and combine multiple .uwar files into more usable output formats. You should only pass in .uwar files that belong to a single recording run from a single device (you should only try to combine files that are uninterrupted recordings). The program will sort your input files for you; however, if it is unable to find an ordering without significant gaps it will reject the sequence. You can use the '-maxGap_sec' argument to increase the allowable gap.
Note these files contain the paths of the source files used to create them (e.g. 'C:\someData\log.uwar') so avoid using anything embarrassing or anything that causes any privacy concerns.

    completeSensorAudio.wav
    is a wav file created from the sensor board audio file with any missing samples filled in
    dataOut.mat
    is a Matlab file containing a struct with the sensor data in a timestamp, data format
    completeData.mat
    is a Matlab file containing a struct with the sensor data placed into fixed sample rate arrays. For example a sensor with a 100Hz sample would have 1 second of data represented by a 100 element array, and a corresponding time array with the start/end time: [0 1.0] (seconds).
    sirfBinaryOutput.bin
    is a binary file containing the serial stream from the sirf GPS unit

Matlab notes

Since uwarToMatlab uses Matlab to output its .mat files you need several .dlls to run the program. If you have the Matlab Component Runtime installed you don't need the associated .dlls in the .zip file (as they will have been added to your path during the MCR install process). If you are using Linux you should have the MCR installed or you can use the libraries included in the .tgz. Note that you may need to set your LD_LIBRARY_PATH to include the directory with the extracted .so files.

usage

    Typical Usage

    To combine two .uwar files and dump them into another target directory: 

    uwarToMatlab log-0001.uwar log-0002.uwar -outputdir "C:\temp\target"

    Remember, uwarToMatlab will only try to combine the .uwar files you give it. If it believes the files shouldn't go together it will exit, if you are missing any files it will either quit or there will be large gaps in the data, and if you run it on multiple .uwar files with the same output directory it will erase the files it creates, not append to them.

    Available Commands

    uwarToMatlab [-options] <input file(s)>
   [-outputdir N]
       Create all output in directory N rather than the inferred directory
       (Default behavior is to figure out the path from the first input file)
   [-NoDataOut]
       Do not create a dataOut mat file
   [-NoCompleteData]
       Do not create a completeData mat file
   [-NoWavFile]
       Do not write a completeSensorAudio.wav file
   [-buffSize N]
       Read buffer size in bytes (default 200k)
   [-maxGap_sec N]
       Maximum gap between .uwar files (in seconds, default 50*60)
   [-disableChecks]
       Disable header comparisons for uwar files, may result in corrupt files
   [-ignoreRollover]
       Ignore roll over events that appear to be errors and continue
   [-printProgress]
       Print progress of the parser every printProgressMax packets (default 10,000)
   [-printProgressMax N]
       Print progress of the parser every N packets (default 10,000)
   [-blacklist file.txt]
       List of uwar packet numbers to ignore
   [-write_timestampFile]
       Write the uwar packet #, time_counter, and interrupt_counter to a file 'timestamp.txt'
   [-integerOutput]
       Write the integer output from the msb2 parser instead of the double output
   [-write_summaryFile]
       Write some summary information to an <input file>.stats file

example output

    There are a couple of notes about the output of the program:
    • Text output:
      • #INFO lines are supplemental information
      • #WARNING lines contain non-critical errors
      • #ERROR lines contain information about fatal processing errors.
    • Warnings about missing samples are ok so long as the table at the end shows relatively little lost data. uwarToMatlab currently doesn't have any threshold for rejecting files containing too many missing samples.
    • When a file error is encountered the program enters a "UWAR Error mode" where it searches the .uwar datastream for a new packet. This should typically only require a 1 byte search, resulting in 0 bytes skipped. However, it is possible for a file to be severly corrupted and this search may skip a large number of unrecoverable samples.
    • The table printed at the end of the run contains summary information about the samples found, be sure to check that the file contains the appropriate samples and is for the most part correct.
    C:\Test Data\home>uwarToMatlab log-0001.uwar log-0002.uwar -outputdir "C:\Test Data\home"
# INFO: Data will be created in directory: C:\Test Data\home\
# INFO: open file [ 1] 'log-0001.uwar'
# INFO: open file [ 2] 'log-0002.uwar'
# INFO: MSB2 sensor schedule:
# INFO:  msb/audio             163840
# INFO:  msb/bar_press             71
# INFO:  msb/compass                0
# INFO:  msb/hf_vis_light        1280
# INFO:  msb/humidity              10
# INFO:  msb/ir_light              30
# INFO:  msb/amb_light             30
# INFO:  msb/temp                  71
# INFO:  msb/temp_sensiron         10
# INFO:  msb/vis_light             30
# INFO:  msb/x_accel             5120
# INFO:  msb/y_accel             5120
# INFO:  msb/z_accel             5120
# INFO:  msb/mag_accel           5120
# INFO:  interrupt count         5120
# INFO: MSB2 Calibration data:
# INFO:  msb2 bar C1            22336
# INFO:  msb2 bar C2             1830
# INFO:  msb2 bar C3              712
# INFO:  msb2 bar C4              570
# INFO:  msb2 bar C5              562
# INFO:  msb2 bar C6               25
# INFO:  msb2 board number    0x0000f103
# INFO:  msb2 hardware ver    0x00000003
# INFO:  msb2 major ver       0x00000001
# INFO:  msb2 minor ver       0x00000005
# INFO:  msb2 micr gain       0x00000000
# INFO: LSB sensor schedule:
# INFO:  lsb/x_gyro               640
# INFO:  lsb/y_gyro               640
# INFO:  lsb/z_gyro               640
# INFO:  lsb/x_gyroTemp           640
# INFO:  lsb/y_gyroTemp           640
# INFO:  lsb/z_gyroTemp           640
# INFO:  lsb/x_mag                320
# INFO:  lsb/y_mag                320
# INFO:  lsb/z_mag                320
# INFO:  interrupt count         5120
# INFO: LSB Calibration data:
# INFO:  lsb word 1             44672
# INFO:  lsb word 2             35993
# INFO:  lsb word 3             36508
# INFO:  lsb word 4             45606
# INFO:  lsb board number     0x00000001
# INFO:  lsb hardware ver     0x00000003
# INFO:  lsb major ver        0x00000001
# INFO:  lsb minor ver        0x00000005
# INFO: [ 1] starts at Wed Dec 31 16:04:46 1969
# INFO: [ 2] starts at Wed Dec 31 16:09:06 1969
# INFO: Sorted order:
# INFO: [ 1] starts at Wed Dec 31 16:04:46 1969
# INFO: [ 2] starts at Wed Dec 31 16:09:06 1969   (  0 hours  4 mins 20.00 seconds from previous file)


# INFO: Sequence appears to be valid
# INFO: Parsing files to determine Matlab array dimensions and to find any errors:
# INFO:   File [ 1]
# INFO:   File [ 2]
# WARNING: file index 12121092, MSB2 Packet CRC invalid
# WARNING: file index 12124690, UWAR Packet's ID is invalid
# WARNING: UWAR Error mode, looking for new uwar packet, start pos: 12124691
# WARNING: UWAR Error mode, found new uwar packet, at pos: 12124691 (0 bytes skipped)


# INFO: Scanning complete
# INFO: File [ 1], MSB2 spans  134923 to   267990     259.90 seconds (  0 hours  4 mins 19.90 seconds)
# INFO: File [ 2], MSB2 spans  267990 to   401053     259.89 seconds (  0 hours  4 mins 19.89 seconds)

# INFO: File [ 1], LSB  spans  133189 to   266255     259.89 seconds (  0 hours  4 mins 19.89 seconds)
# INFO: File [ 2], LSB  spans  266256 to   399317     259.88 seconds (  0 hours  4 mins 19.88 seconds)
# INFO:    266130 MSB2 interrupt counts
# INFO:    266127 LSB  interrupt counts
# INFO: MSB2    519.79 seconds (  0 hours  8 mins 39.79 seconds)
# INFO: LSB     519.78 seconds (  0 hours  8 mins 39.78 seconds)
# INFO:       Sensor Name  SampleRate ExpectedSamples  FoundSamples    Diff  Percent Difference
# INFO:         msb/audio  16384.00Hz         8516160       8513377   -2783   ( 99.9673%)
# INFO:     msb/bar_press      7.10Hz            3690          3690      +0   (100.0000%)
# INFO:       msb/compass      0.00Hz               0             0      +0   (  0.0000%)
# INFO:  msb/hf_vis_light    128.00Hz           66531         66526      -5   ( 99.9925%)
# INFO:      msb/humidity      1.00Hz             519           519      +0   (100.0000%)
# INFO:      msb/ir_light      3.00Hz            1559          1559      +0   (100.0000%)
# INFO:     msb/amb_light      3.00Hz            1559          1559      +0   (100.0000%)
# INFO:          msb/temp      7.10Hz            3691          3691      +0   (100.0000%)
# INFO: msb/temp_sensiron      1.00Hz             519           519      +0   (100.0000%)
# INFO:     msb/vis_light      3.00Hz            1559          1559      +0   (100.0000%)
# INFO:       msb/x_accel    512.00Hz          266130        266101     -29   ( 99.9891%)
# INFO:       msb/y_accel    512.00Hz          266130        266101     -29   ( 99.9891%)
# INFO:       msb/z_accel    512.00Hz          266130        266101     -29   ( 99.9891%)
# INFO:        lsb/x_gyro     64.00Hz           33264         33262      -2   ( 99.9940%)
# INFO:        lsb/y_gyro     64.00Hz           33264         33262      -2   ( 99.9940%)
# INFO:        lsb/z_gyro     64.00Hz           33264         33262      -2   ( 99.9940%)
# INFO:    lsb/x_gyroTemp     64.00Hz           33264         33262      -2   ( 99.9940%)
# INFO:    lsb/y_gyroTemp     64.00Hz           33264         33262      -2   ( 99.9940%)
# INFO:    lsb/z_gyroTemp     64.00Hz           33264         33262      -2   ( 99.9940%)
# INFO:         lsb/x_mag     32.00Hz           16632         16632      +0   (100.0000%)
# INFO:         lsb/y_mag     32.00Hz           16631         16631      +0   (100.0000%)
# INFO:         lsb/z_mag     32.00Hz           16631         16631      +0   (100.0000%)


# INFO: Creating output files
# INFO: Data will be written to: 'C:\Test Data\home\'
# INFO: Opening .wav file:             'C:\Test Data\home\completeSensorAudio.wav'
# INFO: Opening dataOut.mat file:      'C:\Test Data\home\dataOut.mat'
# INFO: Opening completeData.mat file: 'C:\Test Data\home\completeData.mat'
# WARNING: Magnetometer samples are unequal, enforcing minimum size of 16631
# INFO:   File [ 1]
# WARNING: Packet   3155, WAV, Filling  31 missing samples at pos   43424
# WARNING: Packet   3160, WAV, Filling   2 missing samples at pos   43551
# WARNING: Packet  20310, WAV, Filling  32 missing samples at pos  279359
# WARNING: Packet  20312, WAV, Filling   2 missing samples at pos  279423
# WARNING: Packet  40668, WAV, Filling  32 missing samples at pos  559360
# WARNING: Packet  46767, WAV, Filling  33 missing samples at pos  643263
# WARNING: Packet  54531, WAV, Filling  33 missing samples at pos  750079
# WARNING: Packet  76395, WAV, Filling  33 missing samples at pos 1050719
# WARNING: Packet  86143, WAV, Filling  33 missing samples at pos 1184767
# WARNING: Packet  91594, WAV, Filling  34 missing samples at pos 1259807
# WARNING: Packet  91603, WAV, Filling  31 missing samples at pos 1259968
# WARNING: Packet  91605, WAV, Filling   2 missing samples at pos 1260031
# WARNING: Packet  93977, WAV, Filling  33 missing samples at pos 1292575
# WARNING: Packet 100873, WAV, Filling  33 missing samples at pos 1387423
# WARNING: Packet 100875, WAV, Filling   2 missing samples at pos 1387487
# WARNING: Packet 103887, WAV, Filling  32 missing samples at pos 1428895
# WARNING: Packet 103891, WAV, Filling   2 missing samples at pos 1428991
# WARNING: Packet 119670, WAV, Filling  33 missing samples at pos 1645983
# WARNING: Packet 120266, WAV, Filling  33 missing samples at pos 1654175
# WARNING: Packet 125648, WAV, Filling  32 missing samples at pos 1728223
# WARNING: Packet 125651, WAV, Filling   2 missing samples at pos 1728287
# WARNING: Packet 134129, WAV, Filling  32 missing samples at pos 1844895
# WARNING: Packet 134131, WAV, Filling   2 missing samples at pos 1844959
# WARNING: Packet 141766, WAV, Filling  33 missing samples at pos 1949919
# WARNING: Packet 144164, WAV, Filling  33 missing samples at pos 1982847
# WARNING: Packet 144172, WAV, Filling  32 missing samples at pos 1983007
# WARNING: Packet 144176, WAV, Filling   2 missing samples at pos 1983103
# WARNING: Packet 153217, WAV, Filling  32 missing samples at pos 2107519
# WARNING: Packet 153221, WAV, Filling   2 missing samples at pos 2107615
# WARNING: Packet 155041, WAV, Filling  33 missing samples at pos 2132607
# WARNING: Packet 178163, WAV, Filling  33 missing samples at pos 2450624
# WARNING: Packet 195222, WAV, Filling  32 missing samples at pos 2685248
# WARNING: Packet 195236, WAV, Filling   2 missing samples at pos 2685439
# WARNING: Packet 214551, WAV, Filling  33 missing samples at pos 2951008
# WARNING: Packet 232503, WAV, Filling  33 missing samples at pos 3197919
# WARNING: Packet 244593, WAV, Filling  33 missing samples at pos 3364223
# WARNING: Packet 256193, WAV, Filling  33 missing samples at pos 3523808
# WARNING: Packet 256206, WAV, Filling  32 missing samples at pos 3523968
# WARNING: Packet 300972, WAV, Filling  33 missing samples at pos 4139679
# WARNING: Packet 300974, WAV, Filling   2 missing samples at pos 4139743
# INFO:   File [ 2]
# WARNING: Packet   5220, WAV, Filling  31 missing samples at pos 4329920
# WARNING: Packet   5223, WAV, Filling   2 missing samples at pos 4329983
# WARNING: Packet  12590, WAV, Filling  33 missing samples at pos 4431327
# WARNING: Packet  26143, WAV, Filling  34 missing samples at pos 4617791
# WARNING: Packet  37454, WAV, Filling  33 missing samples at pos 4773279
# WARNING: Packet  63151, WAV, Filling  33 missing samples at pos 5126687
# WARNING: Packet  67405, WAV, Filling  33 missing samples at pos 5185183
# WARNING: Packet  80853, WAV, Filling  33 missing samples at pos 5370175
# WARNING: Packet  86136, WAV, Filling  33 missing samples at pos 5442751
# WARNING: Packet  93745, WAV, Filling  33 missing samples at pos 5547455
# WARNING: Packet 102842, WAV, Filling  34 missing samples at pos 5672607
# WARNING: Packet 110762, WAV, Filling  34 missing samples at pos 5781567
# WARNING: Packet 111883, WAV, Filling  33 missing samples at pos 5796991
# WARNING: Packet 124512, WAV, Filling  31 missing samples at pos 5970656
# WARNING: Packet 124518, WAV, Filling   2 missing samples at pos 5970783
# WARNING: Packet 129266, WAV, Filling  33 missing samples at pos 6036031
# WARNING: Packet 148985, WAV, Filling  33 missing samples at pos 6307199
# WARNING: Packet 159037, WAV, Filling  32 missing samples at pos 6445472
# WARNING: Packet 168461, WAV, Filling  34 missing samples at pos 6575071
# WARNING: Packet 178658, WAV, Filling  33 missing samples at pos 6715327
# WARNING: Packet 196278, WAV, Filling  32 missing samples at pos 6957664
# WARNING: Packet 202828, WAV, Filling  31 missing samples at pos 7047776
# WARNING: Packet 202831, WAV, Filling   2 missing samples at pos 7047839
# WARNING: Packet 203008, WAV, Filling  33 missing samples at pos 7050239
# WARNING: Packet 228148, WAV, Filling  33 missing samples at pos 7395935
# WARNING: file index 12121092, MSB2 Packet CRC invalid
# WARNING: file index 12124690, UWAR Packet's ID is invalid
# WARNING: UWAR Error mode, looking for new uwar packet, start pos: 12124691
# WARNING: UWAR Error mode, found new uwar packet, at pos: 12124691 (0 bytes skipped)
# WARNING: Packet 244197, WAV, Filling 961 missing samples at pos 7616767
# WARNING: Packet 244198, Filling  29 missing samples for field accel
# WARNING: Packet 244198, Filling   6 missing samples for field visFreqLight
# WARNING: Packet 244198, Filling  29 missing samples for field accelY
# WARNING: Packet 244198, Filling  29 missing samples for field accelZ
# WARNING: Packet 244201, WAV, Filling   2 missing samples at pos 7617791
# WARNING: Packet 244212, Filling   3 missing samples for field gyro
# WARNING: Packet 244212, Filling   3 missing samples for field gyroTemp
# WARNING: Packet 244212, Filling   3 missing samples for field gyroY
# WARNING: Packet 244212, Filling   3 missing samples for field gyroZ
# WARNING: Packet 244212, Filling   3 missing samples for field gyroTempY
# WARNING: Packet 244212, Filling   3 missing samples for field gyroTempZ
# WARNING: Packet 254806, WAV, Filling  33 missing samples at pos 7763616
# WARNING: Packet 258487, WAV, Filling  34 missing samples at pos 7814239
# WARNING: Packet 264336, WAV, Filling  33 missing samples at pos 7894687
# WARNING: Packet 281685, WAV, Filling  32 missing samples at pos 8133247
# WARNING: Packet 281688, WAV, Filling   2 missing samples at pos 8133311
# WARNING: Packet 297231, WAV, Filling  32 missing samples at pos 8347072
# WARNING: Packet 304508, WAV, Filling  32 missing samples at pos 8447167
# WARNING: Packet 304512, WAV, Filling   2 missing samples at pos 8447263
# WARNING: completeData- Packet 309523, magnetometer artificial ceiling reached, sample dropped, for sensor mag


# INFO: Summary:
# INFO: File [ 1], MSB2 spans  134923 to   267990     259.90 seconds (  0 hours  4 mins 19.90 seconds)
# INFO: File [ 2], MSB2 spans  267990 to   401053     259.89 seconds (  0 hours  4 mins 19.89 seconds)

# INFO: File [ 1], LSB  spans  133189 to   266255     259.89 seconds (  0 hours  4 mins 19.89 seconds)
# INFO: File [ 2], LSB  spans  266256 to   399317     259.89 seconds (  0 hours  4 mins 19.89 seconds)
# INFO:    266130 MSB2 interrupt counts
# INFO:    266127 LSB  interrupt counts
# INFO: MSB2    519.79 seconds (  0 hours  8 mins 39.79 seconds)
# INFO: LSB     519.78 seconds (  0 hours  8 mins 39.78 seconds)
# INFO:       Sensor Name  SampleRate ExpectedSamples  FoundSamples    Diff  Percent Difference
# INFO:         msb/audio  16384.00Hz         8516160       8513377   -2783   ( 99.9673%)
# INFO:     msb/bar_press      7.10Hz            3690          3690      +0   (100.0000%)
# INFO:       msb/compass      0.00Hz               0             0      +0   (  0.0000%)
# INFO:  msb/hf_vis_light    128.00Hz           66531         66526      -5   ( 99.9925%)
# INFO:      msb/humidity      1.00Hz             519           519      +0   (100.0000%)
# INFO:      msb/ir_light      3.00Hz            1559          1559      +0   (100.0000%)
# INFO:     msb/amb_light      3.00Hz            1559          1559      +0   (100.0000%)
# INFO:          msb/temp      7.10Hz            3691          3691      +0   (100.0000%)
# INFO: msb/temp_sensiron      1.00Hz             519           519      +0   (100.0000%)
# INFO:     msb/vis_light      3.00Hz            1559          1559      +0   (100.0000%)
# INFO:       msb/x_accel    512.00Hz          266130        266101     -29   ( 99.9891%)
# INFO:       msb/y_accel    512.00Hz          266130        266101     -29   ( 99.9891%)
# INFO:       msb/z_accel    512.00Hz          266130        266101     -29   ( 99.9891%)
# INFO:        lsb/x_gyro     64.00Hz           33264         33262      -2   ( 99.9940%)
# INFO:        lsb/y_gyro     64.00Hz           33264         33262      -2   ( 99.9940%)
# INFO:        lsb/z_gyro     64.00Hz           33264         33262      -2   ( 99.9940%)
# INFO:    lsb/x_gyroTemp     64.00Hz           33264         33262      -2   ( 99.9940%)
# INFO:    lsb/y_gyroTemp     64.00Hz           33264         33262      -2   ( 99.9940%)
# INFO:    lsb/z_gyroTemp     64.00Hz           33264         33262      -2   ( 99.9940%)
# INFO:         lsb/x_mag     32.00Hz           16632         16632      +0   (100.0000%)
# INFO:         lsb/y_mag     32.00Hz           16631         16631      +0   (100.0000%)
# INFO:         lsb/z_mag     32.00Hz           16631         16631      +0   (100.0000%)
# INFO: Closing completeSensorAudio.wav
# INFO: Closing dataOut.mat
# INFO: Closing completeData.mat

C:\Test Data\home>
Retrieved from "https://ubi.cs.washington.edu/wiki/index.php?title=Uwar_tools&oldid=2170"