CSV and VBO Tools (free)

Do you want to ...

- select a collection of fields from a CSV file and generate a result CSV file
containing only those fields ... optionally:
* ignoring data records with missing/extra fields
* formatting the data fields in the result file
* sorting the data records by a collection of fields
* copying forward values to fill empty fields
* ignoring fields missing values for specified "required fields"
* transposing the field names in the header record
* transposing field values by applying built-in and user-specified formulas
* joining one or more fields into a single field
* changing a space/tab-separated file into a comma-separated file
* generating statistics such as the record-number total, the min/max/average
values and max-min value for each field

- combine 2 CSV files into one CSV file

- add info about your track video to a VBO file so that video can be shown when
using the RaceLogic's Circuit Tools application

- from a RaceCapture CSV/log file, generate a VBO file that can be used by
RaceLogic's Circuit Tools analysis application

- split a RaceCapture CSV/log file into one containing only specified fields
related to timing and another containing only specified fields related to
data-capture

... then see the following free tools:
https://www.derman.com/Resources/Mis...ndVBOtools.zip
(<100 KB)

The following files are contained within the "CSVandVBOtools.zip" file:
- addVideoToVBOdata.sh
- combineCSVfiles.sh
- CSVfieldSpecs.txt
- CSVformulas.txt
- extractCSVdata.sh
- rcCSVtoVBO.sh
- ReadMe - CSV and VBO Tools Info.txt
- splitRClog.sh

The 5 shell scripts have a help option: <script>.sh -h
(that's the documentation)

This version of extractCSVdata.sh is significantly newer than the one included
in the link below.

NOTE: there are also some CAN bus analysis tools I've written ... see:
https://forum.autosportlabs.com/view...p=30219#p30219
tools at:
https://www.derman.com/Resources/Mis...lysisTools.zip
(<5 MB)


Background
---
I developed these tools for my own use. I've used these tools with:

- macOS 10.12.6

- Circuit Tools 1.4.7 on a Mac

- RaceRender 3 on a Mac

- RaceCapture logs (CSV files) from a RaceCapture/Pro MK3 running various
firmware versions over the past 3 years

- for extractCSVdata.sh: various CSV files including many from RaceCapture and
Harry's LapTimer, one from an AIM data logger and some from various dashcams

I'm making this stuff available with the hope that some of it will help others.


Comments About VBO Files
---
- generating a VBO file from the RaceCapture CSV/log file can take a while
(e.g., around a minute for every 15 min of log-file/track-session time)

- with the VBO files, I've only used/linked MP4 videos, specifically those
generated either by RaceRender or by QuickTime export ... NOTE that .MOV can
be changed to .mp4 (supposedly AVI files work, also)

- with the videos I generate, there's always logging data that starts before the
video that was generated (I'm assuming this would be common) ... that means
the video offset I supply to addVideoToVBO.sh is a negative number/offset

- if your VBO-linked video includes an up-counting laptime overlay, then the
video offset value can easily be determined by first using a video offset of 0
then viewing in Circuit Tools, along with the data, to determine the
lap-timing difference based upon a given data-point reference -- then
regenerate the video-linked VBO using an adjusted video offset value based
upon the lap-timing difference you got when viewing in Circuit Tools (sounds
complicated, it's not) and addVideoToVBO.sh runs fairly quickly so it's easy
to iterate ... as a last resort, guess at an offset then use a binary search
technique by either doubling or halving the value to see what happens

- in Circuit Tools, if a video section stops updating when moving through the
data, click on the video to cause it to play a bit and it'll then normally be
OK thereafter (give Circuit Tools time to update the video frame[s] after each
new data repositioning to avoid this)

- (at least on the Mac) Circuit Tools appears to be very "flakey"/touchy, often
crashing while loading a VBO file and is extremely unforgiving WRT the VBO
file format ... small changes from what Circuit Tools is expecting will often
lead to erratic behavior

- I've attempted to test some generated VBO files using Circuit Tools for
Windows running under VMware Fusion on a Mac, but it "fails gracefully" ...
i.e., shows/does nothing after spending much time "initializing" after opening
a .vbo file (no errors, no behavior ... lame!)

- I've attempted to test the latest Mac version of Circuit Tools running under
macOS 10.13 and later via VMware Fusion on a Mac and it seems to work except
that no video is shown, though it's running and the audio can be heard (it
appears that Circuit Tools' way of showing video doesn't work via VMware
Fusion)


Problems?
---
- if you find a bug (Mac only, I have no real Windows) and want to supply (to
the RaceCapture forum listed below):
+ a description of the scripts/command(s) you ran
+ a copy of the scripts you ran (because mine may have been changed)
+ a copy of the file(s) you were processing
... I can likely fix fairly easily


... but wait, there's more! #;-))
=====
Seriously, there are some other RaceCapture-related things I've made available:

Free CAN Message Analysis Tools
---
Do you want to ...
- use your RaceCapture device to find all the OBDII PIDs supported on a CAN bus?
- use your RaceCapture device to capture all or just certain-ID CAN messages on
a CAN bus?
- know more about the inter-message timing on a CAN bus?
- sort captured CAN messages into groupings by CAN ID and/or PID?
- apply a collection of formulas to captured CAN messages and generate a csv
file containing the computed values along with a summary showing the record
number total, the min/max values and the max-min value for each field?
- select a collection of fields from a csv file and generate another csv file
containing only those fields ... along with the record number total, the
min/max values and max-min value for each field?

... then see the following:
https://www.derman.com/Resources/Mis...lysisTools.zip
(<5 MB)


Stuff About Audio/Video Syncing via Sound in RaceRender
---
How I use audio to sync multiple video/audio data sources when creating track
videos using RaceRender, including a RaceCapture-driven beep-sync setup.

... see:
https://www.derman.com/Resources/Mis...ngViaSound.zip
(<4 MB)