{"id":363,"date":"2011-04-01T02:55:34","date_gmt":"2011-04-01T02:55:34","guid":{"rendered":"https:\/\/info.calcofi.com\/2011\/04\/01\/underway-data-processing\/"},"modified":"2011-04-01T02:55:34","modified_gmt":"2011-04-01T02:55:34","slug":"underway-data-processing","status":"publish","type":"post","link":"https:\/\/mindwander.com\/index.php\/2011\/04\/01\/underway-data-processing\/","title":{"rendered":"Underway Data Processing"},"content":{"rendered":"

\n\tUnderway Data Processing Methods <\/o:p><\/p>\n

\n\t1.       <\/span><\/span>Navigate to Underway Data folder and join into a single file, do so using Advanced File Joiner or similar:  <\/span>http:\/\/download.cnet.com\/Advanced-File-Joiner\/3000-2094_4-169639.html<\/span><\/a>

\n\t

\n\t <\/o:p> <\/span><\/p>\n

\n\ta.     <\/span><\/span>CalCOFI collects its own data while aboard SIO ships (New Horizon).  These are saved in the Events folder of the appropriate cruise.  A single file name is similar to: 0901NHMETEV_06012009.csv.  Although each filename is labeled by date, date data my not correlate.  This is of no particular consequence, just be aware of it.

\n\t

\n\t<\/o:p><\/p>\n

\n\tb.     <\/span><\/span>When a cruise is aboard a NOAA ship, it is normal to receive a CD-ROM at the cruise with a variety of SCS data.  These data formats are highly variable by ship and by year.  It will usually take a combination of files to find all that you will require (i.e., one file type for TSG, one for fluorescence, one for lat long).  Just merge each file type, then cut and paste columns into the correct order.  Make all SCS data available on the web.

\n\t

\n\t<\/o:p><\/p>\n

\n\tc.     <\/span><\/span>The resultant  file becomes the RAW UNDERWAY DATA for that cruise.  This file contains all unedited data from the Shipboard Computer System (SCS).  This file may include navigational, meteorological, oceanographic, and other mechanical data.  <\/span>In your local working directory  (e.g., ..\/UnderwayData\/2009\/0911NH) save raw data in tab delimited ASCII format and name as YYMMSS_UnderwayRaw.txt  (e.g., 0901NH_RawUnderway.txt)

\n\t

\n\t<\/o:p><\/p>\n

\n\td.     <\/span><\/span> *Links to SCS information <\/o:p><\/p>\n

\n\t                               <\/span>i.    <\/span><\/span><\/span>http:\/\/oceanexplorer.noaa.gov\/technology\/tools\/scs\/scs.html<\/span><\/a> <\/o:p> <\/span><\/p>\n

\n\t                               <\/span>ii.    <\/span><\/span><\/span>http:\/\/www.unols.org\/meetings\/2006\/200610inm\/SessionIIIa\/SessionIIIa_Shields.pdf<\/span><\/a> <\/o:p> <\/span><\/p>\n

\n\t                               <\/span>iii.    <\/span><\/span><\/span>http:\/\/sdsioa.ucsd.edu\/<\/span><\/a>

\n\t

\n\t <\/o:p> <\/span><\/p>\n

\n\t2.       <\/span><\/span>Open merged file and crop data columns to include the following, in order and with the following column headers:

\n\t

\n\t<\/o:p><\/p>\n

\n\ta.     <\/span><\/span>DATE_UTC, TIME_UTC, LA, LO, ST, SA, FL, FI  (Date(UTC), Time(UTC), Lat, Lon, Temp, Salinity, Fluor, Pump)

\n\t

\n\t<\/o:p><\/p>\n

\n\t                                          <\/span>i.    <\/span><\/span>*Note – There is the possibility that power gets toggled on\/off to underway sensors during the cruise.  This results in changes in the total number of data columns (i.e., the first days may have 100 columns of data and the next few only 98).  Attention  <\/span>is needed to make sure that when cropping data that the correct data is sub sampled.

\n\t

\n\t<\/o:p><\/p>\n

\n\t                                         <\/span>ii.    <\/span><\/span> **Note – Different ships have different underway column headers\/variable designations\/configurations.  Consult Underway Data Formats (link<\/a>) for more information.

\n\t

\n\t<\/o:p><\/p>\n

\n\t3.       <\/span><\/span> Delete rows with:

\n\t

\n\t<\/o:p><\/p>\n

\n\ta.     <\/span><\/span>pump speed not within than manufacturer’s specifications

\n\t

\n\t<\/o:p><\/p>\n

\n\t                                          <\/span>i.    <\/span><\/span> Doubling the flow rate from what we recommend will have at least one main problem. This flow rate will probably be too high for the conductivity cell and overtime you can see a degradation of the conductivity readings due to this flow rate. What can happen is that the high flow rate may slowly remove the platinum on the electrode within the conductivity cell, especially in areas with a lots of debris and particulate in the water. This will change the area of the conductivity cell which can cause the readings to be high of correct. I am unsure how long this process would take but it is something we have seen before. I really suggest using the recommended flow rate for this instrument. ~SBE Technical Support

\n\t

\n\t<\/span><\/i> <\/o:p> <\/span><\/p>\n

\n\tb.     <\/span><\/span>no spatial coordinates

\n\t

\n\t<\/o:p><\/p>\n

\n\tc.     <\/span><\/span>erroneous values

\n\t

\n\t<\/o:p><\/p>\n

\n\t                                          <\/span>i.    <\/span><\/span>*Note – There is the possibility that one or more columns may be missing some or all of the data.   There are several workarounds depending on the missing variable. Consult WTF Moments (link) for more information.

\n\t

\n\t<\/o:p><\/p>\n

\n\t                                         <\/span>ii.    <\/span><\/span>**Note – With upwards of 100,000 rows of data it is inevitable that most bad values will go unnoticed at this point.  Subsequent data processing steps (i.e., plotting) will make it easier to find errors and to remove them.  Just try to find the large and obvious discrepancies. 

\n\t

\n\t<\/o:p><\/p>\n

\n\t4.    <\/span><\/span>Save cropped file as 0901NH_UnderwayCrop.txt  in same folder as the raw file.

\n\t

\n\t<\/o:p><\/p>\n

\n\t5.    <\/span><\/span>It is necessary to compare and calculate correction regressions based on the comparison of CTD to Underway Data.  Navigate to the final CTD Downcast .csv file folder and join all csvs into a single file (as in Step 1).  These csv files are the ones with IEH bottle data added to them.  <\/span>In your local working directory  (e.g., ..\/UnderwayData\/2009\/0901NH) save raw data in tab delimited ASCII format and name as YYMMSS_CTDd.txt (e.g., 0901NH_CTDd.txt).  If the YYMM_CTDd.csv file is available open in Excel and crop columns accordingly (#7).  

\n\t

\n\t<\/o:p><\/p>\n

\n\t6.    <\/span><\/span>Open file with MS Excel and select the first column ‘A’ and Text-to-Column the Data with comma delimiter.

\n\t

\n\t<\/o:p><\/p>\n

\n\ta.   <\/span><\/span>*Note – DO NOT save this file as YYMMSS_CTDd.txt while in Excel.  A host of things can go wrong.  Data will be Cut and Pasted into a new file shortly.  If you must you can save it as some tempfile.xlsx.

\n\t

\n\t<\/o:p><\/p>\n

\n\t7.       <\/span><\/span> Crop data columns to include the following, in order and with the following column headers:

\n\t

\n\t<\/o:p><\/p>\n

\n\ta.     <\/span><\/span> <\/span>Date_Time_UTC, Lat_Dec, Lon_Dec, Temp0, Temp1, BTL_Depth, BTL_Salt, BTL_Chl

\n\t

\n\t<\/o:p><\/p>\n

\n\t                                          <\/span>i.    <\/span><\/span>*Note – If Underway Data was only available with Date and Time in PST select Date_Time_PST instead.

\n\t

\n\t<\/o:p><\/p>\n

\n\t8.    <\/span><\/span>Sort Ascending (Smallest to Largest) based on BTL_Depth and Delete all depths greater than 2m, all rows with no BTL_Depth values, and all Header rows except for one which needs to be cut and pasted back at the top after we are finished sorting.

\n\t

\n\t<\/o:p><\/p>\n

\n\t9.    <\/span><\/span>Sort Ascending based on Date_Time.

\n\t

\n\t<\/o:p><\/p>\n

\n\t10.  <\/span><\/span>Delete any rows that are missing any information.

\n\t

\n\t<\/o:p><\/p>\n

\n\t11.  <\/span><\/span>Delete any duplicate stations.  You can tell if two subsequent rows are from the same station by the time, delete  <\/span>the deeper.

\n\t

\n\t<\/o:p><\/p>\n

\n\t12.  <\/span><\/span>Make sure Header row is at the top.

\n\t

\n\t<\/o:p><\/p>\n

\n\t13.  <\/span><\/span>After the Temp 1 column, insert 2 additional blank columns.

\n\t

\n\t<\/o:p><\/p>\n

\n\ta.   <\/span><\/span>Average the two Temp columns into the first blank column, then copy and paste ‘values only’ into the second.  Delete the Temp0, Temp1, and first column.  Header the remaining column as TempAvg.

\n\t

\n\t<\/o:p><\/p>\n

\n\t14.     <\/span><\/span> Add three extra headers\/columns  at the end of the existing

\n\t

\n\t<\/o:p><\/p>\n

\n\ta.     <\/span><\/span> …, ST, SA, FL, FI (…, T, S, Fluro, Pump)

\n\t

\n\t<\/o:p><\/p>\n

\n\t15.  <\/span><\/span>Open a previously completed copy of YYMMSS_CTDvsUnderway.xlsx.  Select, Copy, and Paste all rows and columns from the sorted and corrected …CTDd.txt file into the recently opened xlsx file.  Save and RENAME the xlsx file accordingly (i.e., NEW YYMMSS ).  Make sure no old data lingers in the spreadsheet as this would cause errors in the fit of the regression.

\n\t

\n\t<\/o:p><\/p>\n

\n\ta.   <\/span><\/span>*Note – We reuse the xlsx file because it already has multiple graphs set up.  You will of course need to resize the data sets for each graph as necessary, and rename the graph titles.  But if you followed the above steps the columns will be in the right places and everything should fall into place.

\n\t

\n\t<\/o:p><\/p>\n

\n\t16.  <\/span><\/span>Open the …UnderwayCrop.txt in MS Excel, Tab delimited.  Arrange both files in Tile View.

\n\t

\n\t<\/o:p><\/p>\n

\n\t17.  <\/span><\/span>Using this new …CTDvsUnderway.xlsx file’s Date_Time as a guide, scroll down the …UnderwayCrop.txt and find a corresponding date and time.  There will be more than one row of data for the time you are trying to match.  Try not to bias the selection of the Underway data and just pick the first one you see. Select the four columns, Temp, Salinity, Fluor, Pump, from UnderwayCrop and Copy them, and Paste them to the xlsx file in the last four columns ST, SA, etc.  Do this for all the rows in the CTDvsUnderway file.

\n\t

\n\t<\/o:p><\/p>\n

\n\ta.     <\/span><\/span>Because the number of rows will change from cruise to cruise, the graph data source will need to be adjusted accordingly.  This is important as the linear regression equations shown on the graphs are used to correct the Underway Data in subsequent steps.  Save file.

\n\t

\n\t<\/o:p><\/p>\n

\n\t18.  <\/span><\/span>For export to MATLAB it is necessary to Cut the xlsx data (numbers no graph info) and paste it into a new UltraEdit .txt file. Name it accordingly (YYMMSS_CTDvsUnderway.txt).  <\/span>This is done to remove any .xlsx formatting that may cause MATLAB to choke on import.

\n\t

\n\t<\/o:p><\/p>\n

\n\t19.  <\/span><\/span>Start Matlab and load a previously completed .m file as a template (e.g., 0901NH.m).  The .m file will have incorrect variable names and notes pertaining to that specific cruise.  Change and update accordingly.  Also make sure you are working in your current underway folder.

\n\t

\n\t<\/o:p><\/p>\n

\n\ta.     <\/span><\/span>Import UnderwayCrop.txt and follow along the notes in the script for error checking and saving data as a .mat file.

\n\t

\n\t<\/o:p><\/p>\n

\n\tb.     <\/span><\/span> Import …CTDvsUnderway.txt.  There will be an error with this as the Date_Time will not ingest properly.  Ignore it, the date is not important for plotting  <\/span>unless GPS has failed and you need the data and time to correlate with the Event Log GPS to get some coordinates to make these maps.

\n\t

\n\t<\/o:p><\/p>\n

\n\tc.     <\/span><\/span> An important step to be aware of is toward the end of the script it asks you to use dlmwrite function to export the final underway data product.

\n\t

\n\t<\/o:p><\/p>\n

\n\t20.     <\/span><\/span>The template m-file will guide you through the entire process.  When you get to the regression corrections take the values from the xlsx file graphs that you generated earlier.  When you make the Matlab plots they should print into your working folder.

\n\t

\n\t<\/o:p><\/p>\n

\n\t21.     <\/span><\/span>Open a previous yymmplot.ai file in Adobe Illustrator and the first plot from Matlab (i.e., t0901NHpath.eps).  Save the .ai file with the current cruise YYMM.

\n\t

\n\t<\/o:p><\/p>\n

\n\ta.     <\/span><\/span>In dealing with the .eps file from Matlab you will find that all we really need is the color path and the gradient bar frame and numbers.  Ungroup and delete all you don’t need (coastline, grid, tick marks, gradient color, etc.)

\n\t

\n\t<\/o:p><\/p>\n

\n\tb.     <\/span><\/span>Since the path is thousands of small circles, select them all, group them, then copy them to the clipboard. Delete the preexisting path in the .ai file and make sure you paste the new one in the correct .ai layer and resize to fit the station layout \/ cruise path.  This isn’t an exacting process, just make it fit best you can.  Make sure to lock\/unlock layers as you go to prevent the pasting \/ editing in the wrong place.

\n\t

\n\t<\/o:p><\/p>\n

\n\tc.     <\/span><\/span>Go back to the eps and copy the gradient bar box, tickmarks, and numbers.  Since each cruise’s graph has a different range of values these gradient bars change.  The colors stay the same (if you followed the Matlab m-file correctly) but we need to make the new numbers fit the template.   Delete the old box and numbers and paste in the new and resize accordingly.  Resizing will skew the font so once everything is in the right place check the font setting and set to 18pt Myriad Pro with 100% font stretching in both directions.  You will also have to change the decimals and numbers a bit as Matlab tend to separate the numbers after the decimal in eps format.  Just delete the end and retype the number using the left half.

\n\t

\n\t<\/o:p><\/p>\n

\n\td.     <\/span><\/span>Unlock and Edit  <\/span>the Title layer to the correct cruise designation and dates and title.

\n\t

\n\t<\/o:p><\/p>\n

\n\te.     <\/span><\/span>In Illustrator to save as pdf “save a copy”, to save as png Use Export (white background, medium resolution, anti aliased) .  Open each png in Photoshop and shrink to 75 pixels keep proportionality and rename with a         <\/span>” _t” at the end for the web thumb. <\/o:p><\/p>\n

\n\t31.   Upload all data and images to calcofi.com FTP Underway_Data directory. <\/o:p><\/p>\n

\n\t32.   Edit calcofi.com Joomla site with new underway data accordingly. <\/o:p><\/p>\n

\n\t <\/p>\n","protected":false},"excerpt":{"rendered":"

Underway Data Processing Methods 1.       Navigate to Underway Data folder and join into a single file, do so using Advanced File Joiner or similar:  http:\/\/download.cnet.com\/Advanced-File-Joiner\/3000-2094_4-169639.html a.     CalCOFI collects its own data while aboard SIO ships (New Horizon).  These are saved in the Events folder of the appropriate cruise.  A single file name is similar to: 0901NHMETEV_06012009.csv.  Although each […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[19],"tags":[],"class_list":["post-363","post","type-post","status-publish","format-standard","hentry","category-methods"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/mindwander.com\/index.php\/wp-json\/wp\/v2\/posts\/363","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mindwander.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mindwander.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mindwander.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mindwander.com\/index.php\/wp-json\/wp\/v2\/comments?post=363"}],"version-history":[{"count":0,"href":"https:\/\/mindwander.com\/index.php\/wp-json\/wp\/v2\/posts\/363\/revisions"}],"wp:attachment":[{"href":"https:\/\/mindwander.com\/index.php\/wp-json\/wp\/v2\/media?parent=363"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mindwander.com\/index.php\/wp-json\/wp\/v2\/categories?post=363"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mindwander.com\/index.php\/wp-json\/wp\/v2\/tags?post=363"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}