Make row validation optional, parse XLSX files, add worksheet column to Excel parser

[tylergannon]

hi, added the following features.

• CSV and Excel parser can now accept :validate_rows as a parameter to their source object. If this is turned off, then the process won't die when a row has the wrong number of fields.
  • This is because in my application it's pretty annoying to have the whole process die: I have varied result sets in my input files and would rather validate and exclude rows then to have to preprocess my inputs before giving them to the ETL.
• Added in a dependency to Roo, which is under more active development than the Spreadsheet gem and can parse XLSX files (big bonus for me)
• If the excel parser is given worksheet_column: :column_name in the definition configuration, the excel parser will add the name of the worksheet to each row in a field by that name.
  • This is great for me since I have a multitude of sheets where the name of the sheet differentiates the data logically.

My change to ETL::Engine is for compatibility with activerecord 3.x, which won't allow mass assignment to fields that have not been declared as accessible in the model. I didn't add any attr_accessible calls on the model because I don't want to mess with backward compatibility.

Lastly, I am not having an easy time with getting the test suite to work. I'm happy to do the leg work in getting these changes properly tested but I wanted to see if you're interested in these changes -- and if so, whether you wouldn't mind helping me out with the test environment.

I have some cool new DSL I've developed, and if I can get my test environment working I'll pull it out of my application and send another pull request to see about adding it to the gem.

Cheers!
Tyler

[thbar]

Hello Tyler!

I took quite a bit to answer, I was very busy launching https://www.wisecashhq.com.

I'm sorry this was not clear, but this specific repository is experimental and is not really maintained!

The maintained repo is here:

https://github.com/activewarehouse/activewarehouse-etl

I'm not sure how much work this is to adapt things to over there (and some of it is probably already integrated, like the mass assignment issue).

What do you think? Would you like to adapt the code so I merge it over there?