ETL and Data Integration tools - Modern ETL software for everyone

I ran a test to see if the latest version is working the way I expected.
It doesn't seem to be. I changed one row in the text file to have bad data in a numeric column.
The import received an error but it still loaded all of the rows except for the one that failed.
I expected that no rows would be added since there was a failure.
Do I have something set up wrong?

Also, I'm finding it difficult to find the correct log file for the package.
Is there any way to associate a package to it's log files?
Maybe the package name or number could be part of the file names in the log folder?

Ron

Ron.

I have spoken with our development tea today about what are trying to achieve

Our software was designed to work with dirty data and to load as much records as possible.

For ODBC connection every insert a is a separate transaction

eg:
Insert
commit
Insert
commit

ETC

While it is possible modify the software to run it as one big transaction it is rather complicated issue which will require a lot of additional testing and will take a while to introduce.

We will let you know when it became available.

What I can suggest for now is to add new field to the table to flag newly inserted records.
If there were any errors during import delete those records

> Maybe the package name or number could be part of the file names in the
> log folder?

The file names are actually stored in the repository.
We will use package number as a part of file name in the next release.

Mike

Thanks. FYI, I did change to use a SQL server connection and not an ODBC connection. I changed the batch size to 1,000,000 on the SQL connection and it still wrote the records without an error.
I expected records without an error to be written using ODBC, I just didn’t expect them to write using a SQL connection with a large batch size.

Having the package id in the log file name will help me. We will probably run 50 to 75 packages a day and it will be difficult trying to identify the log files for a specific package.

Ron