ETL and Data Integration tools - Modern ETL software for everyone

Hi,

I'm interested in trying out Visual Importer and would like to know a thing or two about how the program reads/writes data.

I have tried several ODBC import/export tools. Most of them stores the data in memory then writes it to disk. This becomes a problem when the data set returned from the query is huge. How does your program handle this?

Manuel

Hi.

We do not read all data into the the memory when we fetch data or load it into the database.
We reuse the same buffer.
We have tested it by loading 1.5 million records from odbc source.
At the beginig memory is growing later it becames steady Amount of memory used
depends on how wide is your table.
We also compared memory usage before and after the import in order to avoind memory leaks when people use it to load data on the regular basis.

You are welcome to test it yourself (this why it is shareware)

Hope that clarifies the situation,

Piter,
DB Software laboratory

Peter

Since I specified the source, the program has been running (>1
hour) and taken up 370M of memory. It is still going... When will it stop
reading data??? Until it max out? The source table is >50G!

Why would it read data when I have not yet specified my target,
etc.?
How exactly does the program determine the 'buffer'?

While I'm typing...
The program just keeps taking up memory and I have no idea what's going to happen next.

Help please.

Hi.

Can you provide us with some information

What is your database?
Version of odbc driver and manufacturer

Peter
DB Software laboratory

Source:
HP (Tandem) NonStopSQL/MP, ODBC driver ODBCD35ACH from HP.
Target:
MySQL, ODBC driver

Follow-up to my earlier email... This morning when I checked on the whole
process, memory uptake peaked out at ~800M and there is a XP memory low
warning. Right now, the program memory usage is down to 6M. Don't know what
happened to that 794M of data that it pulled down.

In the source pane of the program, there is no data shown not even the field
names.

I specified the target and was successful in getting the fields. Try to use
auto match. Nothing happened. So... I go ahead and add columns to the source
pane manually (89 columns w/o field names). Then I started to match target
with source manually. First 7 went fine. Upon the 8th, CPU usage went up
from 0 to ~50% and the program locked up. No memory usage changes.

30 minutes has gone by and nothing changes.

If you need more information, please feel free to ask me. But regrettably,
this testing has to be aborted. I'm cannot invest another day on this before
hearing something back from you.

Also in the task manager, two instances of Vimp.exe
shown. One's memory usage has consistently stayed at 668k while the other
went from 6M to 800M. In terms of CPU usage, both together 100%. I only
started the program once so something else is going on here that I am not
comprehending...

Manuel

Hi

I gues HP (Tandem) NonStopSQL/MP, ODBC driver ODBCD35ACH from HP is
not avaliable for download and we can't play with it or test it.

What we can do is to create test application for you with some debug information it will help us to understand what is going on

It will take some time

Can you also email us some screeshots of your odbc driver settings (Tandem)
That could help as well.

Can you also check with HP if updates of ODBC driver are
avaliable. (if possible)

Peter