HomeForum
Welcome, Guest

Transfering huge amount of data
(1 viewing) (1) Guest
Data Warehousing and Data integration
  • Page:
  • 1

TOPIC: Transfering huge amount of data

Transfering huge amount of data 1 year, 10 months ago #904

  • Mars
  • OFFLINE
  • Fresh Boarder
  • Posts: 2
  • Karma: 0
I have to get about 100,000,000 records from one database to another database. I will to this with ETL but I have doubts about performance issues...
For instance; I think that I'd better to commit on every 10000 records
I need suggestions from the gurus. What should ı do?

Thank you in advice

Re:Transferring huge amount of data 1 year, 10 months ago #905

  • admin
  • OFFLINE
  • Moderator
  • Posts: 2183
  • Karma: 12
There a lot of things you should consider
Can you connect from source database to the target database directly?
Or other way around?
If you can't connect you might need to export entire table into a text file transfer it across
Do you have enough disk space on the target server?
Copy 100k rows, check you disk space, check how long did it take.
It will give you the idea how long it will take to load everything
While coping the data monitor the memory usage of both servers,
if it grows all the time that means the server is loading entire table into the memory.
The load might fail if server runs out of memory.
You can avoid this by splitting your transfers
eg select * from very large table where id <100000
select * from very large table where id between 100000 and 200000
You should also consider how to restart your data transfer

BTW what is your database types?

Mike

Re:Transferring huge amount of data 1 year, 10 months ago #906

  • Mars
  • OFFLINE
  • Fresh Boarder
  • Posts: 2
  • Karma: 0
I did not realize that it is not so straight forward

My source database is MS SQL, target Oracle

to mars or snickers?
that is the question...

Re:Transferring huge amount of data 1 year, 10 months ago #908

  • admin
  • OFFLINE
  • Moderator
  • Posts: 2183
  • Karma: 12
If you can't connect from source to the target
BCP out int the text file watch out for disk space
Copy the file across
oracle loader to load the data into oracle database

If you can connect DTS OR SSIS from MS Sql Server or external tables in Oracle.

Or you can use any of our data transformation tools they are not memory hungry and will give decent performance
There is no need to write any scripts just point and click

Hope that helps,
Mike
  • Page:
  • 1
Time to create page: 0.17 seconds

Testimonials

"Advanced ETL Processor is superior software... I highly recommend this to anyone who is building datawarehouse. I would and I  have recommended it to many colleagues and customers."

- Peter Jonson, Data Integration Consultant, MPI

User Login

You only need to log in or register to use our support forum



Our customers

BP

BBC

HSBC


Databases we work with

Go to top