Jump to content

Mysql Databases


TheRaver
 Share

Recommended Posts

Hi All,

I am generating a mysql database this will be an 11TB database(hopefully) Im using pyrit to generate this but because the size of the data I would like to use the archive db engine but i know nothing about this database engine... has anyone one had any interaction using this sort of setup??

I am hoping to generate the big database then only read from it I never want to change records just add new ones and select old ones....

I dont know if this is the best way to do what i want to do but am after any ideas or alternatives

Thanks

The Rave..

Live long and prosper

Link to comment
Share on other sites

Well the ARCHIVE engine does indeed support INSERT and SELECT but the big question is, what performance do you need from the SELECTs? The ARCHIVE engine doesn't support indices so every select will require a full table scan, which with 11TB of data will be a lot of work.

Given that you are planning on having using this with pyrit I suspect that the ARCHIVE engine will not be the best for actual use, and that you would be better off with one that supports indices. My gut instinct would be to go with the InnoDB engine unless you have a good reason to use something different.

Link to comment
Share on other sites

I'd agree with Jason, just use the InnoDB MySQL Enginne, it will be much more reliable and offer a good level of performance. In addition, for incresed read performance, I'd use a NAS device in RAID 5 or 10.

If you need more information, check out the MySQL Engines reference manual.

http://dev.mysql.com/doc/refman/5.1/en/storage-engines.html

Link to comment
Share on other sites

Thanks both I am using the innoDB as this was the default.

I am finding that its taking a week and counting to import a 150gig txt file.

I am finding that when i query the database even when its updating it is still very fast database size is around 20Gig

Connecting to storage at 'mysql://localhost/mydatabase'... connected.

Passwords available: 4475025000

That is approximately a week of importing I have a shed load of these to import.

The only thing i am thinking why its slow i gziped the txt files to save space but bough a couple more drives for the data.

Must get back too it :)

Thanks for your help.....

The Raver

Link to comment
Share on other sites

A useful trick when importing the initial data into MySQL is to switch off the indices on the table "ALTER TABLE <table> DISABLE KEYS;" and then after importing the data in to the table switch them back on "ALTER TABLE <table> ENABLE KEYS;". When they are switched back in it will use a different algorithm to regenerate the indices which is much faster than repeatedly running the routine to update the indices after each insert.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

 Share

  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...