Process large textfiles using PHP

[Jiberish]

I have a textfile with millions of rows. I want to collect one word from each row. The first thing that came to mind was an array, but that would allocate so much memory and it's just not a practical solution because of that.

The reason to why I find an array interesting is that it would be extremely simple to compare the array with another array at a later stage and find the words that only exist in one of the files.

I suppose I could read all the words into a database and take it from there. Any other solutions? Thanks in advance!

[Horus_Kol]

that amount of data... database...
it will make searching through it a whole lot more efficient

[erisco]

Jiberish, unless you know exactly where the words are located you will have to read the entire file. If you only have to do this process once to convert it to something more efficient like a database, then that is fine, you may just need to increase the max memory php is allowed to take.

[Horus_Kol]

or do it with a CLI script instead of a CGI, or just break the file down into bits...

[Jiberish]

Thanks guys! I'm working on something that read each line and then write the word to a database. Since I'm not using an array I'm hoping to be able to do it without exhausting the memory. I can't change the memory limit since I'm on shared hosting.

[erisco]

See if you cannot read it a few bytes at a time until it is all converted to the database.