Lahey Support
08-15-2003, 12:19 AM
If your priority is speed and you have enough memory to hold all the data
in RAM during execution, then there is no contest: read the file at the
beginning into an array. Disk operations are literally thousands of times
slower than RAM operations, and the most expensive (time-consuming) disk
operations are opening/closing files and moving the disk read/write heads
to new tracks. By reading the file all at once you minimize both of those
time-gobbling tasks. If you read the file in bits and pieces as the
program progresses, you run the risk that the disk read/write heads will
be moved out of position by other processes going on at the same time.
Beware of trying to use more memory than your actual physical RAM, though.
It's very easy nowadays to wind up committing more memory than you have in
RAM. What happens in that case is that the overflow is written to disk as
"virtual memory", and you're right back to the slowdown problem you're
trying to avoid in the first place.
Do you have to modify the data file at all and save the changes? If not,
then go ahead and close the file right after you've read it. If you do
have to make changes, then you can reduce the slow-down caused by writing
to the disk if you use disk caching (cacheing? It's pronounced "cashing")
with "lazy writes" that wait until the computer isn't so busy before it
actually commits the changes to disk.
You might want to try it in different ways to see how much of a problem
there really is, though. It might turn out not to matter much in your
particular application.
On Tue, 2 Feb 1999, Dora wrote:
> I have a file that contains data that i will use during the execution of the
> program. I am curius what would be faster: to open the file once read all
> the data and store them during execution in an array or whenever i need
> something from this file to open it get the values i need close the file (or
> maybe leave the file open during execution) etc. without commiting an array?
> Thanks.
>
> Dora Avanidou
>
in RAM during execution, then there is no contest: read the file at the
beginning into an array. Disk operations are literally thousands of times
slower than RAM operations, and the most expensive (time-consuming) disk
operations are opening/closing files and moving the disk read/write heads
to new tracks. By reading the file all at once you minimize both of those
time-gobbling tasks. If you read the file in bits and pieces as the
program progresses, you run the risk that the disk read/write heads will
be moved out of position by other processes going on at the same time.
Beware of trying to use more memory than your actual physical RAM, though.
It's very easy nowadays to wind up committing more memory than you have in
RAM. What happens in that case is that the overflow is written to disk as
"virtual memory", and you're right back to the slowdown problem you're
trying to avoid in the first place.
Do you have to modify the data file at all and save the changes? If not,
then go ahead and close the file right after you've read it. If you do
have to make changes, then you can reduce the slow-down caused by writing
to the disk if you use disk caching (cacheing? It's pronounced "cashing")
with "lazy writes" that wait until the computer isn't so busy before it
actually commits the changes to disk.
You might want to try it in different ways to see how much of a problem
there really is, though. It might turn out not to matter much in your
particular application.
On Tue, 2 Feb 1999, Dora wrote:
> I have a file that contains data that i will use during the execution of the
> program. I am curius what would be faster: to open the file once read all
> the data and store them during execution in an array or whenever i need
> something from this file to open it get the values i need close the file (or
> maybe leave the file open during execution) etc. without commiting an array?
> Thanks.
>
> Dora Avanidou
>