Speed of Applications in VBA vs VB

[Mark HOlcomb]

,I have written an application in VBA that reads a very large binary file.
I am pleased that it works at all .. but it runs too slow to be of any
practical use. I read 4 bytes at a time, then convert. The file is a
mixture of integer, ascii and floating point data. And very large. I want
to read several of these files in succession.
Would the stream reader speed things up? Or, if I transformed it into VB,
would it go faster? Is there an inheriant speed problem with VBA because it
is wrapped up in EXCEL? I only use VBA because I am most familiar with that
user interface.

Any comments are greatly appreciated.

Mark

 

[Jim Thomlinson]

If I recall correctly the engine for VB is actually derived from VBA (which
seems backwards until you think about it). VB just wraps a bunch of
functionality around the VBA engine that is not available in VBA (creating
dll's and such). So converting to VB probably will not help. As for the
stream reader speeding things up... Give it a try and let me know.

Beyond that post your code and let some of the greater minds here have a
crack at it. Perhaps there are some code changes that will optimize your
speed.

 

[Mark HOlcomb]

Thanks .. I have tried to make the code concise but it might be too big to
post.
There is a commerical code that reads the same binary file, but does it very
fast.
I'm missing somthing. I will try the streamreader. perhaps getting the data
into memory, as opposed to hitting the disk every 4 bytes will make a
difference.
I found that the built in math functions dec2hex and hex2dec slowed the code
down a lot. created my own functions to perform this task and made a huge
performance improvement. I will let you know about the streamreader ..

 

[Mat P:son]

Hi Mark,

When I read your post it simply struck me that I had actually seen a
performance comparison between different Excel technologies this afternoon.
As all such investigations, the stats you end up with obviously depend
heavily on exactly what type of processing you're doing, how you're reading
your data, the specific characteristics of teh data you're operating on, what
system configuration you've got, etc, etc. Still, the comparison may be of
some use, as a little rough estimate of what you should be able to expect:

http://www.codematic.net/excel-user-defined-functions.htm

However, I'm not entirely sure what you're trying to achieve; you're saying
you're reading a file and processing it, but at the same time I get the
impression that you're not using Excel at all, apart from "hosting" your VBA
project, are you? Or has that part of the discussion simply been left out of
the discussion?

If you are not using Excel, then I would strongly suggest moving to e.g. the
new .NET technologies. The entire Visual Studio enviroment can actually be
downloaded for free, in a light version (e.g., search for Visual Studio
Express Edition on Google)

Cheers,
/MP

 

[Jim Thomlinson]

I look forward to finding out your results. The commercial code is probably
written in C/C++ which is quite a bit faster than VB. Without seeing your
code one of the biggest cuplrets that slow things down is the calculation
settings. If you are looping and the spreadsheet needs to recalc with each
loop it is going to be slow. Set your application.calculation to xlManual and
then back to xlAutomatic at the end. Deletes and inserts can be slow. Print
settings are slow.

Here is a good link to a site that help with tweaking perfomance...
http://www.decisionmodels.com/index.htm

 

[Mark HOlcomb]

Thanks .. yet another language to learn (I dont know C#)! Great comparison!
Excel is a good container for the data mining. I haven't really used the
spreadsheet piece yet. I will be reading 200 of these files and making
comparision between them.

 

[Mark HOlcomb]

I have turned all of the screen updataing an auto recaluating toggles off.
I'm not really using the spreadsheet yet and it doent make much difference
at this point. I appreciate the feedback. Thanks

Mark

 

[Mat P:son]

Thanks .. yet another language to learn (I dont know C#)!

Heheh, well, if you know half a dozen or so already I s'pose a few extra
languages won't make much difference, right?

Great comparison!

Yeah, I thought it was pretty interesting. However, it's very
compute-centric, i.e., you don't really know what sort of results you would
get had it focused more on e.g. I/O.

Excel is a good container for the data mining.

I agree. However, flexibility and short time-to-market in this case (and
pretty many other cases as well) obviously comes at the price of performance.

I haven't really used the
spreadsheet piece yet. I will be reading 200 of these files and making
comparision between them.

Okay, well, if you plan to read them in all at once make sure you've got
your box fully stacked up with RAM then... (just kidding )

Good luck with the work!

Ta da,
/MP

 

[Tom Ogilvy]

I just wrote a 2.5MB file from excel using a binary write and it was almost
instantaneous. I did it in one write. I would suspect reading would be
faster or comparable. I imagine you problem is your implementation which
won't improve switching to something else. 4 bytes at a time would be slow
I would suspect and probably isn't necessary.

 

[Mark HOlcomb]

Thanks for the response. I wondered if VBA was slower than VB but I gather
from the responses that it is just me. Yes. 4 bytes at at time is not right.
This is a unix binary file so I have been timidly picking my way through it.
Now that I have it decoded I will try and read it into memory (I think the
streamreader is the way?).
Dos thinks it is one long string (I believe) so I am hoping I can somehow
read big chunks into memory then decode long strings ... this is all new
teritory for me.

 

[Tom Ogilvy]

If the file is a text file, (or try it anyway)

use something like this:

Sub readfile()
Dim filebuffer As String
Dim s As String, s1 as String
Dim ff As Long, v as Variant
ff = FreeFile()
Open "H:\xlText\xl_file_text_articles.txt" For Binary Access Read As #1
Debug.Print LOF(ff)
s = String(LOF(ff), " ")
Get #1, 1, s
Close #1
v = Split(s, vbCrLf)


msgbox "Lbound: " & lbound(v) & " and ubound: " & ubound(v)
for i = lbound(v) to 10
s1 = s1 & v(i) & vbNewLine
Next
msgbox s1

End Sub

change vbCrLf to vbLf or vbCr (Unix only uses one, but I don't recall which)

 

[Mark HOlcomb]

I will try that.
I dont think there are any cr lf in the binary file but I'm learning a lot
from your posts. I can adapt what you wrote. The 'good' data is sandwiched
between Hex 0004 (ie 0004 data 0004) Not sure why. My fortran code on the
UNIX side (that runs in seconds for a typical 100Mb file) just reads the data
one line at a time. The dos adaptataion was to look for the 0004 at the
start of a record, figure out if it is character, integer or floating point,
then terminate at the next 0004. The binary file appears to be one long
string (I have a HEX editor to help me here) So maybe I can key in on the
0004 instead of the CRLF using your approach.
Many thanks.

 

[Nicolas Noakes]

Tom,

I seem to use your file number variable "ff" and file number constant #1
interchangably - I would suggest that Mark sticks to using "ff" only.

Regards,
Nicolas

*** Sent via Developersdex http://www.developersdex.com ***

 

[Deke]

I would just like to say a big thank you to everyone who has added comments
to this thread, I was having performance problems with Excel reading a 17meg
text file, I was taking about 5 mins to read the whole file.

With the comments and help added here, I have been able to get this down to
seconds.

Again, a big thank you to everyone.