Photo Storage

D

DavidSt

I have read all the arguments for linking photos and not housing them in an
access database but have a need to house them in a shell "photo" database. I
then LINK that shell database table into my main database and all is
well......Now the shell database is already at 1GB and I anticipate I will
need 3 or 4 GB to get them all in. The most I can store in a single database
is 2GB. I have read that you can create 3 databases each with 1 GB of photos
and link them together via an index in a 4th database.

How do you do this? Ultimately I would like to LINK the 4Th index databse
into my main datbase but am at a loss for how to properly link the three into
the fourth (index) database? Help Please.
 
T

Tom Wickerath

David,

If you've read all the arguments against embedding OLE Objects (images,
sound, documents, etc.), then why would you still be convinced that you have
a need to store such objects in a .mdb file? A JET database is simply a very
inefficient storage mechanism. A better bet is to store these files in
folders, and store the path + filename in the database. You can either store
the path and filename in two separate fields, and concatenate them when
needed, or store the entire string in one field.

Beware that any type of OLE Object field is much more suseptible to
corruption than other data types. If I still haven't convinced you then
perhaps you should purchase DBPix ( http://www.ammara.com/ ). It is reported
to get around the problems associated with OLE Objects. I have no personal
experience with this product, so I'm not necessarily endorsing it, I'm just
providing a link to assist you in investigating further for yourself. There
is a fully functional evaluation copy that you can download.

Tom

http://www.access.qbuilt.com/html/expert_contributors.html
__________________________________________

:

I have read all the arguments for linking photos and not housing them in an
access database but have a need to house them in a shell "photo" database. I
then LINK that shell database table into my main database and all is
well......Now the shell database is already at 1GB and I anticipate I will
need 3 or 4 GB to get them all in. The most I can store in a single database
is 2GB. I have read that you can create 3 databases each with 1 GB of photos
and link them together via an index in a 4th database.

How do you do this? Ultimately I would like to LINK the 4Th index databse
into my main datbase but am at a loss for how to properly link the three into
the fourth (index) database? Help Please.
 
B

bob

A JET database is simply a very inefficient storage mechanism

Jet itself is not inefficient, the ‘OLE Object’ field-type is simply a binary field that can store any
binary info with negligible overhead, particularly when compared to the file-system. In fact the page
allocation size is smaller than typical file-system cluster sizes, leading to less slack space with many
or small items. What is (or can be) inefficient is using ‘OLE Embedding’ with compressed images. This
is entirely a front-end technology that is independent of the back-end storage. The same inefficiency
will occur when using OLE Embedding with a back-end such as SQL Server, Oracle or MySQL binary fields.


As for corruption, if problems were due to random file-system or hardware errors, then of course having
a larger file would increase the likelihood of corruption, but this is extremely rare, and problems are
more typically caused by locking errors or front-end problems. Our experience of working with thousands
of binary storage systems suggest that provided ‘OLE Embedding’ is avoided, corruption issues are no more
common than with non-binary databases, and with journaling file-systems such as NTFS that is extremely
rare.
 
T

Tom Wickerath

Bob,
Jet itself is not inefficient, the ‘OLE Object’ field-type is simply a binary field
that can store any binary info with negligible overhead, particularly when
compared to the file-system.

A fine point. Perhaps I should have said Access used via the user interface,
with a JET database, is simply a very inefficient storage mechanism. While
the same may be true about other BE databases used with Access, I choosing to
limit my comments in this newsgroup to the most widely used engine with
Access.

I stand by my statement of increased chances of corruption with any OLE
Object datatype. A record in a JET 4 database is limited to 2 KB. Memo and
OLE Object datatypes get around this limitation by storing a pointer to the
data for the memo or OLE Object. The memo or OLE Object data is not stored in
the same table, as far as I know. Corruption frequently occurs when these
pointers lose track of the data that they're suppose to be pointing to. In
memo fields, this often times shows up as strange characters.

Since you appear to work for Ammara, or be associated with them in some way,
my question is why can't I find technical reviews of your product published
by independent sources? I see a lot of testimonials from people I've never
heard of on this page:

http://www.ammara.com/testimony.html

I even noticed 4 links at the bottom of this page:

http://www.ammara.com/articles/accesspictureole.html

that point to some very familiar sites, including Tony Toews page, The
Access Web, Larry Linsons page and Stephen Lebans site. However, on each of
these pages, I find precious little about DBPix or Ammara. I did see a link
on Tony's page which points back to Ammara:

Images - Access image storage is a good article on how OLE Objects work.
http://www.ammara.com/articles/imagesaccess.html

I can't say that I've ever seen anything in publications such as Smart
Access, Inside Microsoft Access, Access-VB-SQL Advisor magazine, etc.
regarding DBPix. Perhaps I've just missed all those reviews. If you know of
such reviews, then by all means please provide links to them. Perhaps you can
invite Danny Lesandrini to review this product and post an evaluation at:

http://www.amazecreations.com/datafast/article.aspx


Tom

http://www.access.qbuilt.com/html/expert_contributors.html
__________________________________________

bob said:
A JET database is simply a very inefficient storage mechanism

Jet itself is not inefficient, the ‘OLE Object’ field-type is simply a
binary field that can store any binary info with negligible overhead,
particularly when compared to the file-system. In fact the page allocation
size is smaller than typical file-system cluster sizes, leading to less slack
space with many or small items. What is (or can be) inefficient is using
‘OLE Embedding’ with compressed images. This is entirely a front-end
technology that is independent of the back-end storage. The same inefficiency
will occur when using OLE Embedding with a back-end such as SQL Server,
Oracle or MySQL binary fields.


As for corruption, if problems were due to random file-system or hardware
errors, then of course having a larger file would increase the likelihood of
corruption, but this is extremely rare, and problems are more typically
caused by locking errors or front-end problems. Our experience of working
with thousands of binary storage systems suggest that provided ‘OLE
Embedding’ is avoided, corruption issues are no more common than with
non-binary databases, and with journaling file-systems such as NTFS that is
extremely rare.

--
_______________________________________________________
http://www.ammara.com/ ;
Image Handling Components, Samples, Solutions and Info
DBPix 2.0 - lossless jpeg rotation, EXIF, asynchronous

__________________________________________

David,

If you've read all the arguments against embedding OLE Objects (images,
sound, documents, etc.), then why would you still be convinced that you have
a need to store such objects in a .mdb file? A JET database is simply a very
inefficient storage mechanism. A better bet is to store these files in
folders, and store the path + filename in the database. You can either store
the path and filename in two separate fields, and concatenate them when
needed, or store the entire string in one field.

Beware that any type of OLE Object field is much more suseptible to
corruption than other data types. If I still haven't convinced you then
perhaps you should purchase DBPix ( http://www.ammara.com/ ). It is reported
to get around the problems associated with OLE Objects. I have no personal
experience with this product, so I'm not necessarily endorsing it, I'm just
providing a link to assist you in investigating further for yourself. There
is a fully functional evaluation copy that you can download.

Tom

http://www.access.qbuilt.com/html/expert_contributors.html
__________________________________________

:

I have read all the arguments for linking photos and not housing them in an
access database but have a need to house them in a shell "photo" database. I
then LINK that shell database table into my main database and all is
well......Now the shell database is already at 1GB and I anticipate I will
need 3 or 4 GB to get them all in. The most I can store in a single database
is 2GB. I have read that you can create 3 databases each with 1 GB of photos
and link them together via an index in a 4th database.

How do you do this? Ultimately I would like to LINK the 4Th index databse
into my main datbase but am at a loss for how to properly link the three into
the fourth (index) database? Help Please
 
B

bob

My reason for replying to your post is that statements like ‘Access/Jet is inefficient at storing images’
are so frequent here that many people actually believe just that, without understanding that it only applies
to a specific technique, or that there is another way which is far more suitable, and is indeed the simple
and obvious way to do it (storing the binary directly in a binary field, often what people actually assumed
they were doing when they used OLE Embedding anyway).

Jet is not inefficient at storing images, Access is not inefficient at storing images, OLE Objects are
not inefficient at storing images. What *can* be inefficient, is OLE Embedding, a technique that is now
becoming defunct in this context since Microsoft withdrew the typical OLE Server used for images.

I don’t believe that it is the right place to discuss how our product is marketed, and I neither mentioned
nor advocated our product, but I will try to answer your questions. If you have more, I suggest that this
goes offline. I am here for technical discussion, not explaining why we link to some site or other.
I even noticed 4 links at the bottom of this page:

The links you mention (Tony, Arvin, Stephen, Larry etc) are present because they are excellent resources
that are extremely relevant to our site’s content, and therefore likely to be of specific interest to visitors
researching the subject. Several of them even include non-commercial alternatives to our product, allowing
our visitors to make an informed choice from the available options. Arvin reviewed and tested DBPix for
inclusion in The Access Web’s Products page, and occasionally suggests it as a solution here on Usenet
(thanks again Arvin).
I see a lot of testimonials from people I've never heard of on this page

As requested, we don’t publish comments from MVPs etc, since it could imply an endorsement or commercial
association. The comments are from users who feel strongly enough to voice their views publicly, but there
are a lot of Access developers out there, so it’s hardly surprising that you haven’t heard of any of this
little subset of them.
I can't say that I've ever seen anything in publications

At this time there aren’t any reviews of our product in the publications you mention, most likely because
we haven’t submitted the product to those editors for review, but thanks for the ideas.
I stand by my statement of increased chances of corruption with any OLE …

I also stand by mine, i.e. in nearly 10 years of working every day with Access/Jet in a binary role, with
thousands of customers on disparate systems, it has been an exceptionally reliable and efficient storage
mechanism, provided standard Access design guidelines are followed (and OLE Embedding avoided). I am not
aware of any data loss whatsoever, and the minute number of database corruptions has been unrelated to
the use of binary, easily recoverable, and typically caused by programming errors or general Access glitches
that apply equally to systems that use binary or don’t.
 
T

Tom Wickerath

Bob,

Does the use of DBPix get around the 2K limit for a record in JET 4,
*without* the use of a pointer to point to data stored external to the table?
If your answer is no, then I stand by my original statements. You stated:
" I am not aware of any data loss whatsoever, and the minute number
of database corruptions has been unrelated to the use of binary, easily
recoverable, and typically caused by programming errors or general
Access glitches that apply equally to systems that use binary or don’t."

I have personally helped many people over the years with corrupted records
that included a memo datatype. The memo field only included text, so I don't
think we are talking about OLE Embedding here. Something apparently caused an
interrupted write operation, or some other glitch caused a pointer to the
external data to be lost. In all your years of using Access, you state that
you've never experienced this type of data corruption? I find that amazing,
if I understand you correctly. Your experiences of such reliability would
mean that the likelihood of anyone ever experiencing the issue described by
Access MVP Allen Browne on this page would be practically non-existant. See
the section titled "Symptom: Memo field contains strange characters" here:

http://allenbrowne.com/ser-47.html

Allen included the following statement:

"Access uses a pointer to another location for the data in large fields
(memo, hyperlink, or OLE Object). If the pointer is written incorrectly, the
field displays garbage."

So, my question to you is does the use of DBPix mean that a pointer will be
included to the data? If the answer is yes, then I consider this to be an
inefficient mechanism that can certainly lead to increased chances of
corruption, especially in the data is located on a file server (ie. a network
is involved).
...and I neither mentioned nor advocated our product,...

The fact that you included a link in your signature to your company, along
with the name of the product, would be consided advocating by many people,
including myself. It certainly is mentioning the product:
_______________________________________________________
http://www.ammara.com/ ;
Image Handling Components, Samples, Solutions and Info
DBPix 2.0 - lossless jpeg rotation, EXIF, asynchronous


The links you mention (Tony, Arvin, Stephen, Larry etc) are present
because they are excellent resources...

No disagreement there!
As requested, we don’t publish comments from MVPs etc, since it could
imply an endorsement or commercial association.

I'd be surprised if any Access MVP would feel too concerned about publishing
a review, good or bad, about any product that is intended for use with
Access. If the product is good, and can help users work around problems, why
would they have any hesitation in saying so? A person can certainly include a
statement indicating whether or not they have any commercial association with
the product in question. As you know, I do not have MVP but I have
recommended certain products in newsgroup posts in the past, specifically
FMS's Total Access Analyzer (this product is well worth the $299 cost) and
Splitter for Microsoft Access for helping to split difficult name data ($39).
If, sometime in the future, I am awarded MVP status, I would hope that I
wouldn't be under any pressure to stop suggesting very useful software.
At this time there aren’t any reviews of our product in the publications you
mention, most likely because we haven’t submitted the product to those
editors for review, but thanks for the ideas.

Kind of seems like a "no-brainer" to me, to submit your product for review
not only to these publications, but also to various Access user groups.
You're welcome.


Tom

http://www.access.qbuilt.com/html/expert_contributors.html
__________________________________________

:

My reason for replying to your post is that statements like ‘Access/Jet is
inefficient at storing images’ are so frequent here that many people actually
believe just that, without understanding that it only applies to a specific
technique, or that there is another way which is far more suitable, and is
indeed the simple and obvious way to do it (storing the binary directly in a
binary field, often what people actually assumed they were doing when they
used OLE Embedding anyway).

Jet is not inefficient at storing images, Access is not inefficient at
storing images, OLE Objects are not inefficient at storing images. What
*can* be inefficient, is OLE Embedding, a technique that is now becoming
defunct in this context since Microsoft withdrew the typical OLE Server used
for images.

I don’t believe that it is the right place to discuss how our product is
marketed, and I neither mentioned nor advocated our product, but I will try
to answer your questions. If you have more, I suggest that this goes
offline. I am here for technical discussion, not explaining why we link to
some site or other.
I even noticed 4 links at the bottom of this page:

The links you mention (Tony, Arvin, Stephen, Larry etc) are present because
they are excellent resources that are extremely relevant to our site’s
content, and therefore likely to be of specific interest to visitors
researching the subject. Several of them even include non-commercial
alternatives to our product, allowing
our visitors to make an informed choice from the available options. Arvin
reviewed and tested DBPix for inclusion in The Access Web’s Products page,
and occasionally suggests it as a solution here on Usenet (thanks again
Arvin).
I see a lot of testimonials from people I've never heard of on this page

As requested, we don’t publish comments from MVPs etc, since it could imply
an endorsement or commercial association. The comments are from users who
feel strongly enough to voice their views publicly, but there are a lot of
Access developers out there, so it’s hardly surprising that you haven’t heard
of any of this little subset of them.
I can't say that I've ever seen anything in publications

At this time there aren’t any reviews of our product in the publications you
mention, most likely because we haven’t submitted the product to those
editors for review, but thanks for the ideas.
I stand by my statement of increased chances of corruption with any OLE …

I also stand by mine, i.e. in nearly 10 years of working every day with
Access/Jet in a binary role, with thousands of customers on disparate
systems, it has been an exceptionally reliable and efficient storage
mechanism, provided standard Access design guidelines are followed (and OLE
Embedding avoided). I am not aware of any data loss whatsoever, and the
minute number of database corruptions has been unrelated to the use of
binary, easily recoverable, and typically caused by programming errors or
general Access glitches that apply equally to systems that use binary or
don’t.
 
B

bob

I consider this to be an inefficient mechanism
Why is this less efficient than the file-system (which uses tables of pointers) ?
you've never experienced this type of data corruption?
Once again, I have never experienced corrupted image data stored in an OLE Object field using our product.
You can be quite sure that paying customers would let us know if they experienced it.
some other glitch caused a pointer to the external data to be lost
I guess our software includes no such glitches.
Does the use of DBPix get around the 2K limit for a record in JET 4
It does nothing to modify the way the data is stored by the engine.
I'd be surprised if any Access MVP would feel...
You commented that you didn't know anyone on our testimonial page. Well, we *were* asked not to publish
an MVP's comments on our site. Fact. And we have respected that wish. And that MVP *has* suggested our
product in newsgroups.
would be consided advocating by many people,
The signature makes clear my association. You suggested our product, not me, and my post was not about
the product in any way. Advocating the product would be to say something like 'our product is ideal for
this'. I did not and do not do that. If you have any comments on how this post was in any way in violation
of the charter of this group I will be most pleased to hear them.
 
C

Chris Mills

I consider this to be an inefficient mechanism
Why is this less efficient than the file-system (which uses tables of pointers) ?
notes:
a) quoting Allen Browne was misleading, in that Allen made no comment whatever
on efficiency, reliability, or anything else apart from an error and a cause,
to do with irrelevant memo fields.
b) address indirection is common to computer programs at ALL levels, hardly
anything is not indirected. There are probably millions/sec at the binary
level. It would seem silly to imagine a memo field has more than a few extra
indirections amongst hundreds at the table-manipulation level. There is no
telling how many disc-writes are involved even in saving "Hello World" in any
particular write in Access. (or even notepad)
c) "efficiency" is usually(?) equated to speed or size. Corruption is a
"glitch". A "glitch" is more likely the longer or the bigger(?), but it's a
"downstream effect". Similarly, corruption will be less likely if smaller
images are stored, one types "AFAIK" instead of "As Far As I Know" in a text
field. Furthermore, Microsoft does not claim that each new Access, which is
twice the size of the previous one, is more corruptible <g> otherwise
Access2.0 would be the best system!
d) the usual alternative is linking to external files. Which contains about as
much indirection as anything else!
e) so don't worry about such a minor issue, I would say.
Once again, I have never experienced corrupted image data stored in an OLE
Object field using our product.
You can be quite sure that paying customers would let us know if they experienced it.
I've done image systems (Access OLE, and the recommended alternative of
storing file names (a type of address indirection)). In no case was corruption
a particular problem (apart from the usual odd glitch associated with
humungous files <heh-heh>, but even that wasn't many). No, the primary reason
for choosing one over another is storage space (at least, it seems to be the
one most complained about).
I guess our software includes no such glitches.

It does nothing to modify the way the data is stored by the engine.
This seems to me a valid issue.

I don't think it's "fair" of Ammara to compare their product with either
embedded or linked OLE storage of images. Because not a single person (who has
posted, AFAIK) recommends OLE imaging of either type.

Ammara should compare their product to the recommended alternative of storing
filenames. IMHO.

It may well be, of course, that the Ammara product is better. I just think
it's a more usefull comparison.

And, naturally, there are some issues with storing squillions of files, which
the poster seemed to be concerned about. If he wants it in a "shell", then
Ammara is a product which certainly claims to do it within that "constraint"
(the constraint of the poster). Whether it does what it says, I suppose is
anyone's guess said:
The signature makes clear my association.
Indeed. No probs whatever. Knowing about relevant products available seems to
be valid usage, so long as they don't go overboard, and aren't "just"
advertising.

Chris
 
B

bob

No, the primary reason for choosing one over another is storage
space (at least, it seems to be the one most complained about).

Precisely. Storage in the database is often dismissed because of the mistaken belief that Access or Jet
or OLE Object fields are inherently ‘inefficient at storing images’ (as in Tom’s statement, which started
off this exciting discussion). However, implemented correctly there is no significant difference between
the total space required to store images in a binary field versus the file-system, and my post was intended
merely to avoid propagating this myth that Jet, Access or OLE Object fields inherently incur a significant
size overhead. (I say ‘no significant’ because there is slack space in both approaches that can slew the
results marginally either way).

As for performance, as you say, the indirection should be trivial, and exists in the file-system anyway.
Additionally, with images stored in the database the file is already open and the disk heads likely have
less far to travel.
I don't think it's "fair" of Ammara to compare their product with
either embedded...

Our product came into being specifically to overcome the problems of OLE Embedding, so comparisons are
used to illustrate that it does not suffer those problems, particularly given the widespread misunderstandings
above. Such comparisons are also comparing like with like, i.e. one way of storing images in the database
with another (however DBPix can also be used with external files if desired). Often, when people hit the
2GB size limit, they immediately turn to using external files, but if they knew that they could keep their
images in the database in maybe 20 MB they might well consider continuing to do so.
This seems to me a valid issue.

I’m not sure I follow you here (which statement is valid?). Variable length data is stored outside the
rest of the table (using that trivial additional level of indirection) in order to keep the fixed length
part, well, a fixed length. Our product works with variable-length long binary data, not fixed-length
short data (which is kept inside the 2K limited part), so why should it get around the limit on the total
number/size of integers, text fields, etc, in a record? (Or is Tom suggesting that the image should be
split into 32 bit chunks and stored in integer fields ? <g>).



Of course, none of this is to say that binary storage is suitable for any particular application, that
is a much wider discussion, and should always be application specific, but I do believe, and it is my experience,
that binary storage can and does provide a perfectly good solution in many cases.

Corruption is a "glitch"

But even then, as for ‘interrupted writes’ causing garbage in fields, I can only imagine this occurring
on a non-journaling file-system. My recommendation is that Access databases are run on journaling file-systems
always, without exception, and maybe this goes a long way to explaining why it’s so ‘amazing’ that I don’t
get corrupted databases.

Bob.
 
C

Chris Mills

Bob,
Thanks for your thoughts. I sometimes "repost" a post I kept from Frank Miller
of PSS (or he was a few years ago), on file-based imaging. As you'd expect of
PSS, it has dozens of links, and is quite long, so I wouldn't repost unless
the original poster wanted it. I just found it quite helpfull for myself
(well, I saved it <g>)

I didn't really mean Ammara was "unfair", more a suggestion, and answered with
your explanation of comparisons. On the 2gb limit, I accept that that's a LOT
further away if OLE problems are indeed solved to the extent of your
comparison stuff, that it's "neater" if held in one encompassing file, and
that you do not seem to shy from recommending file-based images where there is
good reason.

I think you left yourself open for criticism by saying you had never
experienced mdb corruption. Or did you just mean "related to your product"
<g>. I estimate corruption at 1% (of what? time, sites, products? It's just a
pure guess feeling, the number doesn't matter)(some sites never have
corruption and some have it all the time until, what, they "improve" their
network or some "sticky-finger operator" leaves probably?

Anyway, "corrupted" mdb's are 99% of the time (there I go again) recoverable
with Repair. That's because (I read somewhere but can't remember where),
Access marks the mdb as writing and removes the flag afterwards. 99% <g> of
the time it says it's corrupt but in fact it's just the "safety flag". (And I
look real good, coz they ring up with a "corrupted database" and I FIXED IT
!!!)

More importantly, there is seldom forensic evidence to pinpoint a precise
reason for corruption, is there?
Chris
 
T

Tom Wickerath

Hi Stephen,

Somehow I missed receiving a notification e-mail when you posted your reply
five days ago. To be honest, I hadn't come back to this thread, because I've
been reluctant to get into a pissing match with either Bob from Ammara or
Chris Mills, especially regarding my statements that OLE datatypes are more
susceptible to corruption. I was spending some time googling for supporting
evidence, when I came upon this thread and noticed your post.

I certainly value and appreciate your opinions. Thank You for sharing your
experiences with this product. Your $.02 worth means a whole lot more to me
than Ammara's entire page of testimonials.

Tom
_______________________________________

:

Hi Tom,
Don't know how I missed this thread initially but just wanted to say that I
have used the DBPix control and was very satisfied with it. I have also
purchased the control and donated it to a non profit group who required the
functionality contained within the control.

The DBPix control exposes a lot of functionality found nowhere else in
Access land.

Just my $.02
--

HTH
Stephen Lebans
http://www.lebans.com
Access Code, Tips and Tricks
Please respond only to the newsgroups so everyone can benefit.
 
Top