[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Proposal: New XML Storage Engine



Hi Lars,

You can find info about their architecture at this link:

http://xml.darmstadt.gmd.de/xql/xql-examples.html

Overview of the PDOM Features

      Caching: A PDOM file is organized in pages, each containing 128 DOM nodes of variable

      length. When a PDOM Node is accessed by a W3C-DOM method, the containing page is
      loaded into a main memory cache. Starting with a default cache size of 100 pages
(12.800
      DOM Nodes), the main memory cache can be resized any time. It will, however, never
      shrink below 20 pages (2.560 DOM Nodes). It is recommended to use the largest cache
      size your machine's main memory can hold without swapping, as a larger cache improves

      overall PDOM performance. The same cache is shared by all PDOM documents opened
      with the same instance of the PDOM engine. The caching strategy used is "least
recently
      used" (LRU).

      Defragmentation: When a node is programmatically inserted, updated or delete by
      W3C-DOM methods, the page containing the node is invalidated ("dirty page"). If a
dirty
      page is displaced from the cache, the modified page is appended at the end of the
PDOM
      file. So a PDOM file will grow during write operations, as the file space occupied by

      invalidated pages will not be removed or reused automatically. Note that just reading
and
      or querying a PDOM file, however, will never change the file size.

      The PDOM file can be defragmented at any time by removing unused pages. During this
      operation a temporary file containing only valid pages is created and finally the
      fragmented PDOM file is replaced with the unfragmented copy. It is possible to define
the
      directory where the temporary file is created. The slack ratio, that is the
percentage of
      wasted file space divided by physical file size can be accessed by user applications.
The
      number is normalized to a double between 0.0 and 1.0. It is up to the user
application to
      start a defragmentation, probably if the slack ratio grows beyond a tolerable mark.

      Full garbage collection: Defragmentation does work on a per-page basis and does not
      free space occupied by DOM nodes that have been deleted within pages. To also free
this
      space, a full garbage collection is required. To avoid dangeling object references, a

      garbage collection is only safe if the PDOM file is not opened by another PDOM engine
and
      no PDocument object is currently bound to the PDOM file. This also includes any child

      nodes of PDocument, which may still be in main memory left from previous operations.
It
      is the duty of the user application to enforce this conditions, else you are in
danger to
      garble the PDOM file. Full garbage collection includes defragmentation.

      Commit points: At any time a user application doing update, delete or insert
operations
      on a PDOM can decide to commit the current status quo of the PDOM. In the commit
      operation the main file index, normally maintained in main memory, is written back to

      disk. If the user application crashes, e.g. because of a "disk full" error, the PDOM
will be in
      the state is was immediately before the last successful commit operation when
re-opened.
      Great care was taken to ensure file consistency even after crashes. There is,
however, a
      minimal chance of corrupting a file if the user application dies during a commit
operation.
      Keep in mind that the PDOM does not try to be a fully fledged database.

      Compression with gzip: Optionally a PDOM file can be compressed on the fly using the
      gzip algorithm. This will result in smaller files, usually half the size of an
uncompressed
      PDOM file. The tradeoff here is speed: a compressed PDOM file usually increases the
      execution for reading and writing pages by 20%. Compression is a one time decision
take
      at creating time of the PDOM file. A file can not be compressed later. All operations

      opening PDOM files will automatically recognize compression and handle this fact
      transparently. User applications never have to care or know about compression when
      dealing with existing PDOM files.

      Multithreaded access: The same PDOM file can be read by multiple threads in parallel
      without problem. Update operations block read and write operations for other threads.

      Given this, all atomic operations on a PDOM file are thread safe. However, composed
      update operations (e.g. reading a node, modifying it and write back to the PDOM)
suffer
      from from the well known transaction difficulties. To ensure atomicity of complex
updates,
      the application has to synchronize the critical block of code with the PDocument
object.


Lars Martin wrote:

> Hi Zvi,
>
> thanks for your fast reply.
>
> On Sam, 12 Aug 2000, Zvi wrote:
> > looks similar to GMD-IPSI PDOM implemetation. Only they using Index of clusters,
> > instead of your scheme, where each cluster points to next cluster.
> > Each node can be identified by Number-of-Cluster in the document and Number-of-Node
> > in the cluster.
>
> Uhh, from where did you get such internal information about GMD-IPSI? Is there
> any whitepaper out there? Are these information available for public?
>
> Anyway, we have try to adapt ozones architecture. And ozone uses cluster for
> object pooling - so we now have ozone clusters containing node clusters. This
> fits much more better into the ozone architecture then thousands of individual
> nodes.
>
> BTW: It is not written anywhere that one may not copy good ideas. ;-)
>
> > There are will be issues of grabage collection - you can not delete Cluster, until at
> > least one node used.
>
> You are right. But the underlaying storage backend is responsible for doing it
> in the most clever way. ;-)
>
> > Also this violate Ozone policy of "object never leave server", b/c Cluster will not
> > live inside server, but wrappers for individual nodes will live on client side.
>
> You saw your mistake already. ;-)
>
> > How transactions will be implemented? Per-node or per-cluster?
>
> Do you mean locks? Normally I would say 'per-cluster' but this highly depends
> on the underlaying storage backend - wizard-store or something else. This
> underlaying storage backend is responsible for locking. I think Falko can
> explain this much better ...
>
> Regards,
> Lars
> --
> ___________________________________________________________________
> Lars Martin                                 mailto:lars@smb-tec.com
> SMB GmbH                                     http://www.smb-tec.com