>>
APLDN Home

>>
Events

>>
Trainings

>>
APL Books

>>
APLDN Links

>>
Discussion Groups

>>
Downloads

>>
Articles

>>
Library

>>
Learning Tools

>>
APLDN User IO

>>
APL2000.com




Features

Author Thread: Colossal File Componet Structure
brent.hildebrand
Colossal File Componet Structure
Posted: Monday, May 24, 2004 7:53 PM (EST)

The structure of components in Colossal Files is discussed in the documentation in some detail, such that reconstruction of parts of a file that is damaged might be possible.  What I am trying to understand is the format of the TimeStamp.  I do not seem to be able to duplication the TimeStamp to match that returned by the []cfrdci function.   So some questions about Colossal Files.

1) what is the format of the TimeStamp in components as stored on in the file?

2) are there functions to read the transaction number?  It can be read from the file, once the pointer to a component if found, but no []cf... type function appears to return this number. 

 

Here is a snipet of code that reads a component given a pointer to the start byte of the component.  My TimeStamp does not work.  The rest seems to be OK.

 

    ’ larg MEntry rarg;Œio;component;comp_size;comp_num;data_size;timestamp;transact;md5;integrity;comp
[1]   © rarg = pointer to start of component in Colossal File
[2]   © if larg = single negative number, == native tie number
[3]   © if larg = vector, larg is the file data
[4]   Œio„0
[5]   :if (1=½,larg)
[6]   :andif larg<0
[7]   ©    do file operations
[8]       :if ~0 221 238 255­AV Œnread larg,82,4,rarg ª 'Not a component' ª …0 ª :end
[9]       comp_size„Œnread larg,323,1,rarg+4
[10]      comp„Œnread larg,82,comp_size,rarg
[11]  :else
[12]  ©    do vector operations
[13]      :if ~0 221 238 255 ­AV larg[rarg+¼4] ª 'Not a component' ª …0 ª :end
[14]      comp_size„323 Œdr larg[rarg+4+¼4]
[15]      comp„larg[rarg+¼comp_size] © component
[16]  :end
[17]
[18]  data_size„323 Œdr comp[8+¼4]
[19]  comp_num„323 Œdr comp[12+¼4]
[20]  ©    timestamp„645 ts„Œdr comp[16+¼8]
[21]  timestamp„256ƒ²AV ts„comp[16+¼8]
[22]  transact„256ƒ²AV comp[24+¼8]
[23]  md5„AV comp[32+¼16]
[24]  integrity„md5­AV 82 Œdr 'md5' Œdr data„comp[48+¼data_size]
[25]
[26]  'comp_size' comp_size
[27]  'data_size' data_size
[28]  'comp_num' comp_num
[29]  'timestamp' timestamp
[30]  'transact' transact
[31]  'md5' md5
[32]  'integrity' integrity
    ’


Comments:

Author Thread:
brent.hildebrand
Colossal File Componet Structure
Posted: Monday, May 24, 2004 8:02 PM (EST)
Hmm, the Match character does not show in the above code such as in [8], [13] and [24] , but a copy and paste into APL does show it.  Interesting.

     

j.merrill
Colossal File Componet Structure
Posted: Tuesday, May 25, 2004 10:04 AM (EST)
If I remember right, the timestamp is stored as a 64-bit integer, the number of ?milliseconds? since some particular point in time. (2*32) basevalue 323 #DR comp[16+iota 8] probably gives you what you need (or you might need to reverse the #DR result). Good luck...

     

Mark.Osborne
Colossal File Componet Structure
Posted: Tuesday, May 25, 2004 12:44 PM (EST)
Timestamps are stored internally as 64 bit integer data in 100 nanosecond increments from a January 1, 1601 epoch (UTC). They are returned as 64 bit floating point values in microsecond increments from a 00:00 01 Jan, 1900 epoch (UTC). The range limit is imposed by the stored form to approximately the year 30,828. But the code test has that rounded to the year 30,000.

     

Mark.Osborne
Colossal File Componet Structure
Posted: Tuesday, May 25, 2004 12:46 PM (EST)
One more detail. The timestamp as stored is precisely the result of the Windows API call GetSystemTimeAsFileTime(); /Mark

     

brent.hildebrand
Colossal File Componet Structure
Posted: Tuesday, May 25, 2004 1:10 PM (EST)

Thank you Mark.  Got it...

 

 

     

brent.hildebrand
Colossal File Componet Structure
Posted: Tuesday, May 25, 2004 1:11 PM (EST)

Thank you Mark.  Got it...

 

 

     

brent.hildebrand
Colossal File Componet Structure
Posted: Tuesday, May 25, 2004 3:52 PM (EST)

Works now, thank you Mark.

      Œcfrdci 1 1
64 1 3293996701785000 2004 5 19 23 5 1 785
      Œwcall 'FileTimeToSystemTime' ts Ð
 1  2004 5 3 19 23 5 1 785

 

I observe that Component 0 also has time information, but []CFRDCI does not work for component 0.  Is there a reason for not having []CFRDCI work for component 0?    With a test file created with []CFCREATE, and putting data into component 0,  a pointer can be found to the 0 component, and it has a timestamp:

      a MEntry 101752
 comp_size  88
 data_size  40
 comp_num  0
 timestamp  1.272948147416566E17
 transact 1
 md5  181 88 79 60 159 74 114 247 88 31 179 30 148 167 86 61
 integrity 1
      Œwcall 'FileTimeToSystemTime' ts Ð
 1  2004 5 3 19 23 4 34 165

 

Of course, []CFRDCI does not work as per the documentation:

      Œcfrdci 1 0
But, the information is no doubt there.  So why not return it with []CFRDCI?

     

Mark.Osborne
Colossal File Componet Structure
Posted: Tuesday, May 25, 2004 4:00 PM (EST)
<P>The next build will have a CFRDCI result for component 0. Currently there is timestamp information for component 0 in CFHIST</P> <P>/Mark</P>

     

brent.hildebrand
Colossal File Componet Structure
Posted: Tuesday, May 25, 2004 9:47 PM (EST)

Ah - CFHIST has the TimeStamp for component 0.  I missed that. 

 

The documentation talks about transaction numbers, and yes I find them in the file structure, but there is no function interface to them from APL, is that correct?  The transaction number is facinating from the fact that one can know how many a file has been modified.  But they currently are not accessible unless you disect the file as in the function above. 

     

Mark.Osborne
Colossal File Componet Structure
Posted: Wednesday, May 26, 2004 10:27 AM (EST)
The transaction number was included to provide a way to distinguish the latest version of multiple components claiming the same component number during file recovery. This short of marking the expired version of the component which would incur additional write overhead and chance for damage if things went wrong. Transaction numbers track drop, replace, and append. There is a "highest transaction" stored in the header, as well as a transaction number stored with each component. It would be reasonable to retrieve those via CFINFO and CFRDCI if there is sufficient use for them.

     

Brian.Chizever
Colossal File Componet Structure
Posted: Wednesday, May 26, 2004 11:28 AM (EST)

I, personally, can't think of any reason why the end APL developer would ever need this. 

 

The only time this is needed is if you're repairing a damaged file by doing native calls, in which case you'll get it that way.

 

-Brian

     

brent.hildebrand
Colossal File Componet Structure
Posted: Thursday, May 27, 2004 11:39 AM (EST)
You are probably right Brian, the use may be minimal.  In fact, the component TimeStamp may serve the same function as the transaction number in knowing the historical sequence of a component in a broken file where two components claim to be the same component; they should have different transation numbers and timestamps in the same accending direction of magnitude.   The difference is the transaction number gives information about the number of times a file has been altered.  I would find it interesting to know if a file has been modified 50 times or 500 million times.  The 5.2 beta documentation is the most I've seen on the internal structure of  a component file I've seen.  The component portion is desribed quite well.  My warped mind like knowing how this stuff works.  And the data is there for transation numbers, why not give simple access to it?

     

William.Rutiser
Colossal File Componet Structure
Posted: Thursday, May 27, 2004 11:51 AM (EST)
Note that transaction numbers are not affected by time warps due to clock synchronization problems. I don't think there is or should be any guarantee that transaction numbers will never be skipped.

     

Brian.Chizever
Colossal File Componet Structure
Posted: Thursday, May 27, 2004 12:05 PM (EST)

Brent,

 

The transaction numbers are there for internal purposes.  There's a lot of stuff in the file that isn't exposed because it's only used internally.

 

If you only want to see this because you would "find it interesting", you can get that using the native calls.

 

Exposing this to the user impacts many things.  It must be coded (new quad function or change to an existing one) documented and supported.  If the end user (the APL developer) isn't going to use it, why bother?

     



APL2000 Official Web Site

To err is human, but when the eraser wears out ahead of the pencil, you're overdoing it.
- Josh Jenkins

APLDN Home   |    |  Events   |  Trainings   |  APL Books   |  APLDN Links   |    |  Discussion Groups   |    |  Downloads   |  Articles   |  Library   |  Learning Tools   |  APLDN User IO   |  APL2000.com   |