View previous topic :: View next topic |
Author |
Message |
Gabriel
Joined: 03 Aug 2009 Posts: 1067 Location: Panama
|
Data Compression |
Posted: Fri Oct 07, 2011 3:10 pm |
|
|
Hi all,
I'm looking for suggestions...
I have a datalogger I've built and coded,
16F876A, 25AA256 eeprom, DS1305.
It takes a single 10bit reading at programmable intervals, takes a time stamp.
It saves data as follows:
Day, Month, Year, Hour, Minutes, Data Hi-byte, Data Low-Byte.
Code works, everything is nice and great, BUT ;)
I'm spending more memory on time stamp info than data.
I'm sure alot of you have gone through this before... so I'm looking for maybe some simple way to compress the time stamp info.
My idea so far is as follows, worst case Scenario:
Code: |
Time: 31/12/99 - 23:59
Bytes 1 2 3 4 5
data 31 12 99 23 59
binary 11111 1100 1100011 10111 1011001
Bytes 1 2 3 4
Result 11111110 01100011 10111101 1001(0000) <- padded
|
Thus resulting in a one byte saving.... seems like alot of bit shifting for just one byte....
I can save 2 bytes if I include the 2 data bytes in the compression "algorithm" described above.
The idea is to be able to quickly and on the fly compres/decompress data for saving and for printing to screen.
Any thoughts? _________________ CCS PCM 5.078 & CCS PCH 5.093 |
|
![](templates/subSilver/images/spacer.gif) |
temtronic
Joined: 01 Jul 2010 Posts: 9170 Location: Greensville,Ontario
|
|
Posted: Fri Oct 07, 2011 4:21 pm |
|
|
Some ideas....
Using 1-365 for the day of the year takes 2 bytes...
Don't save the year byte...have the filename encoded with that.
Same could be used for the month...ie:filename 1102data.txt means data collected in Feb of 2011.So only 1 byte for day of month....
Depending on how often the data is recorded, if you only sample every 15 minutes, the time can be represented in one byte(24 * 4=96) or if you sample every 6 minutes you'll get maximum samples/day(24hr*60min=1440 minutes/256=5.625). |
|
![](templates/subSilver/images/spacer.gif) |
gpsmikey
Joined: 16 Nov 2010 Posts: 588 Location: Kirkland, WA
|
|
Posted: Fri Oct 07, 2011 6:29 pm |
|
|
Another option (sort of a spin on the way windows/unix etc store time) is to have some arbitrary start time that you could store, then each time stamp is stored as an offset in seconds or minutes or whatever incremental resolution you need. If you used minutes, 16 bits would give you 45 days worth of "time stamps".
mikey _________________ mikey
-- you can't have too many gadgets or too much disk space !
old engineering saying: 1+1 = 3 for sufficiently large values of 1 or small values of 3 |
|
![](templates/subSilver/images/spacer.gif) |
Ttelmah
Joined: 11 Mar 2010 Posts: 19359
|
|
Posted: Sat Oct 08, 2011 2:20 am |
|
|
Yes.
Remember the time.h library, has a mktime function, which returns a time converted to a 32bit count of seconds from a start point (based exactly on the Unix version). Use this on a start date/time to give you a 'beginning of time' marker, and subtract this from the values converted the same way from you sample times. Divide by 60, so you are working in minutes, rather than seconds, and store the low 3 bytes of the int32, and you can handle just under 32 years in 3 bytes.....
Best Wishes |
|
![](templates/subSilver/images/spacer.gif) |
Gabriel
Joined: 03 Aug 2009 Posts: 1067 Location: Panama
|
|
Posted: Sat Oct 08, 2011 6:46 am |
|
|
Hey Guys,
Thanks for your suggestions, i had not thought of storing dates as a day out of 365... nice!
i had however thought of using the latter suggested Win/Unix method... but thought it would be alot of coding... and testing....
but i had no idea there was a time.h library! (never needed it)
ill check that out! Thanks!
i was even thinking of Huffman at some point.... i dont suppose there is a Huffman-Compression.h file laying arround? jejejeje...
Thank you for your suggestions!
G. _________________ CCS PCM 5.078 & CCS PCH 5.093 |
|
![](templates/subSilver/images/spacer.gif) |
Ttelmah
Joined: 11 Mar 2010 Posts: 19359
|
|
Posted: Sun Oct 09, 2011 2:26 am |
|
|
I doubt if Huffman coding would help.
It is normally optimal, where you have a stream of symbols with a probability 'tree' on their occurrence, and some having higher probabilities than others. If you use a pure clock counter, probabilities should be equal for all entries. Result, no saving...
Best Wishes |
|
![](templates/subSilver/images/spacer.gif) |
|