FPGA Central

I am looking for an open-source smoother to down-sample data. I know
there are many such as Hull Moving average, but I have data logged at
the ms scale at non-fixed time intervals and I want to sample it on
the second range at fixed time intervals. Does anyone know of a good
smoother to do this? The key is that the input data is not logged at a
constant frequency.

Jessica wrote:
> I am looking for an open-source smoother to down-sample data. I know
> there are many such as Hull Moving average, but I have data logged at
> the ms scale at non-fixed time intervals and I want to sample it on
> the second range at fixed time intervals. Does anyone know of a good
> smoother to do this? The key is that the input data is not logged at a
> constant frequency.


Can't answer your specific question. *BUT* I can suggest
information to give so you can get a meaningful answer:

1. What is the average sample rate?
2. What is the minimum and maximum time between samples?
2a. Possibly, what is the distribution of sample intervals?
3. What is your desired fixed "sample rate"?
4. What accuracy constraints of fit of reconstructed signal to
the actual? [Would linear interpolation between points be
adequate or would a 310947 degree polynomial be needed?
[Alright already, the degree was tongue in cheek, but intent
was serious.]

The art of getting "right answer" is asking "right question" ;>
[I *KNOW* from personal embarrassing experience.]

1. Average sample rate is in the ms range, but can vary to seconds /
minutes when there is no activity to be measured.
2. Most of the sample intervals are in the ms range, but accuracy down
to the microsecond is possible. However, at that level, mostly noise
is recorded. I want to downsample to one measurement per second or
even minute.
3. I am not sure how to quantify this, the signal measured does
contain a lot of noise, but ideally, I would like to use a Hull Moving
Average to smooth the data.

I appreciate your help.

On Feb 1, 5:29*pm, Richard Owlett <rowl...@pcnetinc.com> wrote:
> Jessica wrote:
> > I am looking for an open-source smoother to down-sample data. I know
> > there are many such as Hull Moving average, but I have data logged at
> > the ms scale at non-fixed time intervals and I want to sample it on
> > the second range at fixed time intervals. Does anyone know of a good
> > smoother to do this? The key is that the input data is not logged at a
> > constant frequency.
>
> Can't answer your specific question. *BUT* I can suggest
> information to give so you can get a meaningful answer:
>
> 1. What is the average sample rate?
> 2. What is the minimum and maximum time between samples?
> * * 2a. Possibly, what is the distribution of sample intervals?
> 3. What is your desired fixed "sample rate"?
> 4. What accuracy constraints of fit of reconstructed signal to
> * * the actual? [Would linear interpolation between points be
> * * adequate or would a 310947 degree polynomial be needed?
> * * [Alright already, the degree was tongue in cheek, but intent
> * * *was serious.]
>
> The art of getting "right answer" is asking "right question" ;>
> [I *KNOW* from personal embarrassing experience.]

On 2 Feb, 00:37, Jessica <pt...@live.com> wrote:
> I am looking for an open-source smoother to down-sample data. I know
> there are many such as Hull Moving average, but I have data logged at
> the ms scale at non-fixed time intervals and I want to sample it on
> the second range at fixed time intervals. Does anyone know of a good
> smoother to do this? The key is that the input data is not logged at a
> constant frequency.

You need to decide why you want to do this, and what the end
purpose of the analysis is.

To give you an example what I mean, I have a project going
these days, where I monitor a sound source for hours at the
time. The recorder loggs data at 44 kHz for 12 hrs at the time.
The events I am looking for don't happen too often, but when
they do, they are a lot louder than the background noise.

The fist question is to get an overview of the data and see if
the events occur at all, and if so, in which file.

This can be done in a number of ways, including characterizing
the events and run a detector over the data to look for that
particular signature.

But that's not the only way.

To make quick sense of the data (12 hours worth of data
at CD quality is several GBytes of data), this is what I do:

- Make sure the recorder splits the data into manageable
sized files (64 MBytes)
- Load the data from file
- Split the data into second-long chunks
- Compute the energy of the data in each second-long chunk
- Store the vector of energy per second for each file
- Plot these vectors versus file.

If one does this correctly, the plot shows the background noise
level with the events I am looking for standing out as spikes
in the plot. From the plot I can easily identify exactly
which of the recorded files contain the event, and thus
find the exact data segment and go on with further analysis.

If you try and explain to yourself, your boss, or your
colleague, exactly what you want to obtain from whatever
procedure you are struggling with, you might find the clues
to the answer.

The only thing I can tell you right now is that resampling
a data set from the ms scale down to the minutes scale,
doesn't make much sense from the DSP point of view.

The constraints that govern DSP deal with downsampling
factors 2-10. You are talking about several thousands.

Rune

Dear sir,,

For the Christmas is coming,all the products in our company are sold with the promotion prices,welcome to our website Nike Shoes, Air Jordans Shoes, Wholesale Nike, UGG shoes, mobile phone, ED hardy, Hair Straightener, Jordan Sneakers to choose the products you require~

Tesco Trade Co., Ltd is a manufactory & wholesale and retail company.
We always supply superior quality,prompt delivery and excellent service!

We have many brand products,such as brand mobile phones,blackberry,sony,iphone,apple phones,Motorola,nokia and HTC etc.

We also have brand products such as handbags, t-shirts, hoodies, down jackets, dust coat, watches, belts, sunglasses, mobile phones, laptops etc.

Any questions please feel free to ask me,

MSN: google2008e@hotmail.com
Yahoo:google2008e@yahoo.cn

Enjoy your time in our website~Looking forward to our cooperation~

Happy New Year~

Best wishes,

Sandy

Jessica wrote:
> 1. Average sample rate is in the ms range, but can vary to seconds /
> minutes when there is no activity to be measured.
> 2. Most of the sample intervals are in the ms range, but accuracy down
> to the microsecond is possible. However, at that level, mostly noise
> is recorded. I want to downsample to one measurement per second or
> even minute.
> 3. I am not sure how to quantify this, the signal measured does
> contain a lot of noise, but ideally, I would like to use a Hull Moving
> Average to smooth the data.
>
> I appreciate your help.
>
> On Feb 1, 5:29 pm, Richard Owlett <rowl...@pcnetinc.com> wrote:
>> Jessica wrote:
>>> I am looking for an open-source smoother to down-sample data. I know
>>> there are many such as Hull Moving average, but I have data logged at
>>> the ms scale at non-fixed time intervals and I want to sample it on
>>> the second range at fixed time intervals. Does anyone know of a good
>>> smoother to do this? The key is that the input data is not logged at a
>>> constant frequency.
>> Can't answer your specific question. *BUT* I can suggest
>> information to give so you can get a meaningful answer:
>>
>> 1. What is the average sample rate?
>> 2. What is the minimum and maximum time between samples?
>> 2a. Possibly, what is the distribution of sample intervals?
>> 3. What is your desired fixed "sample rate"?
>> 4. What accuracy constraints of fit of reconstructed signal to
>> the actual? [Would linear interpolation between points be
>> adequate or would a 310947 degree polynomial be needed?
>> [Alright already, the degree was tongue in cheek, but intent
>> was serious.]
>>
>> The art of getting "right answer" is asking "right question" ;>
>> [I *KNOW* from personal embarrassing experience.]
Another question: How does the sampling rate compare to the bandwidth of
your data?

Jerry
--
Engineering is the art of making what you want from things you can get.
¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯ ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯

Jerry wrote:
>Jessica wrote:
>> 1. Average sample rate is in the ms range, but can vary to seconds /
>> minutes when there is no activity to be measured.
>> 2. Most of the sample intervals are in the ms range, but accuracy down
>> to the microsecond is possible. However, at that level, mostly noise
>> is recorded. I want to downsample to one measurement per second or
>> even minute.
>> 3. I am not sure how to quantify this, the signal measured does
>> contain a lot of noise, but ideally, I would like to use a Hull Moving
>> Average to smooth the data.
>>
>> I appreciate your help.
>>
>> On Feb 1, 5:29 pm, Richard Owlett <rowl...@pcnetinc.com> wrote:
>>> Jessica wrote:
>>>> I am looking for an open-source smoother to down-sample data. I know
>>>> there are many such as Hull Moving average, but I have data logge
at
>>>> the ms scale at non-fixed time intervals and I want to sample it on
>>>> the second range at fixed time intervals. Does anyone know of a good
>>>> smoother to do this? The key is that the input data is not logged a
a
>>>> constant frequency.
>>> Can't answer your specific question. *BUT* I can suggest
>>> information to give so you can get a meaningful answer:
>>>
>>> 1. What is the average sample rate?
>>> 2. What is the minimum and maximum time between samples?
>>> 2a. Possibly, what is the distribution of sample intervals?
>>> 3. What is your desired fixed "sample rate"?
>>> 4. What accuracy constraints of fit of reconstructed signal to
>>> the actual? [Would linear interpolation between points be
>>> adequate or would a 310947 degree polynomial be needed?
>>> [Alright already, the degree was tongue in cheek, but intent
>>> was serious.]
>>>
>>> The art of getting "right answer" is asking "right question" ;>
>>> [I *KNOW* from personal embarrassing experience.]
>
>Another question: How does the sampling rate compare to the bandwidth o

>your data?

If this is the same application, and if I'm understanding correctly, ther
is no continuous form of the data:

http://groups.google.com/group/comp....25715e0becd9c/

Knowing the bandwidth would probably boil down to deciding whic
frequencies are "noise", but that doesn't necessarily mean a low-pas
filter is a good idea.

Jessica: what is stopping you from using the HMA that you want to use?
Is it a question of whether it's the right tool? I'm not familiar wit
this estimator (other than reading an article on it just now that gives
formula, but not much insight), but I would think it'd be important to com
up with a model (this is an art, and if you get it wrong, the results wil
probably not tell you; Rune has a lot of good stuff to say about this
of both the trends you want to track, as well as the "noise" you don't.
You might look for: random walks, periodic components (once a day, once a
hour, etc), random ramps, etc..

I appreciate all the advice. After doing more research, I have
determined that I am looking for a filter with "non-uniform
sampling". I also found this thread interesting:

http://www.dsprelated.com/showmessage/4806/2.php

On Feb 2, 9:30*am, "Michael Plante" <michael.pla...@gmail.com> wrote:
> Jerry wrote:
> >Jessica wrote:
> >> 1. Average sample rate is in the ms range, but can vary to seconds /
> >> minutes when there is no activity to be measured.
> >> 2. Most of the sample intervals are in the ms range, but accuracy down
> >> to the microsecond is possible. *However, at that level, mostly noise
> >> is recorded. *I want to downsample to one measurement per second or
> >> even minute.
> >> 3. I am not sure how to quantify this, the signal measured does
> >> contain a lot of noise, but ideally, I would like to use a Hull Moving
> >> Average to smooth the data.
>
> >> I appreciate your help.
>
> >> On Feb 1, 5:29 pm, Richard Owlett <rowl...@pcnetinc.com> wrote:
> >>> Jessica wrote:
> >>>> I am looking for an open-source smoother to down-sample data. I know
> >>>> there are many such as Hull Moving average, but I have data logged
> at
> >>>> the ms scale at non-fixed time intervals and I want to sample it on
> >>>> the second range at fixed time intervals. Does anyone know of a good
> >>>> smoother to do this? The key is that the input data is not logged at
> a
> >>>> constant frequency.
> >>> Can't answer your specific question. *BUT* I can suggest
> >>> information to give so you can get a meaningful answer:
>
> >>> 1. What is the average sample rate?
> >>> 2. What is the minimum and maximum time between samples?
> >>> * * 2a. Possibly, what is the distribution of sample intervals?
> >>> 3. What is your desired fixed "sample rate"?
> >>> 4. What accuracy constraints of fit of reconstructed signal to
> >>> * * the actual? [Would linear interpolation between points be
> >>> * * adequate or would a 310947 degree polynomial be needed?
> >>> * * [Alright already, the degree was tongue in cheek, but intent
> >>> * * *was serious.]
>
> >>> The art of getting "right answer" is asking "right question" ;>
> >>> [I *KNOW* from personal embarrassing experience.]
>
> >Another question: How does the sampling rate compare to the bandwidth of
> >your data?
>
> If this is the same application, and if I'm understanding correctly, there
> is no continuous form of the data:
>
> http://groups.google.com/group/comp....hread/da25715e...
>
> Knowing the bandwidth would probably boil down to deciding which
> frequencies are "noise", but that doesn't necessarily mean a low-pass
> filter is a good idea.
>
> Jessica: *what is stopping you from using the HMA that you want to use?
> Is it a question of whether it's the right tool? *I'm not familiar with
> this estimator (other than reading an article on it just now that gives a
> formula, but not much insight), but I would think it'd be important to come
> up with a model (this is an art, and if you get it wrong, the results will
> probably not tell you; Rune has a lot of good stuff to say about this )
> of both the trends you want to track, as well as the "noise" you don't.
> You might look for: *random walks, periodic components (once a day, once an
> hour, etc), random ramps, etc..

I don't see anything in that thread about non-uniform sampling. I
looked at the PDF files linked to in the thread and I don't see
anything amazing. I would suggest that the author is comparing his
own product to standard filters without us knowing anything about how
they were set up. All filters have parameters that are adjusted to
suit the needs. As to the specific examples, how do you know if the
data was selected to show their filter at its best while other data
might show it as inferior to the others?

Just like when a CPU vendor benchmarks his chips against a competitor,
they don't give the competitor a chance to set up their machine, the
vendor doing the benchmark sets up both machines and obviously tweeks
his own as best he can.

Rick


On Feb 3, 7:18*pm, Jessica <pt...@live.com> wrote:
> I appreciate all the advice. *After doing more research, I have
> determined that I am looking for a filter with "non-uniform
> sampling". *I also found this thread interesting:
>
> http://www.dsprelated.com/showmessage/4806/2.php
>
> On Feb 2, 9:30*am, "Michael Plante" <michael.pla...@gmail.com> wrote:
>
> > Jerry wrote:
> > >Jessica wrote:
> > >> 1. Average sample rate is in the ms range, but can vary to seconds /
> > >> minutes when there is no activity to be measured.
> > >> 2. Most of the sample intervals are in the ms range, but accuracy down
> > >> to the microsecond is possible. *However, at that level, mostly noise
> > >> is recorded. *I want to downsample to one measurement per second or
> > >> even minute.
> > >> 3. I am not sure how to quantify this, the signal measured does
> > >> contain a lot of noise, but ideally, I would like to use a Hull Moving
> > >> Average to smooth the data.
>
> > >> I appreciate your help.
>
> > >> On Feb 1, 5:29 pm, Richard Owlett <rowl...@pcnetinc.com> wrote:
> > >>> Jessica wrote:
> > >>>> I am looking for an open-source smoother to down-sample data. I know
> > >>>> there are many such as Hull Moving average, but I have data logged
> > at
> > >>>> the ms scale at non-fixed time intervals and I want to sample it on
> > >>>> the second range at fixed time intervals. Does anyone know of a good
> > >>>> smoother to do this? The key is that the input data is not logged at
> > a
> > >>>> constant frequency.
> > >>> Can't answer your specific question. *BUT* I can suggest
> > >>> information to give so you can get a meaningful answer:
>
> > >>> 1. What is the average sample rate?
> > >>> 2. What is the minimum and maximum time between samples?
> > >>> * * 2a. Possibly, what is the distribution of sample intervals?
> > >>> 3. What is your desired fixed "sample rate"?
> > >>> 4. What accuracy constraints of fit of reconstructed signal to
> > >>> * * the actual? [Would linear interpolation between points be
> > >>> * * adequate or would a 310947 degree polynomial be needed?
> > >>> * * [Alright already, the degree was tongue in cheek, but intent
> > >>> * * *was serious.]
>
> > >>> The art of getting "right answer" is asking "right question" ;>
> > >>> [I *KNOW* from personal embarrassing experience.]
>
> > >Another question: How does the sampling rate compare to the bandwidth of
> > >your data?
>
> > If this is the same application, and if I'm understanding correctly, there
> > is no continuous form of the data:
>
> >http://groups.google.com/group/comp....hread/da25715e...
>
> > Knowing the bandwidth would probably boil down to deciding which
> > frequencies are "noise", but that doesn't necessarily mean a low-pass
> > filter is a good idea.
>
> > Jessica: *what is stopping you from using the HMA that you want to use?
> > Is it a question of whether it's the right tool? *I'm not familiar with
> > this estimator (other than reading an article on it just now that givesa
> > formula, but not much insight), but I would think it'd be important to come
> > up with a model (this is an art, and if you get it wrong, the results will
> > probably not tell you; Rune has a lot of good stuff to say about this )
> > of both the trends you want to track, as well as the "noise" you don't.
> > You might look for: *random walks, periodic components (once a day, once an
> > hour, etc), random ramps, etc..
>
>

In a not-so-similar situation, I was able to use a Lomb-style periodogra
to "regularize" my data, creating an approximated evenly-sampled tim
series from an interpolated frequency series.

Then I could use all of the standard tools in my toolbox.

But:

(1) This is an expensive way to do data, not for the real-time (dsp?
faint-of-heart. (Are you processing offline?)

(2) You are discussing an extraordinarily ambitious sampled-time domain..
microseconds to minutes... megahertz to millihertz... Wow! Not sur
you'll be able to find the needles in your haystacks.

If you come upon something that works, I'd love to know, and am guessin
others would as well.

Bon chance,

- F.