Real-time wavelet denoising

[Poj]

Does anyone have experience developing real-time wavelet denoising softwar
without using any DSP chip? I have a problem with delay time caused b
buffering data and want to reduce the delay time as much as possible. Th
data I am dealing with is 1-D with sampling rate lower than 1kHz. I hav
developed the application in C++. The program reads data into a buffe
sample by sample until the buffer has enough data to perform DWT
eliminate noise, and IDWT. So, the time to wait for data becomes dela
time. In other words, output of my wavelet is behind real time about 4
seconds or so due to low sampling rate and data buffering. There is n
problem with denoising quality. or processing speed. I just want a mor
realistic real-time. Any suggestions or ideas are highly appreciated.

Thank you and Merry Christmas

[Vladimir Vassilevsky]

"Poj" <[email protected]> wrote in message
news:[email protected]...
> Does anyone have experience developing real-time wavelet denoising
software
> without using any DSP chip? I have a problem with delay time caused by
> buffering data and want to reduce the delay time as much as possible. The
> data I am dealing with is 1-D with sampling rate lower than 1kHz. I have
> developed the application in C++. The program reads data into a buffer
> sample by sample until the buffer has enough data to perform DWT,
> eliminate noise, and IDWT. So, the time to wait for data becomes delay
> time. In other words, output of my wavelet is behind real time about 40
> seconds or so due to low sampling rate and data buffering. There is no
> problem with denoising quality. or processing speed. I just want a more
> realistic real-time. Any suggestions or ideas are highly appreciated.

Throw away the popular wavelet nonsense and implement a reasonable denoising
algorithm, like an adaptive filter or some kind of nonlinear smoothing.
What would be the best algorithm depends on what is the signal and what is
the noise. Depending on how much exactly is the "high appreciation", I might
be able to help you.

Vladimir Vassilevsky
DSP and Mixed Signal Consultant
www.abvolt.com

[Poj]

Thanks for your comment.

For your information, I have tried LPF and smoothing filter such a
Stavisky-Golay on Matlab. Both were unable to effectively produce
satisfied output because the frequencies of intrinsic signal and noise ar
very low (<20Hz) and close to each other. The noise I am talking about i
DC baseline swing and at this frequency region it is difficult to use HP
to block it. Also, LPF cutoff is not sharp enough to separate the signa
and noise in this region. I am very happy with the filtering result fro
my wavelet denoising. I just want to know techniques to reduce bufferin
time delay available.


>
>"Poj" <[email protected]> wrote in message
>news:[email protected]...
>> Does anyone have experience developing real-time wavelet denoising
>software
>> without using any DSP chip? I have a problem with delay time caused by
>> buffering data and want to reduce the delay time as much as possible
The
>> data I am dealing with is 1-D with sampling rate lower than 1kHz.
have
>> developed the application in C++. The program reads data into a buffer
>> sample by sample until the buffer has enough data to perform DWT,
>> eliminate noise, and IDWT. So, the time to wait for data becomes delay
>> time. In other words, output of my wavelet is behind real time abou
40
>> seconds or so due to low sampling rate and data buffering. There is no
>> problem with denoising quality. or processing speed. I just want
more
>> realistic real-time. Any suggestions or ideas are highly appreciated.
>
>Throw away the popular wavelet nonsense and implement a reasonabl
denoising
>algorithm, like an adaptive filter or some kind of nonlinear smoothing.
>What would be the best algorithm depends on what is the signal and wha
is
>the noise. Depending on how much exactly is the "high appreciation",
might
>be able to help you.
>
>Vladimir Vassilevsky
>DSP and Mixed Signal Consultant
>www.abvolt.com
>
>
>
>

[Vladimir Vassilevsky]

If you are using a linear filter to resolve the frequencies, then the
incurred delay can't be less then about 1/transition band. It doesn't
matter if the filter is implemented in a direct form, or by FFT, or as a
wavelet shamanism.

It is sad to hear that you couldn't get a basic HPF/LPF right, so you
had to resort to tomtomes and tambourines.

I suggest you to take a different look at the problem without being
carried away by cool buzzwords such as "Savitsky-Golay", "Wavelet",
"Matlab".


Vladimir Vassilevsky
DSP and Mixed Signal Design Consultant
http://www.abvolt.com




Poj wrote:

> Thanks for your comment.
>
> For your information, I have tried LPF and smoothing filter such as
> Stavisky-Golay on Matlab. Both were unable to effectively produce a
> satisfied output because the frequencies of intrinsic signal and noise are
> very low (<20Hz) and close to each other. The noise I am talking about is
> DC baseline swing and at this frequency region it is difficult to use HPF
> to block it. Also, LPF cutoff is not sharp enough to separate the signal
> and noise in this region. I am very happy with the filtering result from
> my wavelet denoising. I just want to know techniques to reduce buffering
> time delay available.
>
>
>
>>"Poj" <[email protected]> wrote in message
>>news:[email protected]...
>>
>>>Does anyone have experience developing real-time wavelet denoising
>>
>>software
>>
>>>without using any DSP chip? I have a problem with delay time caused by
>>>buffering data and want to reduce the delay time as much as possible.
>
> The
>
>>>data I am dealing with is 1-D with sampling rate lower than 1kHz. I
>
> have
>
>>>developed the application in C++. The program reads data into a buffer
>>>sample by sample until the buffer has enough data to perform DWT,
>>>eliminate noise, and IDWT. So, the time to wait for data becomes delay
>>>time. In other words, output of my wavelet is behind real time about
>
> 40
>
>>>seconds or so due to low sampling rate and data buffering. There is no
>>>problem with denoising quality. or processing speed. I just want a
>
> more
>
>>>realistic real-time. Any suggestions or ideas are highly appreciated.
>>
>>Throw away the popular wavelet nonsense and implement a reasonable
>
> denoising
>
>>algorithm, like an adaptive filter or some kind of nonlinear smoothing.
>>What would be the best algorithm depends on what is the signal and what
>
> is
>
>>the noise. Depending on how much exactly is the "high appreciation", I
>
> might
>
>>be able to help you.
>>
>>Vladimir Vassilevsky
>>DSP and Mixed Signal Consultant
>>www.abvolt.com
>>
>>
>>
>>

[John]

On Dec 27, 11:49 am, Vladimir Vassilevsky <[email protected]>
wrote:
> If you are using a linear filter to resolve the frequencies, then the
> incurred delay can't be less then about 1/transition band. It doesn't
> matter if the filter is implemented in a direct form, or by FFT, or as a
> wavelet shamanism.
>
> It is sad to hear that you couldn't get a basic HPF/LPF right, so you
> had to resort to tomtomes and tambourines.
>
> I suggest you to take a different look at the problem without being
> carried away by cool buzzwords such as "Savitsky-Golay", "Wavelet",
> "Matlab".
>
> Vladimir Vassilevsky
> DSP and Mixed Signal Design Consultanthttp://www.abvolt.com
>
> Poj wrote:
> > Thanks for your comment.
>
> > For your information, I have tried LPF and smoothing filter such as
> > Stavisky-Golay on Matlab. Both were unable to effectively produce a
> > satisfied output because the frequencies of intrinsic signal and noise are
> > very low (<20Hz) and close to each other. The noise I am talking about is
> > DC baseline swing and at this frequency region it is difficult to use HPF
> > to block it. Also, LPF cutoff is not sharp enough to separate the signal
> > and noise in this region. I am very happy with the filtering result from
> > my wavelet denoising. I just want to know techniques to reduce buffering
> > time delay available.
>
> >>"Poj" <[email protected]> wrote in message
> >>news:[email protected]...
>
> >>>Does anyone have experience developing real-time wavelet denoising
>
> >>software
>
> >>>without using any DSP chip? I have a problem with delay time caused by
> >>>buffering data and want to reduce the delay time as much as possible.
>
> > The
>
> >>>data I am dealing with is 1-D with sampling rate lower than 1kHz. I
>
> > have
>
> >>>developed the application in C++. The program reads data into a buffer
> >>>sample by sample until the buffer has enough data to perform DWT,
> >>>eliminate noise, and IDWT. So, the time to wait for data becomes delay
> >>>time. In other words, output of my wavelet is behind real time about
>
> > 40
>
> >>>seconds or so due to low sampling rate and data buffering. There is no
> >>>problem with denoising quality. or processing speed. I just want a
>
> > more
>
> >>>realistic real-time. Any suggestions or ideas are highly appreciated.
>
> >>Throw away the popular wavelet nonsense and implement a reasonable
>
> > denoising
>
> >>algorithm, like an adaptive filter or some kind of nonlinear smoothing.
> >>What would be the best algorithm depends on what is the signal and what
>
> > is
>
> >>the noise. Depending on how much exactly is the "high appreciation", I
>
> > might
>
> >>be able to help you.
>
> >>Vladimir Vassilevsky
> >>DSP and Mixed Signal Consultant
> >>www.abvolt.com

Baseline wander removal is difficult in a realtime system for all the
reasons you cite. I'm guessing you are driving some kind of display
with the result, and can't tolerate much lag between stimulus and
response. My hats off to you if you have found a satisfactory
approach. There is a sliding DWT that you might look at.


John

[Vladimir Vassilevsky]

>>If you are using a linear filter to resolve the frequencies, then the
>>incurred delay can't be less then about 1/transition band.

> Baseline wander removal is difficult in a realtime system for all the
> reasons you cite.

The only way to trick the nature is using parametric models of the
signal and the drift. The parameters should be estimated as ML. However
the results are going to be wild if the models do not match.

VLV

[John]

On Dec 27, 12:59 pm, Vladimir Vassilevsky <[email protected]>
wrote:
> >>If you are using a linear filter to resolve the frequencies, then the
> >>incurred delay can't be less then about 1/transition band.
> > Baseline wander removal is difficult in a realtime system for all the
> > reasons you cite.
>
> The only way to trick the nature is using parametric models of the
> signal and the drift. The parameters should be estimated as ML. However
> the results are going to be wild if the models do not match.
>
> VLV

The drift is difficult to model. In the case of a Galvanic Skin
Response (GSR), sometimes the baseline can be holding flat for many
seconds, then all of a sudden it plunges downward.

John

[Poj]

Hi John,

Thanks for message.

Yes, you are right. Lag between input and output is too much. Do you hav
any articles, links or other information related to sliding DWT?

Actually I have sliding wavelet kernels in my program for both DWT an
IDWT. However, since my wavelet can give the best result at Level 7,
have to freeze the real-time data in the buffer to decompose the data t
Level 7, then extract the signal I want and reconstruct it back to Leve
1. I am still looking for a better way to implement this in real-tim
environment.

I combined Wavelet Denoising with bidirectional HPF (Wavelet Denoisin
first, then forward HPF and finally backward HPF) to extract signal an
remove the baseline wander, gaussian white noise, and compensate phas
shift. This combination gives the best result over other stuff I have bee
testing so far. The bidirectional HPF uses pure sliding kernels and doe
not cause significant lag problem at all.

-poj

>
>Baseline wander removal is difficult in a realtime system for all the
>reasons you cite. I'm guessing you are driving some kind of display
>with the result, and can't tolerate much lag between stimulus and
>response. My hats off to you if you have found a satisfactory
>approach. There is a sliding DWT that you might look at.
>
>
>John
>

[Jerry Avins]

Vladimir Vassilevsky wrote:
>
>
>>> If you are using a linear filter to resolve the frequencies, then the
>>> incurred delay can't be less then about 1/transition band.
>
>> Baseline wander removal is difficult in a realtime system for all the
>> reasons you cite.
>
> The only way to trick the nature is using parametric models of the
> signal and the drift. The parameters should be estimated as ML. However
> the results are going to be wild if the models do not match.

How about fixing the baseline? A muffler on the noise source is usually
more effective than noise-canceling earmuffs.

Jerry
--
Engineering is the art of making what you want from things you can get.
¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯ ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯

[Jerry Avins]

John wrote:
> On Dec 27, 12:59 pm, Vladimir Vassilevsky <[email protected]>
> wrote:
>>>> If you are using a linear filter to resolve the frequencies, then the
>>>> incurred delay can't be less then about 1/transition band.
>>> Baseline wander removal is difficult in a realtime system for all the
>>> reasons you cite.
>> The only way to trick the nature is using parametric models of the
>> signal and the drift. The parameters should be estimated as ML. However
>> the results are going to be wild if the models do not match.
>>
>> VLV
>
> The drift is difficult to model. In the case of a Galvanic Skin
> Response (GSR), sometimes the baseline can be holding flat for many
> seconds, then all of a sudden it plunges downward.

That isn't baseline, it's an as-yet unaccounted-for phenomenon that
you're measuring. There's information in that thar noise. Can you find a
way to use it?

Jerry
--
Engineering is the art of making what you want from things you can get.
¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯ ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯

[John]

On Dec 27, 5:56 pm, Jerry Avins <[email protected]> wrote:
> John wrote:
> > On Dec 27, 12:59 pm, Vladimir Vassilevsky <[email protected]>
> > wrote:
> >>>> If you are using a linear filter to resolve the frequencies, then the
> >>>> incurred delay can't be less then about 1/transition band.
> >>> Baseline wander removal is difficult in a realtime system for all the
> >>> reasons you cite.
> >> The only way to trick the nature is using parametric models of the
> >> signal and the drift. The parameters should be estimated as ML. However
> >> the results are going to be wild if the models do not match.
>
> >> VLV
>
> > The drift is difficult to model. In the case of a Galvanic Skin
> > Response (GSR), sometimes the baseline can be holding flat for many
> > seconds, then all of a sudden it plunges downward.
>
> That isn't baseline, it's an as-yet unaccounted-for phenomenon that
> you're measuring. There's information in that thar noise. Can you find a
> way to use it?
>
> Jerry
> --
> Engineering is the art of making what you want from things you can get.
> ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯ ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯

Baseline is the standard term for it. Baseline wander is caused by
perspiration, body movement, etc. The actual response that is being
measured is summed with the wandering baseline. Google "baseline
wander" for more information.

John

[Randy Yates]

John <[email protected]> writes:

> On Dec 27, 5:56 pm, Jerry Avins <[email protected]> wrote:
>> John wrote:
>> > On Dec 27, 12:59 pm, Vladimir Vassilevsky <[email protected]>
>> > wrote:
>> >>>> If you are using a linear filter to resolve the frequencies, then the
>> >>>> incurred delay can't be less then about 1/transition band.
>> >>> Baseline wander removal is difficult in a realtime system for all the
>> >>> reasons you cite.
>> >> The only way to trick the nature is using parametric models of the
>> >> signal and the drift. The parameters should be estimated as ML. However
>> >> the results are going to be wild if the models do not match.
>>
>> >> VLV
>>
>> > The drift is difficult to model. In the case of a Galvanic Skin
>> > Response (GSR), sometimes the baseline can be holding flat for many
>> > seconds, then all of a sudden it plunges downward.
>>
>> That isn't baseline, it's an as-yet unaccounted-for phenomenon that
>> you're measuring. There's information in that thar noise. Can you find a
>> way to use it?
>>
>> Jerry
>> --
>> Engineering is the art of making what you want from things you can get.
>> ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯ ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯
>
> Baseline is the standard term for it. Baseline wander is caused by
> perspiration, body movement, etc. The actual response that is being
> measured is summed with the wandering baseline. Google "baseline
> wander" for more information.

Isn't this just a fancy term for DC offset, or "slowly time-varying DC offset"?
If so, then couldn't it be removed with a DC notch filter? The IIR form doesn't
have too much of a delay.
--
% Randy Yates % "Ticket to the moon, flight leaves here today
%% Fuquay-Varina, NC % from Satellite 2"
%%% 919-577-9882 % 'Ticket To The Moon'
%%%% <[email protected]> % *Time*, Electric Light Orchestra
http://www.digitalsignallabs.com

[John]

On Dec 27, 7:17 pm, Randy Yates <[email protected]> wrote:
> John <[email protected]> writes:
> > On Dec 27, 5:56 pm, Jerry Avins <[email protected]> wrote:
> >> John wrote:
> >> > On Dec 27, 12:59 pm, Vladimir Vassilevsky <[email protected]>
> >> > wrote:
> >> >>>> If you are using a linear filter to resolve the frequencies, then the
> >> >>>> incurred delay can't be less then about 1/transition band.
> >> >>> Baseline wander removal is difficult in a realtime system for all the
> >> >>> reasons you cite.
> >> >> The only way to trick the nature is using parametric models of the
> >> >> signal and the drift. The parameters should be estimated as ML. However
> >> >> the results are going to be wild if the models do not match.
>
> >> >> VLV
>
> >> > The drift is difficult to model. In the case of a Galvanic Skin
> >> > Response (GSR), sometimes the baseline can be holding flat for many
> >> > seconds, then all of a sudden it plunges downward.
>
> >> That isn't baseline, it's an as-yet unaccounted-for phenomenon that
> >> you're measuring. There's information in that thar noise. Can you find a
> >> way to use it?
>
> >> Jerry
> >> --
> >> Engineering is the art of making what you want from things you can get.
> >> ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯ ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯
>
> > Baseline is the standard term for it. Baseline wander is caused by
> > perspiration, body movement, etc. The actual response that is being
> > measured is summed with the wandering baseline. Google "baseline
> > wander" for more information.
>
> Isn't this just a fancy term for DC offset, or "slowly time-varying DC offset"?
> If so, then couldn't it be removed with a DC notch filter? The IIR form doesn't
> have too much of a delay.
> --
> % Randy Yates % "Ticket to the moon, flight leaves here today
> %% Fuquay-Varina, NC % from Satellite 2"
> %%% 919-577-9882 % 'Ticket To The Moon'
> %%%% <[email protected]> % *Time*, Electric Light Orchestrahttp://www.digitalsignallabs.com

The wandering baseline can sometimes move quickly, as in a rapid
plunge, and a simple filter will probably attenuate the features that
the analyst is trying to measure. That's why one turns to fairly
exotic techniques for this problem, but holding down the delay in a
real time instrument is difficult, as the OP obviously knows.

John

[John]

On Dec 27, 3:16 pm, "Poj" <[email protected]> wrote:
> Hi John,
>
> Thanks for message.
>
> Yes, you are right. Lag between input and output is too much. Do you have
> any articles, links or other information related to sliding DWT?
>


I only have what can be found with a search on IEEE Xplore or Google
Scholar.

John

[Randy Yates]

John <[email protected]> writes:

> On Dec 27, 7:17 pm, Randy Yates <[email protected]> wrote:
>> John <[email protected]> writes:
>> > On Dec 27, 5:56 pm, Jerry Avins <[email protected]> wrote:
>> >> John wrote:
>> >> > On Dec 27, 12:59 pm, Vladimir Vassilevsky <[email protected]>
>> >> > wrote:
>> >> >>>> If you are using a linear filter to resolve the frequencies, then the
>> >> >>>> incurred delay can't be less then about 1/transition band.
>> >> >>> Baseline wander removal is difficult in a realtime system for all the
>> >> >>> reasons you cite.
>> >> >> The only way to trick the nature is using parametric models of the
>> >> >> signal and the drift. The parameters should be estimated as ML. However
>> >> >> the results are going to be wild if the models do not match.
>>
>> >> >> VLV
>>
>> >> > The drift is difficult to model. In the case of a Galvanic Skin
>> >> > Response (GSR), sometimes the baseline can be holding flat for many
>> >> > seconds, then all of a sudden it plunges downward.
>>
>> >> That isn't baseline, it's an as-yet unaccounted-for phenomenon that
>> >> you're measuring. There's information in that thar noise. Can you find a
>> >> way to use it?
>>
>> >> Jerry
>> >> --
>> >> Engineering is the art of making what you want from things you can get.
>> >> ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯ ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯
>>
>> > Baseline is the standard term for it. Baseline wander is caused by
>> > perspiration, body movement, etc. The actual response that is being
>> > measured is summed with the wandering baseline. Google "baseline
>> > wander" for more information.
>>
>> Isn't this just a fancy term for DC offset, or "slowly time-varying DC offset"?
>> If so, then couldn't it be removed with a DC notch filter? The IIR form doesn't
>> have too much of a delay.
>> --
>> % Randy Yates % "Ticket to the moon, flight leaves here today
>> %% Fuquay-Varina, NC % from Satellite 2"
>> %%% 919-577-9882 % 'Ticket To The Moon'
>> %%%% <[email protected]> % *Time*, Electric Light Orchestrahttp://www.digitalsignallabs.com
>
> The wandering baseline can sometimes move quickly, as in a rapid
> plunge, and a simple filter will probably attenuate the features that
> the analyst is trying to measure. That's why one turns to fairly
> exotic techniques for this problem, but holding down the delay in a
> real time instrument is difficult, as the OP obviously knows.

Hi John,

If the bandwidth of the interference intersects with the bandwidth
of the signal, then no linear filter technique (including wavelets)
will suffice. It seems in that case that the best possible solution
is to model the interference and design a canceller, as Vladimir
pointed out.
--
% Randy Yates % "Ticket to the moon, flight leaves here today
%% Fuquay-Varina, NC % from Satellite 2"
%%% 919-577-9882 % 'Ticket To The Moon'
%%%% <[email protected]> % *Time*, Electric Light Orchestra
http://www.digitalsignallabs.com

[Poj]

I think packing all DWT-noise elimination-IDWT into a single sliding kerne
is the best idea. Does anyone know how to do it?

Happy New Year!




>On Dec 27, 3:16 pm, "Poj" <[email protected]> wrote:
>> Hi John,
>>
>> Thanks for message.
>>
>> Yes, you are right. Lag between input and output is too much. Do yo
have
>> any articles, links or other information related to sliding DWT?
>>
>
>
>I only have what can be found with a search on IEEE Xplore or Google
>Scholar.
>
>John
>

[Poj]

I was able to extract the sliding wavelet denoising kernel by injecting a
impulse to the entire DWT-noise elimination-IDWT process. The impuls
response in time-domain is in fact the kernel I want. And, the kernel ca
represent the entire denoising process. So, the problem is resolved. Th
case now is closed.

-poj



>I think packing all DWT-noise elimination-IDWT into a single slidin
kernel
>is the best idea. Does anyone know how to do it?
>
>Happy New Year!
>
>
>
>
>>On Dec 27, 3:16 pm, "Poj" <[email protected]> wrote:
>>> Hi John,
>>>
>>> Thanks for message.
>>>
>>> Yes, you are right. Lag between input and output is too much. Do you
>have
>>> any articles, links or other information related to sliding DWT?
>>>
>>
>>
>>I only have what can be found with a search on IEEE Xplore or Google
>>Scholar.
>>
>>John
>>
>