com.tagtraum.jipes.audio

Class Novelty<I extends AudioBuffer>

java.lang.Object

com.tagtraum.jipes.AbstractSignalProcessor<I,AudioBuffer>

com.tagtraum.jipes.audio.Novelty<I>

All Implemented Interfaces:

SignalProcessor<I,AudioBuffer>, SignalPullProcessor<I,AudioBuffer>, SignalSource<AudioBuffer>

public class Novelty<I extends AudioBuffer>
extends AbstractSignalProcessor<I,AudioBuffer>

Processes AudioBuffer features to create a novelty curve using a kernel and a distance function.

By default this processor does not pad the self-similarity matrix with zeroes, but only calculates novelty values for regions of the matrix that are completely covered by the used kernel. To change this behavior, call setPadSimilarityMatrixWithZeros(boolean).

The similarity matrix is created using a DistanceFunction, which is then turned into a similarity function via a FloatMapFunction. Both functions can be set via their respective setters.

Author:

Hendrik Schreiber

See Also:

Foote, J., and M. Cooper. 2003. Media segmentation using self-similarity decomposition. In M. Yeung, R. Lienhart, and C.-S. Li (Eds.), Proceedings of the SPIE: Storage and Retrieval for Media Databases, Volume 5021, Santa Clara, CA, USA, 167-75. SPIE., J. Foote, "Automatic Audio Segmentation using a Measure of Audio Novelty." In Proceedings of IEEE International Conference on Multimedia and Expo, vol. I, pp. 452-455, 2000.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Novelty.GaussianCheckerboardKernel Gaussian checkboard kernel.
static interface	Novelty.Kernel Kernel.
static class	Novelty.NormGaussianCheckerboardKernel Normalized Gaussian checkerboard kernel. Aggregate values are divided by the maximal possible value a Novelty.GaussianCheckerboardKernel would deliver before they are returned, assuming the input values are between -1 and 1.

Field Summary

Fields

Modifier and Type	Field and Description
static FloatMapFunction	EXP_NEG_DISTANCE Converts a distance value into a similarity value by calculating exp(-distance).
static FloatMapFunction	ONE_MINUS_DISTANCE Converts a distance value into a similarity value by calculating 1 - distance.

Fields inherited from class com.tagtraum.jipes.AbstractSignalProcessor

lastOut, signalProcessorSupport

Constructor Summary

Constructors

Constructor and Description
Novelty() Default processor with a 64 frames normalized Gaussian kernel and no padding.
Novelty(Object id) Default processor with a 64 frames normalized Gaussian kernel and no padding.
Novelty(Object id, DistanceFunction<I> distanceFunction, boolean padSimilarityMatrixWithZeros) Default processor with a 64 frames normalized Gaussian checkerboard kernel.
Novelty(Object id, Novelty.Kernel kernel, DistanceFunction<I> distanceFunction, boolean padSimilarityMatrixWithZeros) Processor with DistanceFunctions.COSINE_DISTANCE and EXP_NEG_DISTANCE mapping to similarity.

Method Summary

Modifier and Type	Method and Description
static DistanceFunction<float[]>	createCosineDistanceFunction(int kernelSize, int minBin, int maxBin) Creates a special cosine distance function implementation specifically for how it is called from a Novelty with a given kernel size.
void	flush() Flushes this processor and its kids.
DistanceFunction<I>	getDistanceFunction() Returns the distance function used to compute the self similarity matrix.
FloatMapFunction	getDistanceToSimilarityFunction() Returns the function used to map a distance value (the result of getDistanceFunction()}) to a similarity value.
Novelty.Kernel	getKernel() Returns the Kernel, a specialized AggregateFunction.
boolean	isPadSimilarityMatrixWithZeros() Indicates whether the self-similarity matrix is zero padded.
void	process(I buffer) Asks this processor to work on the provided data and pipe the output to the connected processors' own SignalProcessor.process(Object) method. Implementing methods must not modify or hold on to the provided data.
protected AudioBuffer	processNext(I buffer) Processes the given input and returns some output.
AudioBuffer	read() Provide the next chunk of output data.
void	reset() Calls SignalSource.reset(), if a source is set.
void	setDistanceFunction(DistanceFunction<I> distanceFunction) Sets the distance function used to compute the self similarity matrix.
void	setDistanceToSimilarityFunction(FloatMapFunction distanceToSimilarityFunction) Sets the mapping function from distance values to similarity values.
void	setKernel(Novelty.Kernel kernel) Sets the kernel.
void	setPadSimilarityMatrixWithZeros(boolean padSimilarityMatrixWithZeros) Configures the processor to zero pad the internal self-similarity matrix, so that novelty values can be computed for the beginning and the end of the matrix' diagonal, where the kernel only partially overlaps with the matrix.
String	toString()

Methods inherited from class com.tagtraum.jipes.AbstractSignalProcessor

connectTo, connectTo, connectTo, disconnectFrom, getConnectedProcessors, getConnectedSource, getId, getOutput, setId

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

ONE_MINUS_DISTANCE

public static final FloatMapFunction ONE_MINUS_DISTANCE

Converts a distance value into a similarity value by calculating 1 - distance.

EXP_NEG_DISTANCE

public static final FloatMapFunction EXP_NEG_DISTANCE

Converts a distance value into a similarity value by calculating exp(-distance).

Constructor Detail

Novelty

public Novelty(Object id,
               Novelty.Kernel kernel,
               DistanceFunction<I> distanceFunction,
               boolean padSimilarityMatrixWithZeros)

Processor with DistanceFunctions.COSINE_DISTANCE and EXP_NEG_DISTANCE mapping to similarity.

Parameters:

id - id

kernel - kernel

distanceFunction - distance function

padSimilarityMatrixWithZeros - true or false

Novelty

public Novelty(Object id,
               DistanceFunction<I> distanceFunction,
               boolean padSimilarityMatrixWithZeros)

Default processor with a 64 frames normalized Gaussian checkerboard kernel.

Parameters:

id - id

distanceFunction - distance function

padSimilarityMatrixWithZeros - true or false

See Also:

Novelty.NormGaussianCheckerboardKernel

Novelty

public Novelty(Object id)

Default processor with a 64 frames normalized Gaussian kernel and no padding.

Parameters:

id - id

Novelty

public Novelty()

Default processor with a 64 frames normalized Gaussian kernel and no padding.

Method Detail

isPadSimilarityMatrixWithZeros

public boolean isPadSimilarityMatrixWithZeros()

Indicates whether the self-similarity matrix is zero padded. Zero padding will lead to results that have the same length as the input, but are somewhat skewed (too high) at the beginning and the end.

Returns:

true or false

setPadSimilarityMatrixWithZeros

public void setPadSimilarityMatrixWithZeros(boolean padSimilarityMatrixWithZeros)

Configures the processor to zero pad the internal self-similarity matrix, so that novelty values can be computed for the beginning and the end of the matrix' diagonal, where the kernel only partially overlaps with the matrix.

Parameters:

padSimilarityMatrixWithZeros - true or false

getKernel

public Novelty.Kernel getKernel()

Returns the Kernel, a specialized AggregateFunction.

Returns:

kernel

setKernel

public void setKernel(Novelty.Kernel kernel)

Sets the kernel.

Parameters:

kernel - kernel

getDistanceFunction

public DistanceFunction<I> getDistanceFunction()

Returns the distance function used to compute the self similarity matrix.

The default value is DistanceFunctions.COSINE_DISTANCE

Returns:

distance function

setDistanceFunction

public void setDistanceFunction(DistanceFunction<I> distanceFunction)

Sets the distance function used to compute the self similarity matrix.

Parameters:

distanceFunction - distance function

See Also:

createCosineDistanceFunction(int, int, int)

getDistanceToSimilarityFunction

public FloatMapFunction getDistanceToSimilarityFunction()

Returns the function used to map a distance value (the result of getDistanceFunction()}) to a similarity value.

The default value is EXP_NEG_DISTANCE

Returns:

map function to map distance to similarity

setDistanceToSimilarityFunction

public void setDistanceToSimilarityFunction(FloatMapFunction distanceToSimilarityFunction)

Sets the mapping function from distance values to similarity values.

Parameters:

distanceToSimilarityFunction - map function

processNext

protected AudioBuffer processNext(I buffer)
                           throws IOException

Description copied from class: AbstractSignalProcessor

Processes the given input and returns some output.

Specified by:

processNext in class AbstractSignalProcessor<I extends AudioBuffer,AudioBuffer>

Parameters:

buffer - input guaranteed not to be null.

Returns:

output produced by this processor

Throws:

IOException - if an IO error occurs

reset

public void reset()

Description copied from class: AbstractSignalProcessor

Calls SignalSource.reset(), if a source is set.

Specified by:

reset in interface SignalSource<AudioBuffer>

Overrides:

reset in class AbstractSignalProcessor<I extends AudioBuffer,AudioBuffer>

read

public AudioBuffer read()
                 throws IOException

Description copied from interface: SignalSource

Provide the next chunk of output data. This method is called by a consumer of SignalSource data, typically an SignalPullProcessor.

Specified by:

read in interface SignalSource<AudioBuffer>

Overrides:

read in class AbstractSignalProcessor<I extends AudioBuffer,AudioBuffer>

Returns:

output data - perhaps a float[] or even a more complex object like a AudioBuffer, null if no more data is available

Throws:

IOException - if something goes wrong

process

public void process(I buffer)
             throws IOException

Description copied from interface: SignalProcessor

Asks this processor to work on the provided data and pipe the output to the connected processors' own SignalProcessor.process(Object) method.
Implementing methods must not modify or hold on to the provided data. In other words: the provided object does not belong to you, its values are only guaranteed to be stable for the duration of the call.

If you need to hold on to data, create a copy of the provided data via a copy constructor or the Object.clone() method.

It is the responsibility of implementing classes to call the SignalProcessor.process(Object) method of any connected processors. To manage connected processors classes may want to use a SignalProcessorSupport instance.

Specified by:

process in interface SignalProcessor<I extends AudioBuffer,AudioBuffer>

Overrides:

process in class AbstractSignalProcessor<I extends AudioBuffer,AudioBuffer>

Parameters:

buffer - data to process

Throws:

IOException - if a processing error occurs

See Also:

SignalProcessor.getConnectedProcessors()

flush

public void flush()
           throws IOException

Description copied from class: AbstractSignalProcessor

Flushes this processor and its kids.

Specified by:

flush in interface SignalProcessor<I extends AudioBuffer,AudioBuffer>

Overrides:

flush in class AbstractSignalProcessor<I extends AudioBuffer,AudioBuffer>

Throws:

IOException - if a processing error occurs

toString

public String toString()

Overrides:

toString in class Object

createCosineDistanceFunction

public static DistanceFunction<float[]> createCosineDistanceFunction(int kernelSize,
                                                                     int minBin,
                                                                     int maxBin)

Creates a special cosine distance function implementation specifically for how it is called from a Novelty with a given kernel size.

Parameters:

kernelSize - kernel size

minBin - min bin

maxBin - max bin

Returns:

specially optimized cosine distance function