Entropies in investigation of dynamical systems and their application to digital image analysis

We discuss entropy characteristics used in various research techniques for investigation of complex dynamical systems including symbolic ones. The dynamics of a system may be studied by analyzing the phase portrait of a system obtained as a digital image. Symbolic dynamics methods allow combining entropy of a given dynamical system with the entropy characteristics of its phase portrait. We apply methods of image analysis based on symbolic dynamics, Rényi entropy, fractal and multifractal characteristics to analyze high resolution images having a complex structure. We also describe the results of applications of described methods to images of biomedical preparations.


Introduction
Scientific investigations of various processes are often based on observations.P. Bak in [1] noted, that observations are statistical in nature, and we use some distribution functions for measurable values to formalize them.It leads to forming informational (statistical) approach to study the behavior of a complex system, which is described by models processes we observe using the characteristics based on distribution functions.
In 1850, R. Clausius introduced the concept of entropy of an isolated system in thermodynamics as a measure of the change of system state under a change of temperature.The term "entropy" denoted the energy that does not perform work, and in any isolated system it can only increase.
L. Boltzmann [2] introduced the statistical approach in thermodynamics -he proposed to describe a system state by using its microstates.The Boltzmann entropy -statistical entropy for equiprobable distribution of a system over  states:  = ln . ( For non-equiprobable distributions, the ensembles of microcanonical subsystems are used so that all its  states are equiprobable with a probability  for every  -th subsystem.The Boltzmann entropy (1) of -th subsystem is  = − ln  .The averaging of ensembles results in the expression for Gibbs-Shannon entropy: The development of informational approach to the description of complex systems led to the creation of other forms of informational entropies, in particular Rényi entropies -a set of functions depending on a real parameter : These functions are important characteristics of a complex system, in particular in image analysis.Rényi spectra are calculated by using Renyi entropies.The Gibbs-Shannon entropy Eq. ( 2) is the Rényi entropy Eq. ( 3) for  = 1.
Then the term of "entropy" began to take on more general meaning not related directly to thermodynamics.In 1948, C. Shannon invented entropy as a term in the information theory, A. Kolmogorov in 1958 created the concept of "dynamical entropy" in the theory of dynamical systems.Now, one can find the term entropy in mathematics, physics, and computer science.In dynamical systems, the entropy characterizes the system complexity.Topological entropy describes the complexity of orbit structure.Moreover, topologically conjugated systems have the same entropy, thus they have the characteristics of a class of systems.To describe the statistical behavior of orbits, the metric entropy of measure-preserving transforms, which is a statistical analog of topological entropy, is introduced.A comprehensive review on entropy in dynamical systems and the relation between them are given in [3,4].
Symbolic dynamical systems form one of fundamentally important classes of dynamical systems.They are obtained in the process of coding the smooth dynamical systems.The coding means that we construct a finite partition of a phase space and assign a symbol from a finite alphabet to any element of the partition.Then we code a trajectory with a symbol sequence by writing one-by-one codes of partition elements which the trajectory visits.We obtain a space of symbol sequences corresponding to an approximation of the set of system trajectories, and define the shift map on the space so that one step along the trajectory is the shift of the corresponding sequence on one symbol.Symbolic dynamical systems having a presentation by the oriented graph are topological Markov chains, which are very important in different applications.
Practical application of symbolic dynamical systems was made possible due to elaboration of applied symbolic dynamics methods.Trajectory coding for description of global behavior of geodesic on the surfaces with negative curvature was applied in 1898 by J. Hadamar [5].The papers of H. Morse and G. Hedlung [6,7] and R. Bowen [8] made an important contribution to the progress of the method.V. Alekseev [9] applied it for celestial mechanics problems.In 1983 G. Osipenko elaborated the method of symbolic image, in which the system dynamics is represented by an oriented graph constructed by some rules.Actually, symbolic image graph is a topological Markov chain, where symbol sequences correspond to paths on the graph.By applying the technique of adaptive partition subdivision, one can construct a sequence of symbolic images that is an approximation of the dynamics of the initial system.This method was successfully applied to the approximation of invariant sets, Morse spectrum and invariant measures [10].
To study the dynamics of a system, one may analyze its phase portraits in various points in time.The operations with symbolic sequences naturally result in the calculation of a statistical characteristic of the system -relative frequency of a symbol occurrence in the sequence, which corresponds to the frequency at which a trajectory of the initial system visits a partition element.When considering phase portraits as digital images, we may interpret the set of frequencies as a measure distribution.Basing on this distribution, we may calculate multifractal spectra, Rényi spectra and divergences.These characteristics use Rényi entropies.
One can simulate the dynamics of diffusion processes illustrated by their phase portraits by constructing an oriented graph and the Markov chain on it.The flow is defined by pixel intensities.Then the stationary flow (state) of the chain is calculated, which maximizes so called weighted entropy.This value interpreted as a "distance" between an initial state, and the stationary one may be used as a classifying sign in image analysis.These methods appear to have a considerable promise for high resolution images with complex structure.
In this work, we consider the application of analysis methods of dynamical systems including image analysis ones, which are based on combining the symbolic dynamics and information approach.Various types of entropies serve as connecting links between different problems and methods.The described methods were used to analyze and classify digital images of biomedical preparations having a complex structure.

Calculation and estimation of topological entropy
Topological entropy may be defined by different ways, for example through minimal coverings or separated sets [11].For simple system having complex structure of orbits, one may calculate the growth rate of periodic orbits.Examples of such systems are linear extending maps, and hyperbolic automorphism of torus.For them, the topological entropy coincides with the growth rate, which may be easily calculated.But for the most part of systems, we need to use methods of estimations.In 1993, S. Newhouse and T. Pignataro [12] proposed a method to estimate the topological entropy of a smooth dynamical system by calculating the logarithmic growth rates of suitably chosen curves in the system.The method may be successfully applied to study complex systems with strange attractors.In [13], the authors described the algorithm for obtaining rigorous lower bound of the topological entropy of planar diffeomorphisms.They used a method of approximation of invariant manifolds of hyperbolic periodic points, and applied it to estimate entropy for Henon and Ikeda maps.An effective algorithm was proposed in [14], where the authors calculated lower bound of the topological entropy and used it to identify regions of high mixing.
Topological entropy with relative ease may be obtained for a topological Markov chain: we should calculate the module of maximal eigenvalue of the adjacency matrix for the corresponding graph.In this connection, we may estimate topological entropy of a system by using the symbolic image method: in accordance with a given partition to construct the oriented graph (which is a topological Markov chain) and calculate eigenvalues of the adjacency matrix.In order to simplify the problem, one may perform this procedure for each component of strong connectivity of the graph.
The technique for estimation of topological entropy for the spaces of symbol sequences ("edge space" and "vertex space") on oriented graphs was proposed in [15].Basing on this approach, G. Osipenko in [10] designed and implemented algorithms for estimation of the entropy of symbolic image and obtained the estimation of topological entropy of Henon and logistic maps.

Metric entropy estimation
The concept of metric entropy of a map is introduced for maps preserving a measure.It means that we should deal with invariant measures of a map.The Krilov-Bogolyubov theorem [4] guarantees that any continuous map on a compact space has an invariant measure.One may construct an approximation to the invariant measure using the symbolic image method.In this case, we should calculate stationary flow on the graph of symbolic image.In doing so, we obtain a stationary distribution for a Markov chain.Then metric entropy of the distribution is a lower bound for the topological entropy of symbolic image.The algorithms of the construction of stationary flow were described and implemented in [16,17], where the estimations for topological entropy of delay map, double logistic map, Henon and Ikeda map were obtained.
In this work, we also use the idea of construction of stationary flow on a graph constructed for a digital image by a specific way.It is known that stationary flow gives maximum to the function dealing with initial and stationary distributions (so called weighted entropy).For digital images weighted entropy may be considered as a classifying sign.

Entropies as goal functions in variational problems
The Shannon and Rényi entropies are often used as goal functions when finding distributions that maximize (minimize) the function for some given restrictions.We consider one problem which is related to multifractal spectrum calculation.
Let a distribution { } with ∑  = 1 be given.Find the distribution  maximizing the Shannon entropy − ∑  ln  on the conditions ∑  = 1 , and the constraints on ln  averaged over  , i.e. ∑  ln  = , where  is a real constant.
Consider the Lagrange function with parameters  and : Equaling partial derivatives of  (given by Eq. ( 4)) by  to zero, we obtain the system: It follows that: Combining Eq. ( 5), we obtain 1 =  ∑  , that taking into account Eq. ( 6) leads to excluding : For given conditions on average values (), one can calculate  and solve the optimization problem.But it is significant that Eq. ( 7) determines a set of solutions depending on the parameter .In fact, it is a transform of the initial measure { } (so called direct multifractal transform).It is easy to see that for  ≠ 0 transforms Eq. ( 7) forms a group, namely if we denote  () =  ∑  , then  ( ) =  ,  () =  and  ( / ) =  , where  =  .For  = 1 we have the fixed point.The set of probabilistic measures falls into non-intersecting transitivity classes.

Weighted entropy
Digital images illustrating diffusion processes may be analyzed by the construction of an oriented graph, which helps to describe the process dynamics.For this purpose, the image is considered as a pixel lattice, number of vertex in the graph is equal to the number of pixels.The measure  of a vertex (pixel)  is assumed to be equal to the pixel intensity.Then we construct edges from every vertex to nearest neighbors (4 or 8) and for every edge (, ), its measure  is equal to  divided into a number of neighbors.After standartizing the distribution  , we obtain a Markov chain on the graph having the property: vertex measure is equal to the sum of measures of outcoming edges.For such a chain, one can construct a stationary distribution (stationary flow)  , namely: the sums of measures of incoming and outcoming edges are equal in every vertex.As it was shown in [18], stationary distribution maximizes so-called weighted entropy: According to the maximum entropy principle, it means that the system goes from an initial state into the state with the maximal entropy -stationary one.
The stationary state of the Markov chain describes a stable (stationary) state of the system generating the considered image.The existence of a stationary flow constructed on the graph related to this image may be interpreted as the existence of an invariant set of the system.
Problems from different subject areas may lead to the construction of а stationary flow on a graph.This method is applied in linear programming (transport tasks), dynamical systems (construction of an invariant measure on the graph of symbolic image).It should be noted that the solution exists if there are cycles on the graph.
In [19], a model based on the calculation of a stationary flow on the graph was considered to obtain a classification sign -weighted entropy -for a digital image.The implementation of the algorithmand results of the classification of pharmacological  solution images is given in [20].In [21], this method and its optimization based on the choice of a cell instead of a pixel were successfully applied to some classes of biological preparations images, and in [22] methods of optimization using parallel calculation and another data representation were implemented.These methods allowed reducing the run time 3-4 times.

Weighted entropy calculation
Weighted entropy is calculated for 4 classes of biomedical preparation images, and each class contains 12 images.The images were obtained by microscope AxioCam MRc5 of the company Carl Zeiss Microimaging GmbH.All the images were made with 200-fold zoom, represented in RGB and with the size of 2584×1936 pixels.
A preliminary classification was performed by an expert.All the images are represented both in grayscale and HSV palette (component wise).

Fractal dimensions and entropy
There is a connection between Shannon entropy and fractal dimensions.The first example is the Egglestone theorem concerning the Hausdorff dimension of a set of numbers in the unit interval with given distribution of relative frequencies of digits.It was described in [23].The author gave the more common definition of Hausdorff's dimension which was used in problems relating to -adic decompositions.He considered a probabilistic measure on Borel sets on unit interval.The dimension of a set of numbers in the interval with a given distribution of relative frequencies of digits was expressed through the entropy of the probabilistic vector and coincided with the Hausdorff dimension.
The presentation of real number  was considered in unit interval in the numbering system with base  > 1, namely: where coefficients  () take values from 0 to  − 1.
The number of occurrences of symbol  is denoted in the sequence  (), … ,  () by  (, ) .Note that the sequence { (, )} has the subadditivity property:  (, ) ≤  (, ) +  (, ).Hence by the Pólya lemma there is: Thus, to any sequence Eq. ( 9), one can match the probabilistic vector formed by relative frequencies of occurrences of symbol  in the sequence.Denote the set of numbers  for which  () satisfy Eq. ( 10) by а ( , …  ).Then the Hausdorff dimension of the set ( , … ,  ) may be calculated as: It should be noted that in [24], H. Furstenberg obtained a more common result using symbolic dynamics.Namely, he considered a set of one-sided r-ary sequences Σ and a closed subset  which is  invariant, where  is shift map.Let : Σ → [0,1] , where ({ }) = ∑ ,  =  ().In other words, by this way, we can match a number from unit interval to a symbol sequence over -symbol alphabet.Let also  = ().Then the Hausdorff dimension of the set  is given by the formula dim  = ( | ) , where ℎ is the topological entropy of .
Hence it can be seen that taking into account the map , a record of a number in a numbering system with the base  may be interpreted as the image of a symbolic sequence which presents the coding of a dynamical system trajectory related to a chosen partition.Note that the set of numbers 1 1 3  in unit interval with a given distribution  () may be thought as a set of encodings for trajectories of a dynamical system, and the Hausdorff's dimension of this set is calculated by Eq. (11).

Multifractal spectrum and entropy
Another example of the connection between fractal dimensions and entropy is the information dimension, which is a member of Rényi dimensions set.This dimension may be interpreted as the dimension of the measure-theoretic support of a particular measure (in other words, the complement of the support has zero Lebesgue measure).
Digital images with complex texture often have fractal or multifractal structures.Fractal sets may be described by a numerical characteristic called as fractal dimension, which reflects the main property of such sets -self-similarity.The last one may be strict (as for Cantor set) and statistical (as for majority of nature objects).Multifractal sets may be interpreted as unions of several fractal subsets.Each of subsets has its own fractal dimension, and they are arranged by a very complex way.Fractal dimensions of these subsets form multifractal spectrum.In the fractal analysis, we use a main assumption based on experimental data: if we divide an image on cells with a given size and calculate a measure of cells then there is power dependence between the measure and size.This exponent is an approximate value of the fractal dimension of the set.
Let us to give the number of cell of a partitions () with size  for a set .In [25], the authors considered a calculation method for the dimension of a measure { } support  (in other words, the dimension of the set of trajectories on the phase portrait which corresponds to the given distribution) as: Taking into account the dependence measure { } on the cell size, and ~1  ⁄ , we obtain Eq. ( 12) rewritten as follows: The last one is the information dimension formula.The authors also proposed to calculate multifractal spectrum by obtaining a sequence of measures from a given initial normalized measure distribution by applying direct multifractal transform, and then calculate information dimensions of supports of measures from the sequence by Eq. (13).
It is interesting to note that substituting  (, ) =  () ∑  () in Eq. ( 14) we obtain:  In other words, on the sequence of measures obtained from an initial measure by direct multifractal transform, the formula for the Legendre transform is made.
We applied the described method in [27]

Rényi divergences
A calculation of Rényi divergences of order  (or -divergences) allows us finding differences in structures of two images.For given probabilistic distributions  = { } and  = { } and the -divergence is defined for some  as follows: It is not difficult to verify that these values are non-negative for any  , and the Rényi divergence as a function of  is non-decreasing.For  = 1 the divergence is given by the formula.
It is known as the Kullback-Leibler divergence, and up to sign the Eq. ( 17) is the Eq. ( 8), which describes the weighted entropy.
For given initial measures, we applied direct fractal transform Eq. ( 7), obtained two sequences of transformed measures and calculated Rényi divergences between corresponding members of these sequences.We applied this approach in [28,29] to analyze and classify some classes of biomedical preparation images.Consider the example of application of this method to the analysis of healthy tissue and tissue with pyelonephritis images.

Conclusions
Nowadays the concept of entropy may be considered as a general characteristic for the description of many processes.It is natural that our descriptions are based on observations, which are statistical by its nature.The mathematical formalization for statistical events is expressed in terms of distribution functions, and the entropy is also written by using these functions.It is significant that one may obtain entropy characteristics both for various dynamical systems, and for digital images which are phase portraits of such systems.Symbolic dynamics allows us giving a natural interpretation of statistical description of the dynamic system behavior: it shows the frequency with which a typical trajectory visits cells of a given partition.Thereby such a characteristic describes a distribution of a measure on the phase space.For a given measure, one 1 1 7 can calculate the partition entropy and metric entropy.Namely metric entropy is the base to calculate the information dimension of the measure support.The set of these dimensions is a kind of multifractal spectrum.Rényi entropies also use the metric entropy.Multifractal characteristics reveal essential peculiarities of image structure and may be used as classification signs.These methods are successfully applied in the analysis of a wide range of biomedical preparations images which have high resolution and very complex structure.So, the concept entropy, which is understood in dynamical systems as the system complexity, preserves its meaning in the application problems: based on entropy, the numerical characteristics of phase portraits show their complexity and in this way the behavior of the initial system.

Fig. 1 Fig. 1 .
shows images from 4 classes of liver tissue: plethora, dystrophy, cirrhosis, metastasis.Images of liver tissue with plethora, dystrophy, cirrhosis and metastasis

2 .
to classify biomedical preparation images.Example: Now the example of calculation of () and () is considered for images of a healthy bone tissue and a tissue with osteoporosis.Images have the size [350, 279].Calculations were performed for a grayscale palette, the parameter  changed within the interval [-7, 3] with the step of 0.5.The cell measure is calculated as the relation of pixel intensities in the cell to the sum of pixel intensities of the image.To calculate () and () by Eqs.(14) and (15) the least square method was used with cell sizes {5, 6, 7, 8, 9, 10, 11}.a) Healthy bone tissue b) Bone tissue with osteoporosis Fig. Images of healthy bone tissue and bone tissue with osteoporosis

Fig. 3 .
Fig. 3. Graphics of parametrized spectra for healthy bone tissueIt should be noted that the choice of interval of values of parameter  depends on the image structure and palette.Nevertheless, the obtained results show that parametrized spectra may be used as classification signs in the image analysis.

Fig. 4 .
Fig. 4. Graphics of parametrized spectra for bone tissue with osteoporosis

Fig. 7 .
Fig. 7. Graphics of Rényi divergences for various parameter values when comparing images of healthy and invaded tissues Fig. 5(a, c)

Table 1 .
152.ENTROPIES IN INVESTIGATION OF DYNAMICAL SYSTEMS.THE APPLICATION TO DIGITAL IMAGE ANALYSIS.In Table1values of weighted entropy for images in Fig.1are given.Experiments show that the values of weighted entropy are different in H component.Weighted entropy values in several palette components NATALIA AMPILOVA, IGOR SOLOVIEV 112 JOURNAL OF MEASUREMENTS IN ENGINEERING.JUNE 2018, VOLUME 6, ISSUE 2