A. Data Matrix

Consider, for example, that measurements from $m$ different meters, taken over $t$ time instants, need to be transmitted through the communications network of a smart distribution system to serve as inputs for a given application. Let this set of measurements be put in the form of a matrix ${\mathbf{X}}$ , with each row of ${\mathbf{X}}$ containing the measurements taken from a given meter at each time instant. Matrix ${\mathbf{X}}$ is illustrated in Fig. 2.

Fig. 2.

Data matrix ${\mathbf{X}}$ .

Show All

This is a convenient form to represent the data, as it can be easily manipulated for data compression, to be shown in the next section.

B. Data Compression

As discussed in Section IV, data matrix ${\mathbf{X}}$ can be factorized into three matrices by applying the SVD. This is shown in (3) and schematically represented in Fig. 3. In order to simplify the notation, matrix ${\mathbf{V}}^{\mathbf{T}}$ shown in (1) is hereafter written as just ${\mathbf{V}}$ $$\begin{equation} {\mathbf{X}}^{({\mathbf{m}}\times {\mathbf{t}})} = {\mathbf{U}}^{({\mathbf{m}}\times {\mathbf{m}})} \boldsymbol {\Sigma }^{({\mathbf{m}}\times {\mathbf{t}})} {\mathbf{V}}^{({\mathbf{t}}\times {\mathbf{t}})} \end{equation}$$ View Source\begin{equation} {\mathbf{X}}^{({\mathbf{m}}\times {\mathbf{t}})} = {\mathbf{U}}^{({\mathbf{m}}\times {\mathbf{m}})} \boldsymbol {\Sigma }^{({\mathbf{m}}\times {\mathbf{t}})} {\mathbf{V}}^{({\mathbf{t}}\times {\mathbf{t}})} \end{equation} where diagonal matrix $\boldsymbol {\Sigma }$ contains the SVs of ${\mathbf{X}}$ , ordered from the highest to the lowest.

Fig. 3.

SVD of data matrix ${\mathbf{X}}$ .

Show All

Data compression can be achieved by taking advantage of the fact that many matrices occurring in practice do exhibit some kind of structure that leads to only a few SVs actually being non-negligible. In such cases, good approximation of matrix ${\mathbf{X}}$ can be obtained by keeping only the SVs found to be significant in matrix $\boldsymbol {\Sigma }$ . Assume that ${r}$ SVs are to be retained in $\boldsymbol {\Sigma }$ and let matrix XR denotes the approximated matrix ${\mathbf{X}}$ . Then, matrix ${\mathbf{XR}}$ can be computed by replacing $\boldsymbol {\Sigma }$ by $\boldsymbol {\Sigma }\boldsymbol{ {'} }$ in (1), yielding expression (4) $$\begin{equation} {\mathbf{XR}} = {\mathbf{U}} \boldsymbol {\Sigma }' {\mathbf{V}} \end{equation}$$ View Source\begin{equation} {\mathbf{XR}} = {\mathbf{U}} \boldsymbol {\Sigma }' {\mathbf{V}} \end{equation} where each matrix can be partitioned into submatrices as $$\begin{align*} {\mathbf{XR}}=&\left [{\begin{array}{cc} {{\mathbf{XR}}_{11}^{^{({\mathbf{r}}\times {\mathbf{r}})} } } &\quad {{\mathbf{XR}}_{12}^{^{({\mathbf{r}}\times ({\mathbf{t}}-{\mathbf{r}}))} } } \\ {{\mathbf{XR}}_{21}^{^{(({\mathbf{m}}-{\mathbf{r}})\times {\mathbf{r}})} } } &\quad {{\mathbf{XR}}_{22}^{^{(({\mathbf{m}}-{\mathbf{r}})\times ({\mathbf{t}}-{\mathbf{r}}))} } } \end{array}}\right ] \\ {\mathbf{U}}=&\left [{\begin{array}{cc} {{\mathbf{U}}_{11}^{^{({\mathbf{r}}\times {\mathbf{r}})} } } &\quad {{\mathbf{U}}_{12}^{^{({\mathbf{r}}\times ({\mathbf{m}}-{\mathbf{r}}))} } } \\ {{\mathbf{U}}_{21}^{^{(({\mathbf{m}}-{\mathbf{r}})\times {\mathbf{r}})} } } &\quad {{\mathbf{U}}_{22}^{^{(({\mathbf{m}}-{\mathbf{r}})\times ({\mathbf{m}}-{\mathbf{r}}))} } } \end{array}}\right ] \\ {\mathbf{V}}=&\left [{\begin{array}{cc} {{\mathbf{V}}_{11}^{^{({\mathbf{r}}\times {\mathbf{r}})} } } &\quad {{\mathbf{V}}_{12}^{^{({\mathbf{r}}\times ({\mathbf{t}}-{\mathbf{r}}))} } } \\ {{\mathbf{V}}_{21}^{^{(({\mathbf{t}}-{\mathbf{r}})\times {\mathbf{r}})} } } &\quad {{\mathbf{V}}_{22}^{^{(({\mathbf{t}}-{\mathbf{r}})\times ({\mathbf{t}}-{\mathbf{r}}))} } } \end{array}}\right ] \\ \boldsymbol {\Sigma }\boldsymbol{ {'} }=&\left [{\begin{array}{cc} { \boldsymbol {\Sigma }_{11}^{^{({\mathbf{r}}\times {\mathbf{r}})} } } &\quad {0} \\ {0} &\quad {0} \end{array}}\right ]. \end{align*}$$ View Source\begin{align*} {\mathbf{XR}}=&\left [{\begin{array}{cc} {{\mathbf{XR}}_{11}^{^{({\mathbf{r}}\times {\mathbf{r}})} } } &\quad {{\mathbf{XR}}_{12}^{^{({\mathbf{r}}\times ({\mathbf{t}}-{\mathbf{r}}))} } } \\ {{\mathbf{XR}}_{21}^{^{(({\mathbf{m}}-{\mathbf{r}})\times {\mathbf{r}})} } } &\quad {{\mathbf{XR}}_{22}^{^{(({\mathbf{m}}-{\mathbf{r}})\times ({\mathbf{t}}-{\mathbf{r}}))} } } \end{array}}\right ] \\ {\mathbf{U}}=&\left [{\begin{array}{cc} {{\mathbf{U}}_{11}^{^{({\mathbf{r}}\times {\mathbf{r}})} } } &\quad {{\mathbf{U}}_{12}^{^{({\mathbf{r}}\times ({\mathbf{m}}-{\mathbf{r}}))} } } \\ {{\mathbf{U}}_{21}^{^{(({\mathbf{m}}-{\mathbf{r}})\times {\mathbf{r}})} } } &\quad {{\mathbf{U}}_{22}^{^{(({\mathbf{m}}-{\mathbf{r}})\times ({\mathbf{m}}-{\mathbf{r}}))} } } \end{array}}\right ] \\ {\mathbf{V}}=&\left [{\begin{array}{cc} {{\mathbf{V}}_{11}^{^{({\mathbf{r}}\times {\mathbf{r}})} } } &\quad {{\mathbf{V}}_{12}^{^{({\mathbf{r}}\times ({\mathbf{t}}-{\mathbf{r}}))} } } \\ {{\mathbf{V}}_{21}^{^{(({\mathbf{t}}-{\mathbf{r}})\times {\mathbf{r}})} } } &\quad {{\mathbf{V}}_{22}^{^{(({\mathbf{t}}-{\mathbf{r}})\times ({\mathbf{t}}-{\mathbf{r}}))} } } \end{array}}\right ] \\ \boldsymbol {\Sigma }\boldsymbol{ {'} }=&\left [{\begin{array}{cc} { \boldsymbol {\Sigma }_{11}^{^{({\mathbf{r}}\times {\mathbf{r}})} } } &\quad {0} \\ {0} &\quad {0} \end{array}}\right ]. \end{align*}

Performing matrix multiplications on the right hand side of (4) it is possible to express matrix ${\mathbf{XR}}$ as in (5), where the submatrices dimensions have been omitted $$\begin{equation} \left [{\begin{array}{cc} {{\mathbf{XR}}_{11} } &\quad {{\mathbf{XR}}_{12} } \\ {{\mathbf{XR}}_{21} } &\quad {{\mathbf{XR}}_{22} } \end{array}}\right ]=\left [{\begin{array}{cc} {{\mathbf{U}}_{11} \boldsymbol {\Sigma }_{11} {\mathbf{V}}_{11} } &\quad {{\mathbf{U}}_{11} \boldsymbol {\Sigma }_{11} {\mathbf{V}}_{12} } \\ {{\mathbf{U}}_{21} \boldsymbol {\Sigma }_{11} {\mathbf{V}}_{11} } &\quad {{\mathbf{U}}_{21} \boldsymbol {\Sigma }_{11} {\mathbf{V}}_{12} } \end{array}}\right ]. \end{equation}$$ View Source\begin{equation} \left [{\begin{array}{cc} {{\mathbf{XR}}_{11} } &\quad {{\mathbf{XR}}_{12} } \\ {{\mathbf{XR}}_{21} } &\quad {{\mathbf{XR}}_{22} } \end{array}}\right ]=\left [{\begin{array}{cc} {{\mathbf{U}}_{11} \boldsymbol {\Sigma }_{11} {\mathbf{V}}_{11} } &\quad {{\mathbf{U}}_{11} \boldsymbol {\Sigma }_{11} {\mathbf{V}}_{12} } \\ {{\mathbf{U}}_{21} \boldsymbol {\Sigma }_{11} {\mathbf{V}}_{11} } &\quad {{\mathbf{U}}_{21} \boldsymbol {\Sigma }_{11} {\mathbf{V}}_{12} } \end{array}}\right ]. \end{equation}

Then, it can be seen that the only submatrices needed to compute the approximated matrix ${\mathbf{XR}}$ are: $\boldsymbol {\Sigma }_{11}^{({\mathbf{r}}\times {\mathbf{r}})}$ ; ${\mathbf{U}}_{11}^{({\mathbf{r}}\times {\mathbf{r}})}$ ; ${\mathbf{U}}_{21}^{(({\mathbf{m}}-{\mathbf{r}})\times {\mathbf{r}})}$ ; ${\mathbf{V}}_{11}^{({\mathbf{r}}\times {\mathbf{r}})}$ ; and ${\mathbf{V}}_{12}^{({\mathbf{r}}\times ({\mathbf{t}}-{\mathbf{r}}))}$ .

Noting the dimensions of each submatrix and observing that $\boldsymbol {\Sigma }_{\mathbf{11}}$ is diagonal, it can be found that the total number of elements to be stored for those submatrices equals $(m+t+1) \times r$ .

C. Compression Ratio

The extent of compression achieved by a coding scheme can be measured by a CR. The term CR has been defined in several ways in the literature. In many contexts, the CR is computed by dividing the size of the original data by the size of the compressed data. A ${\rm CR}= 4$ , for example, means that the data has been compressed with the ratio 4:1. Alternatively, it can be said that the volume of the compressed data is 25% of the original data. In this paper, the CR is computed by (6), which expresses the ratio between the total number of elements in the original matrix ${\mathbf{X}}$ (measurements) and the total number of elements in the submatrices that are needed to compute matrix ${\mathbf{XR}}$ $$\begin{equation} {\rm CR} = \frac {m\times t}{(m+t+1)\times r}. \end{equation}$$ View Source\begin{equation} {\rm CR} = \frac {m\times t}{(m+t+1)\times r}. \end{equation}

It can be noted from (6) that, given a measurement set, the effectiveness of the data compression will basically depend on the number of SVs, $r$ , to be retained after performing the SVD.

Communications network bottlenecks will limit the value of CR that can be considered acceptable for a given data compression task. So, alternatively, it is possible to rearrange expression (6) so that one can directly compute the number of SVs that need to be retained in order to achieve a given CR $$\begin{equation} r = \frac {m\times t}{(m+t+1)\times {\rm CR}}. \end{equation}$$ View Source\begin{equation} r = \frac {m\times t}{(m+t+1)\times {\rm CR}}. \end{equation}

Expression (7) will be used to directly compute the value of $r$ that will meet a given compression requirement. However, it should be noted that according to this expression $r$ can assume any real-valued number, while only integer values are practical, once it represents the number of SVs to be retained in $\boldsymbol {\Sigma }$ . For practical purposes, a rounding of $r$ to the nearest integer value will be a good approximation to meet the specified CR requirement. This will be illustrated in Section VI.

D. Loss of Information

As discussed in Section IV, lossy compression methods can be very effective for data compression, but this comes with a cost, which is the loss of information that will not be retrieved when the original data is reconstructed. Then, data compression should be carried out in a way that a good tradeoff between the CR and loss of information is achieved. In other words, data compression should not result in loss of information that renders the reconstructed matrix of limited use to the applications that would employ it as input data.

In this paper, the loss of information is measured in terms of the mean absolute error (MAE) and the mean percentage error (MPE) observed when comparing the reconstructed data matrix with the original one. The expressions for computing the MAE and MPE are $$\begin{align} {\rm MAE}=&\frac {1}{(m\times t)} \sum _{i=1}^{m}\sum _{j=1}^{t}\left |{X(i,j)-XR(i,j)}\right | \\ {\rm MPE}=&\frac {1}{(m\times t)} \sum _{i=1}^{m}\sum _{j=1}^{t}\left |{\frac {X(i,j)-XR(i,j)}{X(i,j)} }\right | \times 100. \end{align}$$ View Source\begin{align} {\rm MAE}=&\frac {1}{(m\times t)} \sum _{i=1}^{m}\sum _{j=1}^{t}\left |{X(i,j)-XR(i,j)}\right | \\ {\rm MPE}=&\frac {1}{(m\times t)} \sum _{i=1}^{m}\sum _{j=1}^{t}\left |{\frac {X(i,j)-XR(i,j)}{X(i,j)} }\right | \times 100. \end{align}

E. Algorithm

The block diagram in Fig. 4 illustrates the flow of data when employing the proposed methodology. The data compression algorithm performed at the sending end (for example, a data concentrator or a regional data and control center) can be summarized by the following steps.

1. Form data matrix ${\mathbf{X}}$ as illustrated in Fig. 2.

2. Perform SVD to obtain matrices ${\mathbf{U}}$ , $\boldsymbol {\Sigma }$ , and ${\mathbf{V}}$ .

3. Based on a value of $r$ chosen to achieve a given CR, form submatrices $\boldsymbol {\Sigma }_{11}^{({\mathbf{r}}\times {\mathbf{r}})}$ ; ${\mathbf{U}}_{11}^{({\mathbf{r}}\times {\mathbf{r}})}$ ; ${\mathbf{U}}_{21}^{(({\mathbf{m}}-{\mathbf{r}})\times {\mathbf{r}})}$ ; ${\mathbf{V}}_{11}^{({\mathbf{r}}\times {\mathbf{r}})}$ ; and ${\mathbf{V}}_{12}^{({\mathbf{r}}\times ({\mathbf{t}}-{\mathbf{r}}))}$ .

4. Reconstruct matrix ${\mathbf{X}}$ by computing ${\mathbf{XR}}$ according to (5).

5. Evaluate the loss of information by computing MAE and MPE using (8) and (9), respectively.

Fig. 4.

Data flow.

Show All

Before sending the compressed data it is possible to check if the loss of information after data compression is acceptable or not for the targeted applications. If it is not acceptable, data compression can be performed again by increasing the number of SVs to be retained and repeating the steps 3)–5). This will reduce the loss of information, but with the cost of making data compression less effective. Once a good tradeoff between CR and loss of information is achieved, the data is transmitted. At the receiving end, the data matrix can be reconstructed using (5), in the same way indicated by step 4) of the presented algorithm. It is important to observe that when an acceptable tradeoff is not achieved by using SVD, the application of a different data compression technique should be considered, which means that steps 2)–4) of the algorithm are changed accordingly.

It should be noted that other quantities can be calculated to measure the loss of information. As this calculation is performed still at the sending end, information about expected inaccuracies in the reconstructed data can also be sent along with the compressed data, which can be useful depending on the intended application. Among the applications that will benefit from the proposed approach are those devoted to steady state analysis, such as power flow and state estimation [31]. In those studies, forecasted quantities are often employed and those may be far less accurate than the reconstructed data. The investigation of the influence of the proposed approach on specific applications will be the subject of future publication.

A. Data Description

Input data for tests have been preprocessed using raw data from the Electricity North West Company that owns and operates the electric power distribution network in the North West of England, which includes the regions of Cumbria, Lancashire, Greater Manchester and parts of North Yorkshire, Derbyshire, and Cheshire. The company has recently launched the customer load active system services (CLASS) project, which aims to increase the capacity of the electricity network by using voltage control to manage electricity consumption at peak times. As part of the CLASS project a significant amount of data has been collected in the year of 2014, by installing metering equipment in many different substations of a trial area that represents 17% of the company’s network, serving around 470 000 customers. The employed data is available in [32], where several measurements collected since April 2014 can be found. Measurements were collected at 1 min or 1 s time intervals. More details about the CLASS project are described in [32].

The next section shows results obtained with the proposed method for tests with measurement data collected in a 1-min basis, from transformers at 50 substations, during the whole day of December 10, 2014. Those include measurements of three-phase voltages and currents, active power, reactive power, and power factor, taken from a total of 900 m at 1440 time instants (minutes). This corresponds to a total of 1 296 000 measurements and a $900 \times 1440$ data matrix.

It should be noted that the choice to show results obtained with data collected on December 10, 2014 was arbitrary and simply because this was the most recent date with available data when the tests were run. Similar results were obtained when performing additional tests using data collected on other dates.

B. Results

The algorithm presented in Section V-E was tested to evaluate the CR and loss of information for the data matrix containing the 1 296 000 measurements collected on December 10, 2014. Different situations have been considered, in which different CRs were aimed. The obtained results are presented in Table I. The number of retained SVs was calculated using (7), where the target CRs (TCRs) were those shown in the first column of Table I. The second column of Table I shows the number of SVs $r$ computed by (7) in order to achieve the corresponding TCR. As $r$ must be an integer number, the computed values are always rounded to their nearest integer values. This results in the CRs shown in the third column of Table I, some of them slightly different from the TCRs. The mean global absolute and percentage errors are also presented, along with the corresponding margin of errors for a confidence interval of 95%. Mean square errors have also been computed and are very small, ranging from 0.0001 to 0.0079, those extreme values corresponding to the CRs of 4:1 and 92.3:1, respectively.

TABLE I CRs Versus Mean Global Errors

From Table I, it can be seen that very good tradeoff between data compression and loss of information have been achieved.

Tables II and III present more detailed results and show also the mean errors (in percent and absolute) per type of variable being measured. From Table II, it is possible to observe that more significant percentage errors are associated with the reactive power, while voltage, currents, and active power measurements present very low errors when reconstructed. Despite the larger percentage errors observed in the reconstruction of the reactive power measurements, it is also possible to observe that those can be significantly reduced if lower CRs are allowed.

TABLE II Mean Errors per Measured Quantity (in Percent)

TABLE III MAEs per Measured Quantity

The results in Table III show that the reconstructed data also presents low absolute errors, considering that the measurement values found in the original data lie in the ranges shown in Table IV for each variable.

TABLE IV Data Values Ranges

Regarding power factor and, particularly, reactive power measurements, it was also observed that data reconstructed with high percentage errors are not necessarily associated with high absolute errors. In many cases, high percentage errors are obtained when reconstructing power factor or reactive power measurements whose values are close to zero. In such cases, small absolute errors may be associated with large percentage errors.

The maximum errors for voltage, current, and active power measurements are presented in Table V. However, it is important to note that those maximum errors can be seen as outliers. As an example, consider the CR of 92.3:1, which is associated with the largest reconstruction errors. In this case, 95.38% of the reconstructed active power measurements present errors below 5%, the same happening to 94.31% and 100% of current and voltage magnitude measurements, respectively. Maximum errors associated with power factor and reactive power measurements can be very high, as explained before. So, these errors are not meaningful and are not shown in Table V. However, considering again the worst case of the CR of 92.25:1, 82.35% of the reconstructed reactive power measurements present errors below the corresponding MPE shown in Table II. The same happens to 86.02% of the power factor measurements.

TABLE V Maximum Errors per Measured Quantity (in Percent)

C. Comparative Analysis

In order to compare the performance of the data compression obtained when employing SVD, additional tests have been carried out using the DWT for the compression of the same dataset. Different Daubechies’ wavelets [33] have been tested, as well as different thresholds and levels of decomposition. Table VI shows results obtained with SVD and with the order 2 Daubechies’ wavelet (db2) and five levels of decomposition. Similar results were obtained when testing different Daubechies’ wavelets.

TABLE VI Results With SVD and DWT—Mean Errors

It can be seen that SVD is capable of achieving better tradeoff for higher CRs. This can be associated with the fact that in SVD the first SVs carry the most relevant information about the data. Fig. 5 shows the 150 largest SVs obtained when applying the SVD technique. In order to allow a better visualization of the SVs decay, the three largest SVs have been omitted. Regarding the db2 model, different thresholds have been employed, aiming to retain the number of wavelet coefficients that would result in the same CRs shown for SVD. However, due to the discrete nature of this problem, it was not always possible to exactly match the CRs obtained with both techniques. As a result, the CRs presented in the first and in the fourth columns of Table VI are in some cases slightly different.

Fig. 5.

Decay of the 150 largest SVs.

Show All

When using the DWT, the margin of errors associated with MPE and MAE, for a confidence interval of 95%, presented characteristics similar to those obtained when applying SVD and shown in Table I.

The maximum errors observed with DWT are presented in Table VII, along with the ones previously obtained with SVD. When comparing such results it is possible to observe that for higher CRs the maximum errors obtained with DWT are much larger than those obtained with SVD. On the other hand, maximum errors obtained with the application of DWT are smaller for lower CRs. In order to achieve higher CRs with the DWT (db2), it is necessary to discard not only detail coefficients, but also some approximation coefficients [33], resulting in larger reconstruction errors.

TABLE VII Results With SVD and DWT—Maximum Errors (in Percent)

As it happened for the results obtained with SVD, maximum errors associated with power factor and reactive power measurements presented very large values when using DWT. As discussed in Section VI-B, this happens when attempting to reconstruct power factor or reactive power measurements whose original values are close to zero, which may result in high percentage errors.

For lower CRs the results obtained with the DWT tend to be better. However, regarding the tested data, the reconstruction errors for lower CRs are already small for both methods, with the exception of the errors associated with the reconstruction of reactive power and power factor measurements. The MPEs associated with reactive power measurements (MPE_Q) and power factor measurements (MPE_PF), computed when using SVD and DWT, are shown in Table VIII.

TABLE VIII Results for Reactive Power and Power Factor Measurements

From the obtained results, one can think of combine SVD and DWT in order to obtain a tradeoff that suits a given data compression problem. If, for example, MPEs below 2%, per measured quantity, are acceptable for a given application, then SVD can be employed to compress only voltage, current and active power measurements, while a DWT is employed to compress reactive power and power factor measurements. Table IX shows the CRs achieved and the associated errors when using such strategy. Taking into account the number of measurements associated with each variable, the global CR, for the whole data set, will be 27.8:1. The resulting CR is better than the one that would have been obtained if SVD or DWT alone was employed.

TABLE IX Results Combining SVD and DWT

The CRs achieved with the tested data when applying lossless methods, such as Block Zip 2 and Lempel-Ziv-Markov chain-Algorithm, were approximately 6.6:1 and 7.8:1, respectively. As previously mentioned, it is not the objective of this paper to select the best technique to perform a given data compression task in the smart grid, but to present SVD as a technique that is worth applying and exploring in such context. The application of SVD is straightforward, not requiring the exploration of different parameters or models, and can lead to competitive results. As illustrated in Table IX, it can also be combined with other data compression technique to achieve better data compression results. This strategy can be easily addressed by the proposed methodology.

D. Comments

According to the proposed methodology, the measurement data will be reconstructed at the receiving end by performing simple matrix multiplications using the transmitted compressed data. One important feature of the proposed approach is that the whole process, including the reconstruction phase, can be performed in advance, before sending the compressed data. This brings some advantages, which are discussed next.