I Exported Them Again From Matlab
Last year I wrote an article on improving the operation of the save function. The commodity discussed various ways by which we tin can store Matlab data on disk. However, in many cases nosotros are interested in a byte-stream serialization, in social club to transmit information to external processes.
The asking to get a serialized byte-stream of Matlab data has been around for many years (case), but MathWorks has never released a documented way of serializing and unserializing data, except by storing onto a disk file and later loading it from file. Naturally, using a disk file significantly degrades performance. Nosotros could always use a RAM-disk or wink memory for improved performance, but in any case this seems like a major overkill to such a simple requirement.
In last yr's article, I presented a File Substitution utility for such generic serialization/deserialization. However, that utility is express in the types of data that it supports, and while it is relatively fast, there is a much better, more generic and faster solution.
The solution appears to employ the undocumented built-in functions getByteStreamFromArray and getArrayFromByteStream , which are apparently used internally by the save and load functions. The usage is very simple:
byteStream = getByteStreamFromArray(anyData); % 1xN uint8 array anyData = getArrayFromByteStream(byteStream);
byteStream = getByteStreamFromArray(anyData); % 1xN uint8 array anyData = getArrayFromByteStream(byteStream);
Many Matlab functions, documented and undocumented alike, are divers in XML files within the %matlabroot%/bin/registry/ binder; our specific functions can be found in %matlabroot%/bin/registry/hgbuiltins.xml. While other functions include information nearly their location and number of input/output args, these functions do non. Their only XML attribute is type = ":all:", which seems to indicate that they take all data types as input. Despite the fact that the functions are defined in hgbuiltins.xml, they are not limited to HG objects – we can serialize basically whatsoever Matlab data: structs, form objects, numeric/cell arrays, sparse data, Coffee handles, timers, etc. For case:
% Simple Matlab data >> byteStream = getByteStreamFromArray( pi ) % 1x72 uint8 array byteStream = Columns 1 through 19 0 i 73 77 0 0 0 0 14 0 0 0 56 0 0 0 half dozen 0 0 Columns 20 through 38 0 8 0 0 0 6 0 0 0 0 0 0 0 v 0 0 0 8 0 Columns 39 through 57 0 0 1 0 0 0 ane 0 0 0 one 0 0 0 0 0 0 0 9 Columns 58 through 72 0 0 0 8 0 0 0 24 45 68 84 251 33 9 64 >> getArrayFromByteStream(byteStream) ans = 3.14159265358979 % A cell assortment of several data types >> byteStream = getByteStreamFromArray( { pi, 'abc', struct ( 'a',5 ) } ); % 1x312 uint8 assortment >> getArrayFromByteStream(byteStream) ans = [ 3.14159265358979 ] 'abc' [1x1 struct ] % A Java object >> byteStream = getByteStreamFromArray(coffee.awt.Color.cherry-red ); % 1x408 uint8 assortment >> getArrayFromByteStream(byteStream) ans = coffee.awt.Color [r=255,g=0,b=0 ] % A Matlab timer >> byteStream = getByteStreamFromArray(timer); % 1x2160 uint8 assortment >> getArrayFromByteStream(byteStream) Timer Object: timer-two Timer Settings ExecutionMode: singleShot Menses: 1 BusyMode: drib Running: off Callbacks TimerFcn: '' ErrorFcn: '' StartFcn: '' StopFcn: '' % A Matlab class object >> byteStream = getByteStreamFromArray(matlab.Arrangement ); % 1x1760 uint8 array >> getArrayFromByteStream(byteStream) ans = System: matlab.Arrangement
% Unproblematic Matlab data >> byteStream = getByteStreamFromArray(pi) % 1x72 uint8 assortment byteStream = Columns ane through xix 0 i 73 77 0 0 0 0 14 0 0 0 56 0 0 0 6 0 0 Columns twenty through 38 0 8 0 0 0 6 0 0 0 0 0 0 0 five 0 0 0 8 0 Columns 39 through 57 0 0 1 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 nine Columns 58 through 72 0 0 0 viii 0 0 0 24 45 68 84 251 33 9 64 >> getArrayFromByteStream(byteStream) ans = 3.14159265358979 % A cell array of several data types >> byteStream = getByteStreamFromArray({pi, 'abc', struct('a',5)}); % 1x312 uint8 array >> getArrayFromByteStream(byteStream) ans = [iii.14159265358979] 'abc' [1x1 struct] % A Java object >> byteStream = getByteStreamFromArray(coffee.awt.Color.blood-red); % 1x408 uint8 assortment >> getArrayFromByteStream(byteStream) ans = java.awt.Colour[r=255,g=0,b=0] % A Matlab timer >> byteStream = getByteStreamFromArray(timer); % 1x2160 uint8 array >> getArrayFromByteStream(byteStream) Timer Object: timer-2 Timer Settings ExecutionMode: singleShot Period: 1 BusyMode: driblet Running: off Callbacks TimerFcn: '' ErrorFcn: '' StartFcn: '' StopFcn: '' % A Matlab class object >> byteStream = getByteStreamFromArray(matlab.System); % 1x1760 uint8 array >> getArrayFromByteStream(byteStream) ans = Organisation: matlab.System
Serializing HG objects
Of course, we can also serialize/deserialize also HG controls, plots/axes and even entire figures. When doing and so, information technology is important to serialize the handle of the object, rather than its numeric handle, since we are interested in serializing the graphic object, not the scalar numeric value of the handle:
% Serializing a simple figure with toolbar and menubar takes most 0.5 MB ! >> hFig = handle( figure ); % a new default Matlab figure >> length (getByteStreamFromArray(hFig) ) ans = 479128 % Removing the menubar and toolbar removes much of this amount: >> set up (hFig, 'menuBar','none', 'toolbar','none' ) >> length (getByteStreamFromArray(hFig) ) ans = 11848 %!!! % Plot lines are not about as "expensive" as the toolbar/menubar >> x=0:.01:five; hp=plot (10,sin (x) ); >> byteStream = getByteStreamFromArray(hFig); >> length (byteStream) ans = 33088 >> delete (hFig); >> hFig2 = getArrayFromByteStream(byteStream) hFig2 = figure
% Serializing a unproblematic figure with toolbar and menubar takes nearly 0.5 MB ! >> hFig = handle(figure); % a new default Matlab figure >> length(getByteStreamFromArray(hFig)) ans = 479128 % Removing the menubar and toolbar removes much of this amount: >> set(hFig, 'menuBar','none', 'toolbar','none') >> length(getByteStreamFromArray(hFig)) ans = 11848 %!!! % Plot lines are not nearly every bit "expensive" as the toolbar/menubar >> ten=0:.01:five; hp=plot(x,sin(x)); >> byteStream = getByteStreamFromArray(hFig); >> length(byteStream) ans = 33088 >> delete(hFig); >> hFig2 = getArrayFromByteStream(byteStream) hFig2 = figure
The interesting thing hither is that when we deserialize a byte-stream of an HG object, it is automatically rendered onscreen. This could be very useful for persistence mechanisms of GUI applications. For example, we tin can relieve the figure handles in file so that if the awarding crashes and relaunches, information technology simply loads the file and we get exactly the same GUI state, consummate with graphs and what-not, simply equally before the crash. Although the figure was deleted in the concluding instance, deserializing the data caused the figure to reappear.
Nosotros practice not demand to serialize the entire figure. Instead, we could choose to serialize only a specific plot line or axes. For example:
>> 10=0:0.01:5; hp=plot (x,sin (10) ); >> byteStream = getByteStreamFromArray(handle(hp) ); % 1x13080 uint8 array >> hLine = getArrayFromByteStream(byteStream) ans = graph2d.lineseries
>> x=0:0.01:5; hp=plot(ten,sin(x)); >> byteStream = getByteStreamFromArray(handle(hp)); % 1x13080 uint8 array >> hLine = getArrayFromByteStream(byteStream) ans = graph2d.lineseries
This could too be used to easily clone (copy) whatsoever figure or other HG object, by simply calling getArrayFromByteStream (note the corresponding copyobj function, which I bet uses the same underlying mechanism).
Also notation that unlike HG objects, deserialized timers are NOT automatically restarted; perhaps the Running property is labeled transient or dependent. Properties defined with these attributes are apparently not serialized.
Performance aspects
Using the builtin getByteStreamFromArray and getArrayFromByteStream functions tin provide meaning performance speedups when caching Matlab data. In fact, it could be used to shop otherwise unsupported objects using the salve -v6 or savefast alternatives, which I discussed in my save operation article. Robin Ince has shown how this can exist used to reduce the combined caching/uncaching run-time from 115 secs with plain-vanilla salvage , to just 11 secs using savefast . Robin hasn't tested this in his post, just since the serialized data is a elementary uint8 array, it is intrinsically supported by the save -v6 option, which is the fastest alternative of all:
>> byteStream = getByteStreamFromArray(hFig); >> tic, save ( 'examination.mat','-v6','byteStream' ); toc Elapsed time is 0.001924 seconds. >> load ( 'test.mat' ) >> data = load ( 'exam.mat' ) information = byteStream: [1x33256 uint8 ] >> getArrayFromByteStream(data.byteStream ) ans = figure
>> byteStream = getByteStreamFromArray(hFig); >> tic, save('test.mat','-v6','byteStream'); toc Elapsed time is 0.001924 seconds. >> load('test.mat') >> data = load('test.mat') data = byteStream: [1x33256 uint8] >> getArrayFromByteStream(data.byteStream) ans = figure
Moreover, nosotros can now use coffee.util.Hashtable to store a cache map of whatever Matlab data, rather than use the much slower and more limited containers.Map class provided in Matlab.
Finally, note that equally built-in functions, these functions could change without prior find on any future Matlab release.
MEX interface – mxSerialize/mxDeserialize
To complete the picture, MEX includes a couple of undocumented functions mxSerialize and mxDeserialize, which correspond to the above functions. getByteStreamFromArray and getArrayFromByteStream apparently call them internally, since they provide the same results. Back in 2007, Brad Phelan wrote a MEX wrapper that could exist used straight in Matlab (mxSerialize.c, mxDeserialize.c). The C interface was very elementary, and then was the usage:
#include "mex.h" EXTERN_C mxArray* mxSerialize(mxArray const * ) ; EXTERN_C mxArray* mxDeserialize( const void *, size_t ) ; void mexFunction( int nlhs, mxArray *plhs[ ], int nrhs, const mxArray *prhs[ ] ) { if (nlhs && nrhs) { plhs[ 0 ] = (mxArray * ) mxSerialize(prhs[ 0 ] ) ; //plhs[0] = (mxArray *) mxDeserialize(mxGetData(prhs[0]), mxGetNumberOfElements(prhs[0])); } }
#include "mex.h" EXTERN_C mxArray* mxSerialize(mxArray const *); EXTERN_C mxArray* mxDeserialize(const void *, size_t); void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[]) { if (nlhs && nrhs) { plhs[0] = (mxArray *) mxSerialize(prhs[0]); //plhs[0] = (mxArray *) mxDeserialize(mxGetData(prhs[0]), mxGetNumberOfElements(prhs[0])); } }
Unfortunately, MathWorks has removed the C interface for these functions from libmx in R2014a, keeping simply their C++ interfaces:
mxArray* matrix:: detail :: noninlined :: mx_array_api :: mxSerialize (mxArray const *anyData) mxArray* matrix:: detail :: noninlined :: mx_array_api :: mxDeserialize ( void const *byteStream, unsigned __int64 numberOfBytes) mxArray* matrix:: particular :: noninlined :: mx_array_api :: mxDeserializeWithTag ( void const *byteStream, unsigned __int64 numberOfBytes, char const * *tagName)
mxArray* matrix::detail::noninlined::mx_array_api::mxSerialize(mxArray const *anyData) mxArray* matrix::detail::noninlined::mx_array_api::mxDeserialize(void const *byteStream, unsigned __int64 numberOfBytes) mxArray* matrix::detail::noninlined::mx_array_api::mxDeserializeWithTag(void const *byteStream, unsigned __int64 numberOfBytes, char const* *tagName)
These are not the but MEX functions that were removed from libmx in R2014a. Hundreds of other C functions were too removed with them, some of them quite of import (e.g., mxCreateSharedDataCopy). A few hundred new C++ functions were added in their place, but I fearfulness that these are non attainable to MEX users without a lawmaking alter (meet beneath). libmx has e'er changed between Matlab releases, only not so drastically for many years. If you rely on any undocumented MEX functions in your lawmaking, at present would be a good time to recheck information technology, earlier R2014a is officially released.
Thanks to Bastian Ebeling, we tin can still use these interfaces in our MEX code by just renaming the MEX file from .c to .cpp and modifying the lawmaking as follows:
#include "mex.h" // MX_API_VER has unfortunately not changed between R2013b and R2014a, // so nosotros apply the new MATRIX_DLL_EXPORT_SYM as an ugly hack instead #if divers(__cplusplus) && divers(MATRIX_DLL_EXPORT_SYM) #define EXTERN_C extern namespace matrix{ namespace particular{ namespace noninlined{ namespace mx_array_api{
0 Response to "I Exported Them Again From Matlab"
Post a Comment