diff --git a/.gitignore b/.gitignore index 8d35cb3..e3fbbf7 100644 --- a/.gitignore +++ b/.gitignore @@ -1,2 +1,7 @@ __pycache__ -*.pyc +*.py[cod] + +dist/ + +requirements-dev.lock +requirements.lock diff --git a/README b/README.md similarity index 100% rename from README rename to README.md diff --git a/igor/binarywave.py b/igor/binarywave.py deleted file mode 100644 index 6d87d14..0000000 --- a/igor/binarywave.py +++ /dev/null @@ -1,655 +0,0 @@ -# Copyright (C) 2010-2012 W. Trevor King -# -# This file is part of igor. -# -# igor is free software: you can redistribute it and/or modify it under the -# terms of the GNU Lesser General Public License as published by the Free -# Software Foundation, either version 3 of the License, or (at your option) any -# later version. -# -# igor is distributed in the hope that it will be useful, but WITHOUT ANY -# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR -# A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more -# details. -# -# You should have received a copy of the GNU Lesser General Public License -# along with igor. If not, see . - -"Read IGOR Binary Wave files into Numpy arrays." - -# Based on WaveMetric's Technical Note 003, "Igor Binary Format" -# ftp://ftp.wavemetrics.net/IgorPro/Technical_Notes/TN003.zip -# From ftp://ftp.wavemetrics.net/IgorPro/Technical_Notes/TN000.txt -# We place no restrictions on copying Technical Notes, with the -# exception that you cannot resell them. So read, enjoy, and -# share. We hope IGOR Technical Notes will provide you with lots of -# valuable information while you are developing IGOR applications. - -from __future__ import absolute_import -import array as _array -import struct as _struct -import sys as _sys -import types as _types - -import numpy as _numpy - -from . import LOG as _LOG -from .struct import Structure as _Structure -from .struct import DynamicStructure as _DynamicStructure -from .struct import Field as _Field -from .struct import DynamicField as _DynamicField -from .util import assert_null as _assert_null -from .util import byte_order as _byte_order -from .util import need_to_reorder_bytes as _need_to_reorder_bytes -from .util import checksum as _checksum - - -# Numpy doesn't support complex integers by default, see -# http://mail.python.org/pipermail/python-dev/2002-April/022408.html -# http://mail.scipy.org/pipermail/numpy-discussion/2007-October/029447.html -# So we roll our own types. See -# http://docs.scipy.org/doc/numpy/user/basics.rec.html -# http://docs.scipy.org/doc/numpy/reference/generated/numpy.dtype.html -complexInt8 = _numpy.dtype([('real', _numpy.int8), ('imag', _numpy.int8)]) -complexInt16 = _numpy.dtype([('real', _numpy.int16), ('imag', _numpy.int16)]) -complexInt32 = _numpy.dtype([('real', _numpy.int32), ('imag', _numpy.int32)]) -complexUInt8 = _numpy.dtype([('real', _numpy.uint8), ('imag', _numpy.uint8)]) -complexUInt16 = _numpy.dtype( - [('real', _numpy.uint16), ('imag', _numpy.uint16)]) -complexUInt32 = _numpy.dtype( - [('real', _numpy.uint32), ('imag', _numpy.uint32)]) - - -class StaticStringField (_DynamicField): - _null_terminated = False - _array_size_field = None - def __init__(self, *args, **kwargs): - if 'array' not in kwargs: - kwargs['array'] = True - super(StaticStringField, self).__init__(*args, **kwargs) - - def post_unpack(self, parents, data): - wave_structure = parents[-1] - wave_data = self._get_structure_data(parents, data, wave_structure) - d = self._normalize_string(wave_data[self.name]) - wave_data[self.name] = d - - def _normalize_string(self, d): - if isinstance(d, bytes): - pass - elif hasattr(d, 'tobytes'): - d = d.tobytes() - elif hasattr(d, 'tostring'): # Python 2 compatibility - d = d.tostring() - else: - d = b''.join(d) - if self._array_size_field: - start = 0 - strings = [] - for count in self.counts: - end = start + count - if end > start: - strings.append(d[start:end]) - if self._null_terminated: - strings[-1] = strings[-1].split(b'\x00', 1)[0] - start = end - elif self._null_terminated: - d = d.split(b'\x00', 1)[0] - return d - - -class NullStaticStringField (StaticStringField): - _null_terminated = True - - -# Begin IGOR constants and typedefs from IgorBin.h - -# From IgorMath.h -TYPE_TABLE = { # (key: integer flag, value: numpy dtype) - 0:None, # Text wave, not handled in ReadWave.c - 1:_numpy.complex, # NT_CMPLX, makes number complex. - 2:_numpy.float32, # NT_FP32, 32 bit fp numbers. - 3:_numpy.complex64, - 4:_numpy.float64, # NT_FP64, 64 bit fp numbers. - 5:_numpy.complex128, - 8:_numpy.int8, # NT_I8, 8 bit signed integer. Requires Igor Pro - # 2.0 or later. - 9:complexInt8, - 0x10:_numpy.int16,# NT_I16, 16 bit integer numbers. Requires Igor - # Pro 2.0 or later. - 0x11:complexInt16, - 0x20:_numpy.int32,# NT_I32, 32 bit integer numbers. Requires Igor - # Pro 2.0 or later. - 0x21:complexInt32, -# 0x40:None, # NT_UNSIGNED, Makes above signed integers -# # unsigned. Requires Igor Pro 3.0 or later. - 0x48:_numpy.uint8, - 0x49:complexUInt8, - 0x50:_numpy.uint16, - 0x51:complexUInt16, - 0x60:_numpy.uint32, - 0x61:complexUInt32, -} - -# From wave.h -MAXDIMS = 4 - -# From binary.h -BinHeader1 = _Structure( # `version` field pulled out into Wave - name='BinHeader1', - fields=[ - _Field('l', 'wfmSize', help='The size of the WaveHeader2 data structure plus the wave data plus 16 bytes of padding.'), - _Field('h', 'checksum', help='Checksum over this header and the wave header.'), - ]) - -BinHeader2 = _Structure( # `version` field pulled out into Wave - name='BinHeader2', - fields=[ - _Field('l', 'wfmSize', help='The size of the WaveHeader2 data structure plus the wave data plus 16 bytes of padding.'), - _Field('l', 'noteSize', help='The size of the note text.'), - _Field('l', 'pictSize', default=0, help='Reserved. Write zero. Ignore on read.'), - _Field('h', 'checksum', help='Checksum over this header and the wave header.'), - ]) - -BinHeader3 = _Structure( # `version` field pulled out into Wave - name='BinHeader3', - fields=[ - _Field('l', 'wfmSize', help='The size of the WaveHeader2 data structure plus the wave data plus 16 bytes of padding.'), - _Field('l', 'noteSize', help='The size of the note text.'), - _Field('l', 'formulaSize', help='The size of the dependency formula, if any.'), - _Field('l', 'pictSize', default=0, help='Reserved. Write zero. Ignore on read.'), - _Field('h', 'checksum', help='Checksum over this header and the wave header.'), - ]) - -BinHeader5 = _Structure( # `version` field pulled out into Wave - name='BinHeader5', - fields=[ - _Field('h', 'checksum', help='Checksum over this header and the wave header.'), - _Field('l', 'wfmSize', help='The size of the WaveHeader5 data structure plus the wave data.'), - _Field('l', 'formulaSize', help='The size of the dependency formula, if any.'), - _Field('l', 'noteSize', help='The size of the note text.'), - _Field('l', 'dataEUnitsSize', help='The size of optional extended data units.'), - _Field('l', 'dimEUnitsSize', help='The size of optional extended dimension units.', count=MAXDIMS, array=True), - _Field('l', 'dimLabelsSize', help='The size of optional dimension labels.', count=MAXDIMS, array=True), - _Field('l', 'sIndicesSize', help='The size of string indicies if this is a text wave.'), - _Field('l', 'optionsSize1', default=0, help='Reserved. Write zero. Ignore on read.'), - _Field('l', 'optionsSize2', default=0, help='Reserved. Write zero. Ignore on read.'), - ]) - - -# From wave.h -MAX_WAVE_NAME2 = 18 # Maximum length of wave name in version 1 and 2 - # files. Does not include the trailing null. -MAX_WAVE_NAME5 = 31 # Maximum length of wave name in version 5 - # files. Does not include the trailing null. -MAX_UNIT_CHARS = 3 - -# Header to an array of waveform data. - -# `wData` field pulled out into DynamicWaveDataField1 -WaveHeader2 = _DynamicStructure( - name='WaveHeader2', - fields=[ - _Field('h', 'type', help='See types (e.g. NT_FP64) above. Zero for text waves.'), - _Field('P', 'next', default=0, help='Used in memory only. Write zero. Ignore on read.'), - NullStaticStringField('c', 'bname', help='Name of wave plus trailing null.', count=MAX_WAVE_NAME2+2), - _Field('h', 'whVersion', default=0, help='Write 0. Ignore on read.'), - _Field('h', 'srcFldr', default=0, help='Used in memory only. Write zero. Ignore on read.'), - _Field('P', 'fileName', default=0, help='Used in memory only. Write zero. Ignore on read.'), - _Field('c', 'dataUnits', default=0, help='Natural data units go here - null if none.', count=MAX_UNIT_CHARS+1, array=True), - _Field('c', 'xUnits', default=0, help='Natural x-axis units go here - null if none.', count=MAX_UNIT_CHARS+1, array=True), - _Field('l', 'npnts', help='Number of data points in wave.'), - _Field('h', 'aModified', default=0, help='Used in memory only. Write zero. Ignore on read.'), - _Field('d', 'hsA', help='X value for point p = hsA*p + hsB'), - _Field('d', 'hsB', help='X value for point p = hsA*p + hsB'), - _Field('h', 'wModified', default=0, help='Used in memory only. Write zero. Ignore on read.'), - _Field('h', 'swModified', default=0, help='Used in memory only. Write zero. Ignore on read.'), - _Field('h', 'fsValid', help='True if full scale values have meaning.'), - _Field('d', 'topFullScale', help='The min full scale value for wave.'), # sic, 'min' should probably be 'max' - _Field('d', 'botFullScale', help='The min full scale value for wave.'), - _Field('c', 'useBits', default=0, help='Used in memory only. Write zero. Ignore on read.'), - _Field('c', 'kindBits', default=0, help='Reserved. Write zero. Ignore on read.'), - _Field('P', 'formula', default=0, help='Used in memory only. Write zero. Ignore on read.'), - _Field('l', 'depID', default=0, help='Used in memory only. Write zero. Ignore on read.'), - _Field('L', 'creationDate', help='DateTime of creation. Not used in version 1 files.'), - _Field('c', 'wUnused', default=0, help='Reserved. Write zero. Ignore on read.', count=2, array=True), - _Field('L', 'modDate', help='DateTime of last modification.'), - _Field('P', 'waveNoteH', help='Used in memory only. Write zero. Ignore on read.'), - ]) - -# `sIndices` pointer unset (use Wave5_data['sIndices'] instead). This -# field is filled in by DynamicStringIndicesDataField. -# `wData` field pulled out into DynamicWaveDataField5 -WaveHeader5 = _DynamicStructure( - name='WaveHeader5', - fields=[ - _Field('P', 'next', help='link to next wave in linked list.'), - _Field('L', 'creationDate', help='DateTime of creation.'), - _Field('L', 'modDate', help='DateTime of last modification.'), - _Field('l', 'npnts', help='Total number of points (multiply dimensions up to first zero).'), - _Field('h', 'type', help='See types (e.g. NT_FP64) above. Zero for text waves.'), - _Field('h', 'dLock', default=0, help='Reserved. Write zero. Ignore on read.'), - _Field('c', 'whpad1', default=0, help='Reserved. Write zero. Ignore on read.', count=6, array=True), - _Field('h', 'whVersion', default=1, help='Write 1. Ignore on read.'), - NullStaticStringField('c', 'bname', help='Name of wave plus trailing null.', count=MAX_WAVE_NAME5+1), - _Field('l', 'whpad2', default=0, help='Reserved. Write zero. Ignore on read.'), - _Field('P', 'dFolder', default=0, help='Used in memory only. Write zero. Ignore on read.'), - # Dimensioning info. [0] == rows, [1] == cols etc - _Field('l', 'nDim', help='Number of of items in a dimension -- 0 means no data.', count=MAXDIMS, array=True), - _Field('d', 'sfA', help='Index value for element e of dimension d = sfA[d]*e + sfB[d].', count=MAXDIMS, array=True), - _Field('d', 'sfB', help='Index value for element e of dimension d = sfA[d]*e + sfB[d].', count=MAXDIMS, array=True), - # SI units - _Field('c', 'dataUnits', default=0, help='Natural data units go here - null if none.', count=MAX_UNIT_CHARS+1, array=True), - _Field('c', 'dimUnits', default=0, help='Natural dimension units go here - null if none.', count=(MAXDIMS, MAX_UNIT_CHARS+1), array=True), - _Field('h', 'fsValid', help='TRUE if full scale values have meaning.'), - _Field('h', 'whpad3', default=0, help='Reserved. Write zero. Ignore on read.'), - _Field('d', 'topFullScale', help='The max and max full scale value for wave'), # sic, probably "max and min" - _Field('d', 'botFullScale', help='The max and max full scale value for wave.'), # sic, probably "max and min" - _Field('P', 'dataEUnits', default=0, help='Used in memory only. Write zero. Ignore on read.'), - _Field('P', 'dimEUnits', default=0, help='Used in memory only. Write zero. Ignore on read.', count=MAXDIMS, array=True), - _Field('P', 'dimLabels', default=0, help='Used in memory only. Write zero. Ignore on read.', count=MAXDIMS, array=True), - _Field('P', 'waveNoteH', default=0, help='Used in memory only. Write zero. Ignore on read.'), - _Field('l', 'whUnused', default=0, help='Reserved. Write zero. Ignore on read.', count=16, array=True), - # The following stuff is considered private to Igor. - _Field('h', 'aModified', default=0, help='Used in memory only. Write zero. Ignore on read.'), - _Field('h', 'wModified', default=0, help='Used in memory only. Write zero. Ignore on read.'), - _Field('h', 'swModified', default=0, help='Used in memory only. Write zero. Ignore on read.'), - _Field('c', 'useBits', default=0, help='Used in memory only. Write zero. Ignore on read.'), - _Field('c', 'kindBits', default=0, help='Reserved. Write zero. Ignore on read.'), - _Field('P', 'formula', default=0, help='Used in memory only. Write zero. Ignore on read.'), - _Field('l', 'depID', default=0, help='Used in memory only. Write zero. Ignore on read.'), - _Field('h', 'whpad4', default=0, help='Reserved. Write zero. Ignore on read.'), - _Field('h', 'srcFldr', default=0, help='Used in memory only. Write zero. Ignore on read.'), - _Field('P', 'fileName', default=0, help='Used in memory only. Write zero. Ignore on read.'), - _Field('P', 'sIndices', default=0, help='Used in memory only. Write zero. Ignore on read.'), - ]) - - -class DynamicWaveDataField1 (_DynamicField): - def pre_pack(self, parents, data): - raise NotImplementedError() - - def pre_unpack(self, parents, data): - full_structure = parents[0] - wave_structure = parents[-1] - wave_header_structure = wave_structure.fields[1].format - wave_data = self._get_structure_data(parents, data, wave_structure) - version = data['version'] - bin_header = wave_data['bin_header'] - wave_header = wave_data['wave_header'] - - self.count = wave_header['npnts'] - self.data_size = self._get_size(bin_header, wave_header_structure.size) - - type_ = TYPE_TABLE.get(wave_header['type'], None) - if type_: - self.shape = self._get_shape(bin_header, wave_header) - else: # text wave - type_ = _numpy.dtype('S1') - self.shape = (self.data_size,) - # dtype() wrapping to avoid numpy.generic and - # getset_descriptor issues with the builtin numpy types - # (e.g. int32). It has no effect on our local complex - # integers. - self.dtype = _numpy.dtype(type_).newbyteorder( - wave_structure.byte_order) - if (version == 3 and - self.count > 0 and - bin_header['formulaSize'] > 0 and - self.data_size == 0): - """From TN003: - - Igor Pro 2.00 included support for dependency formulae. If - a wave was governed by a dependency formula then the - actual wave data was not written to disk for that wave, - because on loading the wave Igor could recalculate the - data. However,this prevented the wave from being loaded - into an experiment other than the original - experiment. Consequently, in a version of Igor Pro 3.0x, - we changed it so that the wave data was written even if - the wave was governed by a dependency formula. When - reading a binary wave file, you can detect that the wave - file does not contain the wave data by examining the - wfmSize, formulaSize and npnts fields. If npnts is greater - than zero and formulaSize is greater than zero and - the waveDataSize as calculated above is zero, then this is - a file governed by a dependency formula that was written - without the actual wave data. - """ - self.shape = (0,) - elif TYPE_TABLE.get(wave_header['type'], None) is not None: - assert self.data_size == self.count * self.dtype.itemsize, ( - self.data_size, self.count, self.dtype.itemsize, self.dtype) - else: - assert self.data_size >= 0, ( - bin_header['wfmSize'], wave_header_structure.size) - - def _get_size(self, bin_header, wave_header_size): - return bin_header['wfmSize'] - wave_header_size - 16 - - def _get_shape(self, bin_header, wave_header): - return (self.count,) - - def unpack(self, stream): - data_b = stream.read(self.data_size) - try: - data = _numpy.ndarray( - shape=self.shape, - dtype=self.dtype, - buffer=data_b, - order='F', - ) - except: - _LOG.error( - 'could not reshape data from {} to {}'.format( - self.shape, data_b)) - raise - return data - - -class DynamicWaveDataField5 (DynamicWaveDataField1): - "Adds support for multidimensional data." - def _get_size(self, bin_header, wave_header_size): - return bin_header['wfmSize'] - wave_header_size - - def _get_shape(self, bin_header, wave_header): - return [n for n in wave_header['nDim'] if n > 0] or (0,) - - -# End IGOR constants and typedefs from IgorBin.h - - -class DynamicStringField (StaticStringField): - _size_field = None - - def pre_unpack(self, parents, data): - size = self._get_size_data(parents, data) - if self._array_size_field: - self.counts = size - self.count = sum(self.counts) - else: - self.count = size - self.setup() - - def _get_size_data(self, parents, data): - wave_structure = parents[-1] - wave_data = self._get_structure_data(parents, data, wave_structure) - bin_header = wave_data['bin_header'] - return bin_header[self._size_field] - - -class DynamicWaveNoteField (DynamicStringField): - _size_field = 'noteSize' - - -class DynamicDependencyFormulaField (DynamicStringField): - """Optional wave dependency formula - - Excerpted from TN003: - - A wave has a dependency formula if it has been bound by a - statement such as "wave0 := sin(x)". In this example, the - dependency formula is "sin(x)". The formula is stored with - no trailing null byte. - """ - _size_field = 'formulaSize' - # Except when it is stored with a trailing null byte :p. See, for - # example, test/data/mac-version3Dependent.ibw. - _null_terminated = True - - -class DynamicDataUnitsField (DynamicStringField): - """Optional extended data units data - - Excerpted from TN003: - - dataUnits - Present in versions 1, 2, 3, 5. The dataUnits field - stores the units for the data represented by the wave. It is a C - string terminated with a null character. This field supports - units of 0 to 3 bytes. In version 1, 2 and 3 files, longer units - can not be represented. In version 5 files, longer units can be - stored using the optional extended data units section of the - file. - """ - _size_field = 'dataEUnitsSize' - - -class DynamicDimensionUnitsField (DynamicStringField): - """Optional extended dimension units data - - Excerpted from TN003: - - xUnits - Present in versions 1, 2, 3. The xUnits field stores the - X units for a wave. It is a C string terminated with a null - character. This field supports units of 0 to 3 bytes. In - version 1, 2 and 3 files, longer units can not be represented. - - dimUnits - Present in version 5 only. This field is an array of 4 - strings, one for each possible wave dimension. Each string - supports units of 0 to 3 bytes. Longer units can be stored using - the optional extended dimension units section of the file. - """ - _size_field = 'dimEUnitsSize' - _array_size_field = True - - -class DynamicLabelsField (DynamicStringField): - """Optional dimension label data - - From TN003: - - If the wave has dimension labels for dimension d then the - dimLabelsSize[d] field of the BinHeader5 structure will be - non-zero. - - A wave will have dimension labels if a SetDimLabel command has - been executed on it. - - A 3 point 1D wave has 4 dimension labels. The first dimension - label is the label for the dimension as a whole. The next three - dimension labels are the labels for rows 0, 1, and 2. When Igor - writes dimension labels to disk, it writes each dimension label as - a C string (null-terminated) in a field of 32 bytes. - """ - _size_field = 'dimLabelsSize' - _array_size_field = True - - def post_unpack(self, parents, data): - wave_structure = parents[-1] - wave_data = self._get_structure_data(parents, data, wave_structure) - bin_header = wave_data['bin_header'] - d = wave_data[self.name] - dim_labels = [] - start = 0 - for size in bin_header[self._size_field]: - end = start + size - if end > start: - dim_data = d[start:end] - chunks = [] - for i in range(size//32): - chunks.append(dim_data[32*i:32*(i+1)]) - labels = [b''] - for chunk in chunks: - labels[-1] = labels[-1] + b''.join(chunk) - if b'\x00' in chunk: - labels.append(b'') - labels.pop(-1) - start = end - else: - labels = [] - dim_labels.append(labels) - wave_data[self.name] = dim_labels - - -class DynamicStringIndicesDataField (_DynamicField): - """String indices used for text waves only - """ - def pre_pack(self, parents, data): - raise NotImplementedError() - - def pre_unpack(self, parents, data): - wave_structure = parents[-1] - wave_data = self._get_structure_data(parents, data, wave_structure) - bin_header = wave_data['bin_header'] - wave_header = wave_data['wave_header'] - self.string_indices_size = bin_header['sIndicesSize'] - self.count = self.string_indices_size // 4 - if self.count: # make sure we're in a text wave - assert TYPE_TABLE[wave_header['type']] is None, wave_header - self.setup() - - def post_unpack(self, parents, data): - if not self.count: - return - wave_structure = parents[-1] - wave_data = self._get_structure_data(parents, data, wave_structure) - wave_header = wave_data['wave_header'] - wdata = wave_data['wData'] - strings = [] - start = 0 - for i,offset in enumerate(wave_data['sIndices']): - if offset > start: - chars = wdata[start:offset] - strings.append(b''.join(chars)) - start = offset - elif offset == start: - strings.append(b'') - else: - raise ValueError((offset, wave_data['sIndices'])) - wdata = _numpy.array(strings) - shape = [n for n in wave_header['nDim'] if n > 0] or (0,) - try: - wdata = wdata.reshape(shape) - except ValueError: - _LOG.error( - 'could not reshape strings from {} to {}'.format( - shape, wdata.shape)) - raise - wave_data['wData'] = wdata - - -class DynamicVersionField (_DynamicField): - def pre_pack(self, parents, byte_order): - raise NotImplementedError() - - def post_unpack(self, parents, data): - wave_structure = parents[-1] - wave_data = self._get_structure_data(parents, data, wave_structure) - version = wave_data['version'] - if wave_structure.byte_order in '@=': - need_to_reorder_bytes = _need_to_reorder_bytes(version) - wave_structure.byte_order = _byte_order(need_to_reorder_bytes) - _LOG.debug( - 'get byte order from version: {} (reorder? {})'.format( - wave_structure.byte_order, need_to_reorder_bytes)) - else: - need_to_reorder_bytes = False - - old_format = wave_structure.fields[-1].format - if version == 1: - wave_structure.fields[-1].format = Wave1 - elif version == 2: - wave_structure.fields[-1].format = Wave2 - elif version == 3: - wave_structure.fields[-1].format = Wave3 - elif version == 5: - wave_structure.fields[-1].format = Wave5 - elif not need_to_reorder_bytes: - raise ValueError( - 'invalid binary wave version: {}'.format(version)) - - if wave_structure.fields[-1].format != old_format: - _LOG.debug('change wave headers from {} to {}'.format( - old_format, wave_structure.fields[-1].format)) - wave_structure.setup() - elif need_to_reorder_bytes: - wave_structure.setup() - - # we might need to unpack again with the new byte order - return need_to_reorder_bytes - - -class DynamicWaveField (_DynamicField): - def post_unpack(self, parents, data): - return - raise NotImplementedError() # TODO - checksum_size = bin.size + wave.size - wave_structure = parents[-1] - if version == 5: - # Version 5 checksum does not include the wData field. - checksum_size -= 4 - c = _checksum(b, parents[-1].byte_order, 0, checksum_size) - if c != 0: - raise ValueError( - ('This does not appear to be a valid Igor binary wave file. ' - 'Error in checksum: should be 0, is {}.').format(c)) - -Wave1 = _DynamicStructure( - name='Wave1', - fields=[ - _Field(BinHeader1, 'bin_header', help='Binary wave header'), - _Field(WaveHeader2, 'wave_header', help='Wave header'), - DynamicWaveDataField1('f', 'wData', help='The start of the array of waveform data.', count=0, array=True), - ]) - -Wave2 = _DynamicStructure( - name='Wave2', - fields=[ - _Field(BinHeader2, 'bin_header', help='Binary wave header'), - _Field(WaveHeader2, 'wave_header', help='Wave header'), - DynamicWaveDataField1('f', 'wData', help='The start of the array of waveform data.', count=0, array=True), - _Field('x', 'padding', help='16 bytes of padding in versions 2 and 3.', count=16, array=True), - DynamicWaveNoteField('c', 'note', help='Optional wave note data', count=0, array=True), - ]) - -Wave3 = _DynamicStructure( - name='Wave3', - fields=[ - _Field(BinHeader3, 'bin_header', help='Binary wave header'), - _Field(WaveHeader2, 'wave_header', help='Wave header'), - DynamicWaveDataField1('f', 'wData', help='The start of the array of waveform data.', count=0, array=True), - _Field('x', 'padding', help='16 bytes of padding in versions 2 and 3.', count=16, array=True), - DynamicWaveNoteField('c', 'note', help='Optional wave note data', count=0, array=True), - DynamicDependencyFormulaField('c', 'formula', help='Optional wave dependency formula', count=0, array=True), - ]) - -Wave5 = _DynamicStructure( - name='Wave5', - fields=[ - _Field(BinHeader5, 'bin_header', help='Binary wave header'), - _Field(WaveHeader5, 'wave_header', help='Wave header'), - DynamicWaveDataField5('f', 'wData', help='The start of the array of waveform data.', count=0, array=True), - DynamicDependencyFormulaField('c', 'formula', help='Optional wave dependency formula.', count=0, array=True), - DynamicWaveNoteField('c', 'note', help='Optional wave note data.', count=0, array=True), - DynamicDataUnitsField('c', 'data_units', help='Optional extended data units data.', count=0, array=True), - DynamicDimensionUnitsField('c', 'dimension_units', help='Optional dimension label data', count=0, array=True), - DynamicLabelsField('c', 'labels', help="Optional dimension label data", count=0, array=True), - DynamicStringIndicesDataField('P', 'sIndices', help='Dynamic string indices for text waves.', count=0, array=True), - ]) - -Wave = _DynamicStructure( - name='Wave', - fields=[ - DynamicVersionField('h', 'version', help='Version number for backwards compatibility.'), - DynamicWaveField(Wave1, 'wave', help='The rest of the wave data.'), - ]) - - -def load(filename): - if hasattr(filename, 'read'): - f = filename # filename is actually a stream object - else: - f = open(filename, 'rb') - try: - Wave.byte_order = '=' - Wave.setup() - data = Wave.unpack_stream(f) - finally: - if not hasattr(filename, 'read'): - f.close() - - return data - - -def save(filename): - raise NotImplementedError diff --git a/igor/igorpy.py b/igor/igorpy.py deleted file mode 100644 index b28de2e..0000000 --- a/igor/igorpy.py +++ /dev/null @@ -1,283 +0,0 @@ -# This program is in the public domain -"""`igor.py` compatibility layer on top of the `igor` package. - -igor.load('filename') or igor.loads('data') loads the content of an igore file -into memory as a folder structure. - -Returns the root folder. - -Folders have name, path and children. -Children can be indexed by folder[i] or by folder['name']. -To see the whole tree, use: print folder.format() - -The usual igor folder types are given in the technical reports -PTN003.ifn and TN003.ifn. -""" -from __future__ import absolute_import -import io as _io -import locale as _locale -import re as _re -import sys as _sys - -import numpy as _numpy - -from .binarywave import MAXDIMS as _MAXDIMS -from .packed import load as _load -from .record.base import UnknownRecord as _UnknownRecord -from .record.folder import FolderStartRecord as _FolderStartRecord -from .record.folder import FolderEndRecord as _FolderEndRecord -from .record.history import HistoryRecord as _HistoryRecord -from .record.history import GetHistoryRecord as _GetHistoryRecord -from .record.history import RecreationRecord as _RecreationRecord -from .record.packedfile import PackedFileRecord as _PackedFileRecord -from .record.procedure import ProcedureRecord as _ProcedureRecord -from .record.wave import WaveRecord as _WaveRecord -from .record.variables import VariablesRecord as _VariablesRecord - - -__version__='0.10' - - -ENCODING = _locale.getpreferredencoding() or _sys.getdefaultencoding() -PYKEYWORDS = set(('and','as','assert','break','class','continue', - 'def','elif','else','except','exec','finally', - 'for','global','if','import','in','is','lambda', - 'or','pass','print','raise','return','try','with', - 'yield')) -PYID = _re.compile(r"^[^\d\W]\w*$", _re.UNICODE) -def valid_identifier(s): - """Check if a name is a valid identifier""" - return PYID.match(s) and s not in PYKEYWORDS - - -class IgorObject(object): - """ Parent class for all objects the parser can return """ - pass - -class Variables(IgorObject): - """ - Contains system numeric variables (e.g., K0) and user numeric and string variables. - """ - def __init__(self, record): - self.sysvar = record.variables['variables']['sysVars'] - self.uservar = record.variables['variables']['userVars'] - self.userstr = record.variables['variables']['userStrs'] - self.depvar = record.variables['variables'].get('dependentVars', {}) - self.depstr = record.variables['variables'].get('dependentStrs', {}) - - def format(self, indent=0): - return " "*indent+""\ - %(len(self.sysvar), - len(self.uservar)+len(self.userstr), - len(self.depvar)+len(self.depstr)) - -class History(IgorObject): - """ - Contains the experiment's history as plain text. - """ - def __init__(self, data): - self.data = data - def format(self, indent=0): - return " "*indent+"" - -class Wave(IgorObject): - """ - Contains the data for a wave - """ - def __init__(self, record): - d = record.wave['wave'] - self.name = d['wave_header']['bname'].decode(ENCODING) - self.data = d['wData'] - self.fs = d['wave_header']['fsValid'] - self.fstop = d['wave_header']['topFullScale'] - self.fsbottom = d['wave_header']['botFullScale'] - version = record.wave['version'] - if version in [1,2,3]: - dims = [d['wave_header']['npnts']] + [0]*(_MAXDIMS-1) - sfA = [d['wave_header']['hsA']] + [0]*(_MAXDIMS-1) - sfB = [d['wave_header']['hsB']] + [0]*(_MAXDIMS-1) - self.data_units = [d['wave_header']['dataUnits']] - self.axis_units = [d['wave_header']['xUnits']] - else: - dims = d['wave_header']['nDim'] - sfA = d['wave_header']['sfA'] - sfB = d['wave_header']['sfB'] - # TODO find example with multiple data units - if version == 5: - self.data_units = [d['data_units'].decode(ENCODING)] - self.axis_units = [b''.join(d).decode(ENCODING) - for d in d['wave_header']['dimUnits']] - else: - self.data_units = [d['data_units'].decode(ENCODING)] - self.axis_units = [d['dimension_units'].decode(ENCODING)] - - self.data_units.extend(['']*(_MAXDIMS-len(self.data_units))) - self.data_units = tuple(self.data_units) - self.axis_units.extend(['']*(_MAXDIMS-len(self.axis_units))) - self.axis_units = tuple(self.axis_units) - self.axis = [_numpy.linspace(b,b + a * (c - 1),c) for a,b,c in zip(sfA, sfB, dims)] - self.formula = d.get('formula', '') - self.notes = d.get('note', '') - def format(self, indent=0): - if isinstance(self.data, list): - type,size = "text", "%d"%len(self.data) - else: - type,size = "data", "x".join(str(d) for d in self.data.shape) - return " "*indent+"%s %s (%s)"%(self.name, type, size) - - def __array__(self): - return self.data - - __repr__ = __str__ = lambda s: "" % s.format() - -class Recreation(IgorObject): - """ - Contains the experiment's recreation procedures as plain text. - """ - def __init__(self, data): - self.data = data - def format(self, indent=0): - return " "*indent + "" -class Procedure(IgorObject): - """ - Contains the experiment's main procedure window text as plain text. - """ - def __init__(self, data): - self.data = data - def format(self, indent=0): - return " "*indent + "" -class GetHistory(IgorObject): - """ - Not a real record but rather, a message to go back and read the history text. - - The reason for GetHistory is that IGOR runs Recreation when it loads the - datafile. This puts entries in the history that shouldn't be there. The - GetHistory entry simply says that the Recreation has run, and the History - can be restored from the previously saved value. - """ - def __init__(self, data): - self.data = data - def format(self, indent=0): - return " "*indent + "" -class PackedFile(IgorObject): - """ - Contains the data for a procedure file or notebook in packed form. - """ - def __init__(self, data): - self.data = data - def format(self, indent=0): - return " "*indent + "" -class Unknown(IgorObject): - """ - Record type not documented in PTN003/TN003. - """ - def __init__(self, data, type): - self.data = data - self.type = type - def format(self, indent=0): - return " "*indent + ""%self.type - - -class Folder(IgorObject): - """ - Hierarchical record container. - """ - def __init__(self, path): - self.name = path[-1] - self.path = path - self.children = [] - - def __getitem__(self, key): - if isinstance(key, int): - return self.children[key] - else: - for r in self.children: - if isinstance(r, (Folder,Wave)) and r.name == key: - return r - raise KeyError("Folder %s does not exist"%key) - - def __str__(self): - return "" % "/".join(self.path) - - __repr__ = __str__ - - def append(self, record): - """ - Add a record to the folder. - """ - self.children.append(record) - try: - # Record may not have a name, the name may be invalid, or it - # may already be in use. The noname case will be covered by - # record.name raising an attribute error. The others we need - # to test for explicitly. - if valid_identifier(record.name) and not hasattr(self, record.name): - setattr(self, record.name, record) - except AttributeError: - pass - - def format(self, indent=0): - parent = " "*indent+self.name - children = [r.format(indent=indent+2) for r in self.children] - return "\n".join([parent]+children) - - -def loads(s, **kwargs): - """Load an igor file from string""" - stream = _io.BytesIO(s) - return load(stream, **kwargs) - -def load(filename, **kwargs): - """Load an igor file""" - try: - packed_experiment = _load( - filename, initial_byte_order=kwargs.pop('initial_byte_order', '=')) - except ValueError as e: - if e.args[0].startswith('not enough data for the next record header'): - raise IOError('invalid record header; bad pxp file?') - elif e.args[0].startswith('not enough data for the next record'): - raise IOError('final record too long; bad pxp file?') - raise - return _convert(packed_experiment, **kwargs) - -def _convert(packed_experiment, ignore_unknown=True): - records, filesystem = packed_experiment - stack = [Folder(path=['root'])] - for record in records: - if isinstance(record, _UnknownRecord): - if ignore_unknown: - continue - else: - r = Unknown(record.data, type=record.header['recordType']) - elif isinstance(record, _GetHistoryRecord): - r = GetHistory(record.text) - elif isinstance(record, _HistoryRecord): - r = History(record.text) - elif isinstance(record, _PackedFileRecord): - r = PackedFile(record.text) - elif isinstance(record, _ProcedureRecord): - r = Procedure(record.text) - elif isinstance(record, _RecreationRecord): - r = Recreation(record.text) - elif isinstance(record, _VariablesRecord): - r = Variables(record) - elif isinstance(record, _WaveRecord): - r = Wave(record) - else: - r = None - - if isinstance(record, _FolderStartRecord): - path = stack[-1].path + [ - record.null_terminated_text.decode(ENCODING)] - folder = Folder(path) - stack[-1].append(folder) - stack.append(folder) - elif isinstance(record, _FolderEndRecord): - stack.pop() - elif r is None: - raise NotImplementedError(record) - else: - stack[-1].append(r) - if len(stack) != 1: - raise IOError("FolderStart records do not match FolderEnd records") - return stack[0] diff --git a/igor/record/variables.py b/igor/record/variables.py deleted file mode 100644 index a8eaccf..0000000 --- a/igor/record/variables.py +++ /dev/null @@ -1,319 +0,0 @@ -# Copyright (C) 2012 W. Trevor King -# -# This file is part of igor. -# -# igor is free software: you can redistribute it and/or modify it under the -# terms of the GNU Lesser General Public License as published by the Free -# Software Foundation, either version 3 of the License, or (at your option) any -# later version. -# -# igor is distributed in the hope that it will be useful, but WITHOUT ANY -# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR -# A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more -# details. -# -# You should have received a copy of the GNU Lesser General Public License -# along with igor. If not, see . - -import io as _io - -from .. import LOG as _LOG -from ..binarywave import TYPE_TABLE as _TYPE_TABLE -from ..binarywave import NullStaticStringField as _NullStaticStringField -from ..binarywave import DynamicStringField as _DynamicStringField -from ..struct import Structure as _Structure -from ..struct import DynamicStructure as _DynamicStructure -from ..struct import Field as _Field -from ..struct import DynamicField as _DynamicField -from ..util import byte_order as _byte_order -from ..util import need_to_reorder_bytes as _need_to_reorder_bytes -from .base import Record - - -class ListedStaticStringField (_NullStaticStringField): - """Handle string conversions for multi-count dynamic parents. - - If a field belongs to a multi-count dynamic parent, the parent is - called multiple times to parse each count, and the field's - post-unpack hook gets called after the field is unpacked during - each iteration. This requires alternative logic for getting and - setting the string data. The actual string formatting code is not - affected. - """ - def post_unpack(self, parents, data): - parent_structure = parents[-1] - parent_data = self._get_structure_data(parents, data, parent_structure) - d = self._normalize_string(parent_data[-1][self.name]) - parent_data[-1][self.name] = d - - -class ListedStaticStringField (_NullStaticStringField): - """Handle string conversions for multi-count dynamic parents. - - If a field belongs to a multi-count dynamic parent, the parent is - called multiple times to parse each count, and the field's - post-unpack hook gets called after the field is unpacked during - each iteration. This requires alternative logic for getting and - setting the string data. The actual string formatting code is not - affected. - """ - def post_unpack(self, parents, data): - parent_structure = parents[-1] - parent_data = self._get_structure_data(parents, data, parent_structure) - d = self._normalize_string(parent_data[-1][self.name]) - parent_data[-1][self.name] = d - - -class ListedDynamicStrDataField (_DynamicStringField, ListedStaticStringField): - _size_field = 'strLen' - _null_terminated = False - - def _get_size_data(self, parents, data): - parent_structure = parents[-1] - parent_data = self._get_structure_data(parents, data, parent_structure) - return parent_data[-1][self._size_field] - - -class DynamicVarDataField (_DynamicField): - def __init__(self, *args, **kwargs): - if 'array' not in kwargs: - kwargs['array'] = True - super(DynamicVarDataField, self).__init__(*args, **kwargs) - - def pre_pack(self, parents, data): - raise NotImplementedError() - - def post_unpack(self, parents, data): - var_structure = parents[-1] - var_data = self._get_structure_data(parents, data, var_structure) - data = var_data[self.name] - d = {} - for i,value in enumerate(data): - key,value = self._normalize_item(i, value) - d[key] = value - var_data[self.name] = d - - def _normalize_item(self, index, value): - raise NotImplementedError() - - -class DynamicSysVarField (DynamicVarDataField): - def _normalize_item(self, index, value): - name = 'K{}'.format(index) - return (name, value) - - -class DynamicUserVarField (DynamicVarDataField): - def _normalize_item(self, index, value): - name = value['name'] - value = value['num'] - return (name, value) - - -class DynamicUserStrField (DynamicVarDataField): - def _normalize_item(self, index, value): - name = value['name'] - value = value['data'] - return (name, value) - - -class DynamicVarNumField (_DynamicField): - def post_unpack(self, parents, data): - parent_structure = parents[-1] - parent_data = self._get_structure_data(parents, data, parent_structure) - d = self._normalize_numeric_variable(parent_data[-1][self.name]) - parent_data[-1][self.name] = d - - def _normalize_numeric_variable(self, num_var): - t = _TYPE_TABLE[num_var['numType']] - if num_var['numType'] % 2: # complex number - return t(complex(num_var['realPart'], num_var['imagPart'])) - else: - return t(num_var['realPart']) - - -class DynamicFormulaField (_DynamicStringField): - _size_field = 'formulaLen' - _null_terminated = True - - -# From Variables.h -VarHeader1 = _Structure( # `version` field pulled out into VariablesRecord - name='VarHeader1', - fields=[ - _Field('h', 'numSysVars', help='Number of system variables (K0, K1, ...).'), - _Field('h', 'numUserVars', help='Number of user numeric variables -- may be zero.'), - _Field('h', 'numUserStrs', help='Number of user string variables -- may be zero.'), - ]) - -# From Variables.h -VarHeader2 = _Structure( # `version` field pulled out into VariablesRecord - name='VarHeader2', - fields=[ - _Field('h', 'numSysVars', help='Number of system variables (K0, K1, ...).'), - _Field('h', 'numUserVars', help='Number of user numeric variables -- may be zero.'), - _Field('h', 'numUserStrs', help='Number of user string variables -- may be zero.'), - _Field('h', 'numDependentVars', help='Number of dependent numeric variables -- may be zero.'), - _Field('h', 'numDependentStrs', help='Number of dependent string variables -- may be zero.'), - ]) - -# From Variables.h -UserStrVarRec1 = _DynamicStructure( - name='UserStrVarRec1', - fields=[ - ListedStaticStringField('c', 'name', help='Name of the string variable.', count=32), - _Field('h', 'strLen', help='The real size of the following array.'), - ListedDynamicStrDataField('c', 'data'), - ]) - -# From Variables.h -UserStrVarRec2 = _DynamicStructure( - name='UserStrVarRec2', - fields=[ - ListedStaticStringField('c', 'name', help='Name of the string variable.', count=32), - _Field('l', 'strLen', help='The real size of the following array.'), - _Field('c', 'data'), - ]) - -# From Variables.h -VarNumRec = _Structure( - name='VarNumRec', - fields=[ - _Field('h', 'numType', help='Type from binarywave.TYPE_TABLE'), - _Field('d', 'realPart', help='The real part of the number.'), - _Field('d', 'imagPart', help='The imag part if the number is complex.'), - _Field('l', 'reserved', help='Reserved - set to zero.'), - ]) - -# From Variables.h -UserNumVarRec = _DynamicStructure( - name='UserNumVarRec', - fields=[ - ListedStaticStringField('c', 'name', help='Name of the string variable.', count=32), - _Field('h', 'type', help='0 = string, 1 = numeric.'), - DynamicVarNumField(VarNumRec, 'num', help='Type and value of the variable if it is numeric. Not used for string.'), - ]) - -# From Variables.h -UserDependentVarRec = _DynamicStructure( - name='UserDependentVarRec', - fields=[ - ListedStaticStringField('c', 'name', help='Name of the string variable.', count=32), - _Field('h', 'type', help='0 = string, 1 = numeric.'), - _Field(VarNumRec, 'num', help='Type and value of the variable if it is numeric. Not used for string.'), - _Field('h', 'formulaLen', help='The length of the dependency formula.'), - DynamicFormulaField('c', 'formula', help='Start of the dependency formula. A C string including null terminator.'), - ]) - - -class DynamicVarHeaderField (_DynamicField): - def pre_pack(self, parents, data): - raise NotImplementedError() - - def post_unpack(self, parents, data): - var_structure = parents[-1] - var_data = self._get_structure_data( - parents, data, var_structure) - var_header_structure = self.format - data = var_data['var_header'] - sys_vars_field = var_structure.get_field('sysVars') - sys_vars_field.count = data['numSysVars'] - sys_vars_field.setup() - user_vars_field = var_structure.get_field('userVars') - user_vars_field.count = data['numUserVars'] - user_vars_field.setup() - user_strs_field = var_structure.get_field('userStrs') - user_strs_field.count = data['numUserStrs'] - user_strs_field.setup() - if 'numDependentVars' in data: - dependent_vars_field = var_structure.get_field('dependentVars') - dependent_vars_field.count = data['numDependentVars'] - dependent_vars_field.setup() - dependent_strs_field = var_structure.get_field('dependentStrs') - dependent_strs_field.count = data['numDependentStrs'] - dependent_strs_field.setup() - var_structure.setup() - - -Variables1 = _DynamicStructure( - name='Variables1', - fields=[ - DynamicVarHeaderField(VarHeader1, 'var_header', help='Variables header'), - DynamicSysVarField('f', 'sysVars', help='System variables', count=0), - DynamicUserVarField(UserNumVarRec, 'userVars', help='User numeric variables', count=0), - DynamicUserStrField(UserStrVarRec1, 'userStrs', help='User string variables', count=0), - ]) - - -Variables2 = _DynamicStructure( - name='Variables2', - fields=[ - DynamicVarHeaderField(VarHeader2, 'var_header', help='Variables header'), - DynamicSysVarField('f', 'sysVars', help='System variables', count=0), - DynamicUserVarField(UserNumVarRec, 'userVars', help='User numeric variables', count=0), - DynamicUserStrField(UserStrVarRec2, 'userStrs', help='User string variables', count=0), - _Field(UserDependentVarRec, 'dependentVars', help='Dependent numeric variables.', count=0, array=True), - _Field(UserDependentVarRec, 'dependentStrs', help='Dependent string variables.', count=0, array=True), - ]) - - -class DynamicVersionField (_DynamicField): - def pre_pack(self, parents, byte_order): - raise NotImplementedError() - - def post_unpack(self, parents, data): - variables_structure = parents[-1] - variables_data = self._get_structure_data( - parents, data, variables_structure) - version = variables_data['version'] - if variables_structure.byte_order in '@=': - need_to_reorder_bytes = _need_to_reorder_bytes(version) - variables_structure.byte_order = _byte_order(need_to_reorder_bytes) - _LOG.debug( - 'get byte order from version: {} (reorder? {})'.format( - variables_structure.byte_order, need_to_reorder_bytes)) - else: - need_to_reorder_bytes = False - - old_format = variables_structure.fields[-1].format - if version == 1: - variables_structure.fields[-1].format = Variables1 - elif version == 2: - variables_structure.fields[-1].format = Variables2 - elif not need_to_reorder_bytes: - raise ValueError( - 'invalid variables record version: {}'.format(version)) - - if variables_structure.fields[-1].format != old_format: - _LOG.debug('change variables record from {} to {}'.format( - old_format, variables_structure.fields[-1].format)) - variables_structure.setup() - elif need_to_reorder_bytes: - variables_structure.setup() - - # we might need to unpack again with the new byte order - return need_to_reorder_bytes - - -VariablesRecordStructure = _DynamicStructure( - name='VariablesRecord', - fields=[ - DynamicVersionField('h', 'version', help='Version number for this header.'), - _Field(Variables1, 'variables', help='The rest of the variables data.'), - ]) - - -class VariablesRecord (Record): - def __init__(self, *args, **kwargs): - super(VariablesRecord, self).__init__(*args, **kwargs) - # self.header['version'] # record version always 0? - VariablesRecordStructure.byte_order = '=' - VariablesRecordStructure.setup() - stream = _io.BytesIO(bytes(self.data)) - self.variables = VariablesRecordStructure.unpack_stream(stream) - self.namespace = {} - for key,value in self.variables['variables'].items(): - if key not in ['var_header']: - _LOG.debug('update namespace {} with {} for {}'.format( - self.namespace, value, key)) - self.namespace.update(value) diff --git a/igor/script.py b/igor/script.py deleted file mode 100644 index 83fde93..0000000 --- a/igor/script.py +++ /dev/null @@ -1,91 +0,0 @@ -# Copyright (C) 2012-2016 W. Trevor King -# -# This file is part of igor. -# -# igor is free software: you can redistribute it and/or modify it under the -# terms of the GNU Lesser General Public License as published by the Free -# Software Foundation, either version 3 of the License, or (at your option) any -# later version. -# -# igor is distributed in the hope that it will be useful, but WITHOUT ANY -# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR -# A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more -# details. -# -# You should have received a copy of the GNU Lesser General Public License -# along with igor. If not, see . - -"Common code for scripts distributed with the `igor` package." - -from __future__ import absolute_import -import argparse as _argparse -import logging as _logging -import sys as _sys - -try: - import matplotlib as _matplotlib - import matplotlib.pyplot as _matplotlib_pyplot -except ImportError as _matplotlib_import_error: - _matplotlib = None - -from . import __version__ -from . import LOG as _LOG - - -class Script (object): - log_levels = [_logging.ERROR, _logging.WARNING, _logging.INFO, _logging.DEBUG] - - def __init__(self, description=None, filetype='IGOR Binary Wave (.ibw) file'): - self.parser = _argparse.ArgumentParser(description=description) - self.parser.add_argument( - '--version', action='version', - version='%(prog)s {}'.format(__version__)) - self.parser.add_argument( - '-f', '--infile', metavar='FILE', default='-', - help='input {}'.format(filetype)) - self.parser.add_argument( - '-o', '--outfile', metavar='FILE', default='-', - help='file for ASCII output') - self.parser.add_argument( - '-p', '--plot', action='store_const', const=True, - help='use Matplotlib to plot any IGOR waves') - self.parser.add_argument( - '-V', '--verbose', action='count', default=0, - help='increment verbosity') - self._num_plots = 0 - - def run(self, *args, **kwargs): - args = self.parser.parse_args(*args, **kwargs) - if args.infile == '-': - args.infile = _sys.stdin - if args.outfile == '-': - args.outfile = _sys.stdout - if args.verbose > 1: - log_level = self.log_levels[min(args.verbose-1, len(self.log_levels)-1)] - _LOG.setLevel(log_level) - self._run(args) - self.display_plots() - - def _run(self, args): - raise NotImplementedError() - - def plot_wave(self, args, wave, title=None): - if not args.plot: - return # no-op - if not _matplotlib: - raise _matplotlib_import_error - if title is None: - title = wave['wave']['wave_header']['bname'] - figure = _matplotlib_pyplot.figure() - axes = figure.add_subplot(1, 1, 1) - axes.set_title(title) - try: - axes.plot(wave['wave']['wData'], 'r.') - except ValueError as error: - _LOG.error('error plotting {}: {}'.format(title, error)) - pass - self._num_plots += 1 - - def display_plots(self): - if self._num_plots: - _matplotlib_pyplot.show() diff --git a/legacy_files/setup.py b/legacy_files/setup.py new file mode 100644 index 0000000..2ebe62c --- /dev/null +++ b/legacy_files/setup.py @@ -0,0 +1,75 @@ +# Copyright (C) 2011-2016 Paul Kienzle +# W. Trevor King +# +# This file is part of igor. +# +# igor is free software: you can redistribute it and/or modify it under the +# terms of the GNU Lesser General Public License as published by the Free +# Software Foundation, either version 3 of the License, or (at your option) any +# later version. +# +# igor is distributed in the hope that it will be useful, but WITHOUT ANY +# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR +# A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more +# details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with igor. If not, see . + +"igor: interface for reading binary IGOR files." + +from distutils.core import setup +import os.path + +from igor import __version__ + + +package_name = "igor" +_this_dir = os.path.dirname(__file__) + +setup( + name=package_name, + version=__version__, + author="W. Trevor King", + author_email="wking@tremily.us", + maintainer="Conrad Stansbury", + maintainer_email="chstan@berkeley.edu", + url="https://github.com/chstan/igorpy", + download_url="https://github.com/chstan/igorpy/tarball/master", + license="GNU Lesser General Public License v3 or later (LGPLv3+)", + platforms=["all"], + description=__doc__, + long_description=open(os.path.join(_this_dir, "README"), "r").read(), + classifiers=[ + "Development Status :: 3 - Alpha", + "Environment :: Console", + "Intended Audience :: Developers", + "Intended Audience :: Science/Research", + "Operating System :: OS Independent", + "License :: OSI Approved :: GNU Lesser General Public License v3 or later (LGPLv3+)", + "Programming Language :: Python", + "Programming Language :: Python :: 2.7", + "Programming Language :: Python :: 3.2", + "Programming Language :: Python :: 3.3", + "Programming Language :: Python :: 3.4", + "Programming Language :: Python :: 3.5", + "Programming Language :: Python :: 3.6", + "Programming Language :: Python :: 3.7", + "Programming Language :: Python :: 3.8", + "Programming Language :: Python :: 3.9", + "Programming Language :: Python :: 3.10", + "Programming Language :: Python :: 3.11", + "Programming Language :: Python :: 3.12", + "Topic :: Scientific/Engineering", + "Topic :: Software Development :: Libraries :: Python Modules", + ], + packages=[ + "igor", + "igor.record", + ], + scripts=[ + "bin/igorbinarywave.py", + "bin/igorpackedexperiment.py", + ], + provides=["igor ({})".format(__version__)], +) diff --git a/pyproject.toml b/pyproject.toml new file mode 100644 index 0000000..a87dd40 --- /dev/null +++ b/pyproject.toml @@ -0,0 +1,31 @@ +[project] +name = "igorpy" +version = "0.3.2" +description = "Add your description here" +authors = [ + { name = "W. Trevor King", email = "wking@tremily.us" }, + { name = "Conrad Stansbury", email = "chstan@berkely.edu" }, +] +maintainers = [ + { name = "Ryuichi Arafune", email = "ryuichi.arafune@gmail.com" }, +] +dependencies = ["numpy"] +readme = "README.md" +requires-python = ">= 3.8" + +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[tool.rye] +managed = true +dev-dependencies = ["pytest>=7.3.2"] + +[tool.hatch.metadata] +allow-direct-references = true + +[tool.mypy] +ignore_missing_imports = true + +[tool.hatch.build.targets.wheel] +packages = ["src/igor"] diff --git a/setup.py b/setup.py deleted file mode 100644 index 0f3a04a..0000000 --- a/setup.py +++ /dev/null @@ -1,67 +0,0 @@ -# Copyright (C) 2011-2016 Paul Kienzle -# W. Trevor King -# -# This file is part of igor. -# -# igor is free software: you can redistribute it and/or modify it under the -# terms of the GNU Lesser General Public License as published by the Free -# Software Foundation, either version 3 of the License, or (at your option) any -# later version. -# -# igor is distributed in the hope that it will be useful, but WITHOUT ANY -# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR -# A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more -# details. -# -# You should have received a copy of the GNU Lesser General Public License -# along with igor. If not, see . - -"igor: interface for reading binary IGOR files." - -from distutils.core import setup -import os.path - -from igor import __version__ - - -package_name = 'igor' -_this_dir = os.path.dirname(__file__) - -setup(name=package_name, - version=__version__, - author='W. Trevor King', - author_email='wking@tremily.us', - maintainer='Conrad Stansbury', - maintainer_email='chstan@berkeley.edu', - url='https://github.com/chstan/igorpy', - download_url='https://github.com/chstan/igorpy/tarball/master', - license='GNU Lesser General Public License v3 or later (LGPLv3+)', - platforms=['all'], - description=__doc__, - long_description=open(os.path.join(_this_dir, 'README'), 'r').read(), - classifiers=[ - 'Development Status :: 3 - Alpha', - 'Environment :: Console', - 'Intended Audience :: Developers', - 'Intended Audience :: Science/Research', - 'Operating System :: OS Independent', - 'License :: OSI Approved :: GNU Lesser General Public License v3 or later (LGPLv3+)', - 'Programming Language :: Python', - 'Programming Language :: Python :: 2.7', - 'Programming Language :: Python :: 3.2', - 'Programming Language :: Python :: 3.3', - 'Programming Language :: Python :: 3.4', - 'Programming Language :: Python :: 3.5', - 'Topic :: Scientific/Engineering', - 'Topic :: Software Development :: Libraries :: Python Modules', - ], - packages=[ - 'igor', - 'igor.record', - ], - scripts=[ - 'bin/igorbinarywave.py', - 'bin/igorpackedexperiment.py', - ], - provides=['igor ({})'.format(__version__)], - ) diff --git a/igor/__init__.py b/src/igor/__init__.py similarity index 66% rename from igor/__init__.py rename to src/igor/__init__.py index 7f0220a..9bdcbc0 100644 --- a/igor/__init__.py +++ b/src/igor/__init__.py @@ -15,16 +15,21 @@ # You should have received a copy of the GNU Lesser General Public License # along with igor. If not, see . +# Memo: there is a difference with igor2 + "Interface for reading binary IGOR files." -__version__ = '0.3.1' +from logging import DEBUG, INFO, getLogger, Formatter, StreamHandler +__all__ = ("__version__",) -import logging as _logging +__version__ = "0.3.2" -LOG = _logging.getLogger('igor') -LOG.setLevel(_logging.ERROR) -LOG.addHandler(_logging.StreamHandler()) -LOG.handlers[-1].setFormatter( - _logging.Formatter('%(name)s - %(levelname)s - %(message)s')) +LOGLEVELS = (DEBUG, INFO) +LOGLEVEL = LOGLEVELS[1] +LOG = getLogger("igor") +LOG.setLevel(LOGLEVEL) +LOG.addHandler(StreamHandler()) +formatter = Formatter("%(name)s - %(levelname)s - %(message)s") +LOG.handlers[-1].setFormatter(formatter) diff --git a/src/igor/__init__.pyi b/src/igor/__init__.pyi new file mode 100644 index 0000000..dd79831 --- /dev/null +++ b/src/igor/__init__.pyi @@ -0,0 +1,4 @@ +from __future__ import annotations +from _typeshed import Incomplete + +LOG: Incomplete diff --git a/src/igor/_typing.py b/src/igor/_typing.py new file mode 100644 index 0000000..4ff0435 --- /dev/null +++ b/src/igor/_typing.py @@ -0,0 +1,56 @@ +from __future__ import annotations + +import numpy as np +from typing import TypedDict, TypeAlias, Literal +from numpy.typing import NDArray + + +from _typeshed import Incomplete +from .record.base import UnknownRecord +from .record.folder import FolderStartRecord +from .record.folder import FolderEndRecord +from .record.history import HistoryRecord +from .record.history import GetHistoryRecord +from .record.history import RecreationRecord +from .record.procedure import ProcedureRecord +from .record.wave import WaveRecord +from .record.variables import VariablesRecord + +BYTEORDER = Literal[">", "<", "=", "@", "!", ""] + + +class WAVETYPE(TypedDict, total=False): + binheader: dict[str, int | NDArray[np.int_]] + waveheader: dict[ + str, + str | bytes | float | NDArray[np.str_] | NDArray[np.int_] | NDArray[np.float_], + ] + wData: NDArray[np.float_] | NDArray[np.complex_] + formula: bytes + note: bytes + data_units: bytes + dimension_units: bytes + labels: list[list[Incomplete]] + sIndics: NDArray[np.float_] + + +class IBW(TypedDict, total=False): + version: int + wave: WAVETYPE + + +class PXP(TypedDict, total=False): + pass + + +RECORDS: TypeAlias = list[ + UnknownRecord + | VariablesRecord + | HistoryRecord + | WaveRecord + | FolderStartRecord + | FolderEndRecord + | RecreationRecord + | ProcedureRecord + | GetHistoryRecord +] diff --git a/src/igor/binarywave.py b/src/igor/binarywave.py new file mode 100644 index 0000000..8caff98 --- /dev/null +++ b/src/igor/binarywave.py @@ -0,0 +1,997 @@ +# Copyright (C) 2010-2012 W. Trevor King +# +# This file is part of igor. +# +# igor is free software: you can redistribute it and/or modify it under the +# terms of the GNU Lesser General Public License as published by the Free +# Software Foundation, either version 3 of the License, or (at your option) any +# later version. +# +# igor is distributed in the hope that it will be useful, but WITHOUT ANY +# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR +# A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more +# details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with igor. If not, see . + +# Memo not essential diff + +"Read IGOR Binary Wave files into Numpy arrays." + +# Based on WaveMetric's Technical Note 003, "Igor Binary Format" +# ftp://ftp.wavemetrics.net/IgorPro/Technical_Notes/TN003.zip +# From ftp://ftp.wavemetrics.net/IgorPro/Technical_Notes/TN000.txt +# We place no restrictions on copying Technical Notes, with the +# exception that you cannot resell them. So read, enjoy, and +# share. We hope IGOR Technical Notes will provide you with lots of +# valuable information while you are developing IGOR applications. + + +import numpy as np + +from . import LOG +from .struct import Structure +from .struct import DynamicStructure +from .struct import DynamicField +from .struct import Field +from .util import byte_order +from .util import need_to_reorder_bytes as _need_to_reorder_bytes +from .util import checksum + + +# Numpy doesn't support complex integers by default, see +# http://mail.python.org/pipermail/python-dev/2002-April/022408.html +# http://mail.scipy.org/pipermail/numpy-discussion/2007-October/029447.html +# So we roll our own types. See +# http://docs.scipy.org/doc/numpy/user/basics.rec.html +# http://docs.scipy.org/doc/numpy/reference/generated/numpy.dtype.html +complexInt8 = np.dtype([("real", np.int8), ("imag", np.int8)]) +complexInt16 = np.dtype([("real", np.int16), ("imag", np.int16)]) +complexInt32 = np.dtype([("real", np.int32), ("imag", np.int32)]) +complexUInt8 = np.dtype([("real", np.uint8), ("imag", np.uint8)]) +complexUInt16 = np.dtype([("real", np.uint16), ("imag", np.uint16)]) +complexUInt32 = np.dtype([("real", np.uint32), ("imag", np.uint32)]) + + +class StaticStringField(DynamicField): + _null_terminated = False + _array_size_field = None + + def __init__(self, *args, **kwargs): + if "array" not in kwargs: + kwargs["array"] = True + super().__init__(*args, **kwargs) + + def post_unpack(self, parents, data): + wave_structure = parents[-1] + wave_data = self._get_structure_data(parents, data, wave_structure) + d = self._normalize_string(wave_data[self.name]) + wave_data[self.name] = d + + def _normalize_string(self, d): + if isinstance(d, bytes): + pass + elif hasattr(d, "tobytes"): + d = d.tobytes() + elif hasattr(d, "tostring"): # Python 2 compatibility + d = d.tostring() + else: + d = b"".join(d) + if self._array_size_field: + start = 0 + strings = [] + for count in self.counts: + end = start + count + if end > start: + strings.append(d[start:end]) + if self._null_terminated: + strings[-1] = strings[-1].split(b"\x00", 1)[0] + start = end + elif self._null_terminated: + d = d.split(b"\x00", 1)[0] + return d + + +class NullStaticStringField(StaticStringField): + _null_terminated = True + + +# Begin IGOR constants and typedefs from IgorBin.h + +# From IgorMath.h +TYPE_TABLE = { # (key: integer flag, value: numpy dtype) + 0: None, # Text wave, not handled in ReadWave.c + 1: complex, # NT_CMPLX, makes number complex. + 2: np.float32, # NT_FP32, 32 bit fp numbers. + 3: np.complex64, + 4: np.float64, # NT_FP64, 64 bit fp numbers. + 5: np.complex128, + 8: np.int8, # NT_I8, 8 bit signed integer. Requires Igor Pro + # 2.0 or later. + 9: complexInt8, + 0x10: np.int16, # NT_I16, 16 bit integer numbers. Requires Igor + # Pro 2.0 or later. + 0x11: complexInt16, + 0x20: np.int32, # NT_I32, 32 bit integer numbers. Requires Igor + # Pro 2.0 or later. + 0x21: complexInt32, + # 0x40:None, # NT_UNSIGNED, Makes above signed integers + # # unsigned. Requires Igor Pro 3.0 or later. + 0x48: np.uint8, + 0x49: complexUInt8, + 0x50: np.uint16, + 0x51: complexUInt16, + 0x60: np.uint32, + 0x61: complexUInt32, +} + +# From wave.h +MAXDIMS = 4 + +# From binary.h +BinHeader1 = Structure( # `version` field pulled out into Wave + name="BinHeader1", + fields=[ + Field( + "l", + "wfmSize", + help="The size of the WaveHeader2 data structure plus the wave data plus 16 bytes of padding.", + ), + Field("h", "checksum", help="Checksum over this header and the wave header."), + ], +) + +BinHeader2 = Structure( # `version` field pulled out into Wave + name="BinHeader2", + fields=[ + Field( + "l", + "wfmSize", + help="The size of the WaveHeader2 data structure plus the wave data plus 16 bytes of padding.", + ), + Field("l", "noteSize", help="The size of the note text."), + Field("l", "pictSize", default=0, help="Reserved. Write zero. Ignore on read."), + Field("h", "checksum", help="Checksum over this header and the wave header."), + ], +) + +BinHeader3 = Structure( # `version` field pulled out into Wave + name="BinHeader3", + fields=[ + Field( + "l", + "wfmSize", + help="The size of the WaveHeader2 data structure plus the wave data plus 16 bytes of padding.", + ), + Field("l", "noteSize", help="The size of the note text."), + Field("l", "formulaSize", help="The size of the dependency formula, if any."), + Field("l", "pictSize", default=0, help="Reserved. Write zero. Ignore on read."), + Field("h", "checksum", help="Checksum over this header and the wave header."), + ], +) + +BinHeader5 = Structure( # `version` field pulled out into Wave + name="BinHeader5", + fields=[ + Field("h", "checksum", help="Checksum over this header and the wave header."), + Field( + "l", + "wfmSize", + help="The size of the WaveHeader5 data structure plus the wave data.", + ), + Field("l", "formulaSize", help="The size of the dependency formula, if any."), + Field("l", "noteSize", help="The size of the note text."), + Field("l", "dataEUnitsSize", help="The size of optional extended data units."), + Field( + "l", + "dimEUnitsSize", + help="The size of optional extended dimension units.", + count=MAXDIMS, + array=True, + ), + Field( + "l", + "dimLabelsSize", + help="The size of optional dimension labels.", + count=MAXDIMS, + array=True, + ), + Field( + "l", + "sIndicesSize", + help="The size of string indices if this is a text wave.", + ), + Field( + "l", "optionsSize1", default=0, help="Reserved. Write zero. Ignore on read." + ), + Field( + "l", "optionsSize2", default=0, help="Reserved. Write zero. Ignore on read." + ), + ], +) + + +# From wave.h +MAX_WAVE_NAME2 = 18 # Maximum length of wave name in version 1 and 2 +# files. Does not include the trailing null. +MAX_WAVE_NAME5 = 31 # Maximum length of wave name in version 5 +# files. Does not include the trailing null. +MAX_UNIT_CHARS = 3 + +# Header to an array of waveform data. + +# `wData` field pulled out into DynamicWaveDataField1 +WaveHeader2 = DynamicStructure( + name="WaveHeader2", + fields=[ + Field("h", "type", help="See types (e.g. NT_FP64) above. Zero for text waves."), + Field( + "P", + "next", + default=0, + help="Used in memory only. Write zero. Ignore on read.", + ), + NullStaticStringField( + "c", + "bname", + help="Name of wave plus trailing null.", + count=MAX_WAVE_NAME2 + 2, + ), + Field("h", "whVersion", default=0, help="Write 0. Ignore on read."), + Field( + "h", + "srcFldr", + default=0, + help="Used in memory only. Write zero. Ignore on read.", + ), + Field( + "P", + "fileName", + default=0, + help="Used in memory only. Write zero. Ignore on read.", + ), + Field( + "c", + "dataUnits", + default=0, + help="Natural data units go here - null if none.", + count=MAX_UNIT_CHARS + 1, + array=True, + ), + Field( + "c", + "xUnits", + default=0, + help="Natural x-axis units go here - null if none.", + count=MAX_UNIT_CHARS + 1, + array=True, + ), + Field("l", "npnts", help="Number of data points in wave."), + Field( + "h", + "aModified", + default=0, + help="Used in memory only. Write zero. Ignore on read.", + ), + Field("d", "hsA", help="X value for point p = hsA*p + hsB"), + Field("d", "hsB", help="X value for point p = hsA*p + hsB"), + Field( + "h", + "wModified", + default=0, + help="Used in memory only. Write zero. Ignore on read.", + ), + Field( + "h", + "swModified", + default=0, + help="Used in memory only. Write zero. Ignore on read.", + ), + Field("h", "fsValid", help="True if full scale values have meaning."), + Field( + "d", "topFullScale", help="The min full scale value for wave." + ), # sic, 'min' should probably be 'max' + Field("d", "botFullScale", help="The min full scale value for wave."), + Field( + "c", + "useBits", + default=0, + help="Used in memory only. Write zero. Ignore on read.", + ), + Field("c", "kindBits", default=0, help="Reserved. Write zero. Ignore on read."), + Field( + "P", + "formula", + default=0, + help="Used in memory only. Write zero. Ignore on read.", + ), + Field( + "l", + "depID", + default=0, + help="Used in memory only. Write zero. Ignore on read.", + ), + Field( + "L", + "creationDate", + help="DateTime of creation. Not used in version 1 files.", + ), + Field( + "c", + "wUnused", + default=0, + help="Reserved. Write zero. Ignore on read.", + count=2, + array=True, + ), + Field("L", "modDate", help="DateTime of last modification."), + Field( + "P", "waveNoteH", help="Used in memory only. Write zero. Ignore on read." + ), + ], +) + +# `sIndices` pointer unset (use Wave5_data['sIndices'] instead). This +# field is filled in by DynamicStringIndicesDataField. +# `wData` field pulled out into DynamicWaveDataField5 +WaveHeader5 = DynamicStructure( + name="WaveHeader5", + fields=[ + Field("P", "next", help="link to next wave in linked list."), + Field("L", "creationDate", help="DateTime of creation."), + Field("L", "modDate", help="DateTime of last modification."), + Field( + "l", + "npnts", + help="Total number of points (multiply dimensions up to first zero).", + ), + Field("h", "type", help="See types (e.g. NT_FP64) above. Zero for text waves."), + Field("h", "dLock", default=0, help="Reserved. Write zero. Ignore on read."), + Field( + "c", + "whpad1", + default=0, + help="Reserved. Write zero. Ignore on read.", + count=6, + array=True, + ), + Field("h", "whVersion", default=1, help="Write 1. Ignore on read."), + NullStaticStringField( + "c", + "bname", + help="Name of wave plus trailing null.", + count=MAX_WAVE_NAME5 + 1, + ), + Field("l", "whpad2", default=0, help="Reserved. Write zero. Ignore on read."), + Field( + "P", + "dFolder", + default=0, + help="Used in memory only. Write zero. Ignore on read.", + ), + # Dimensioning info. [0] == rows, [1] == cols etc + Field( + "l", + "nDim", + help="Number of of items in a dimension -- 0 means no data.", + count=MAXDIMS, + array=True, + ), + Field( + "d", + "sfA", + help="Index value for element e of dimension d = sfA[d]*e + sfB[d].", + count=MAXDIMS, + array=True, + ), + Field( + "d", + "sfB", + help="Index value for element e of dimension d = sfA[d]*e + sfB[d].", + count=MAXDIMS, + array=True, + ), + # SI units + Field( + "c", + "dataUnits", + default=0, + help="Natural data units go here - null if none.", + count=MAX_UNIT_CHARS + 1, + array=True, + ), + Field( + "c", + "dimUnits", + default=0, + help="Natural dimension units go here - null if none.", + count=(MAXDIMS, MAX_UNIT_CHARS + 1), + array=True, + ), + Field("h", "fsValid", help="TRUE if full scale values have meaning."), + Field("h", "whpad3", default=0, help="Reserved. Write zero. Ignore on read."), + Field( + "d", "topFullScale", help="The max and max full scale value for wave" + ), # sic, probably "max and min" + Field( + "d", "botFullScale", help="The max and max full scale value for wave." + ), # sic, probably "max and min" + Field( + "P", + "dataEUnits", + default=0, + help="Used in memory only. Write zero. Ignore on read.", + ), + Field( + "P", + "dimEUnits", + default=0, + help="Used in memory only. Write zero. Ignore on read.", + count=MAXDIMS, + array=True, + ), + Field( + "P", + "dimLabels", + default=0, + help="Used in memory only. Write zero. Ignore on read.", + count=MAXDIMS, + array=True, + ), + Field( + "P", + "waveNoteH", + default=0, + help="Used in memory only. Write zero. Ignore on read.", + ), + Field( + "l", + "whUnused", + default=0, + help="Reserved. Write zero. Ignore on read.", + count=16, + array=True, + ), + # The following stuff is considered private to Igor. + Field( + "h", + "aModified", + default=0, + help="Used in memory only. Write zero. Ignore on read.", + ), + Field( + "h", + "wModified", + default=0, + help="Used in memory only. Write zero. Ignore on read.", + ), + Field( + "h", + "swModified", + default=0, + help="Used in memory only. Write zero. Ignore on read.", + ), + Field( + "c", + "useBits", + default=0, + help="Used in memory only. Write zero. Ignore on read.", + ), + Field("c", "kindBits", default=0, help="Reserved. Write zero. Ignore on read."), + Field( + "P", + "formula", + default=0, + help="Used in memory only. Write zero. Ignore on read.", + ), + Field( + "l", + "depID", + default=0, + help="Used in memory only. Write zero. Ignore on read.", + ), + Field("h", "whpad4", default=0, help="Reserved. Write zero. Ignore on read."), + Field( + "h", + "srcFldr", + default=0, + help="Used in memory only. Write zero. Ignore on read.", + ), + Field( + "P", + "fileName", + default=0, + help="Used in memory only. Write zero. Ignore on read.", + ), + Field( + "P", + "sIndices", + default=0, + help="Used in memory only. Write zero. Ignore on read.", + ), + ], +) + + +class DynamicWaveDataField1(DynamicField): + def pre_pack(self, parents, data): + raise NotImplementedError() + + def pre_unpack(self, parents, data): + parents[0] + wave_structure = parents[-1] + wave_header_structure = wave_structure.fields[1].format + wave_data = self._get_structure_data(parents, data, wave_structure) + version = data["version"] + bin_header = wave_data["bin_header"] + wave_header = wave_data["wave_header"] + + self.count = wave_header["npnts"] + self.data_size = self._get_size(bin_header, wave_header_structure.size) + + type_ = TYPE_TABLE.get(wave_header["type"], None) + if type_: + self.shape = self._get_shape(bin_header, wave_header) + else: # text wave + type_ = np.dtype("S1") + self.shape = (self.data_size,) + # dtype() wrapping to avoid numpy.generic and + # getset_descriptor issues with the builtin numpy types + # (e.g. int32). It has no effect on our local complex + # integers. + self.dtype = np.dtype(type_).newbyteorder(wave_structure.byte_order) + if ( + version == 3 + and self.count > 0 + and bin_header["formulaSize"] > 0 + and self.data_size == 0 + ): + """From TN003: + + Igor Pro 2.00 included support for dependency formulae. If + a wave was governed by a dependency formula then the + actual wave data was not written to disk for that wave, + because on loading the wave Igor could recalculate the + data. However,this prevented the wave from being loaded + into an experiment other than the original + experiment. Consequently, in a version of Igor Pro 3.0x, + we changed it so that the wave data was written even if + the wave was governed by a dependency formula. When + reading a binary wave file, you can detect that the wave + file does not contain the wave data by examining the + wfmSize, formulaSize and npnts fields. If npnts is greater + than zero and formulaSize is greater than zero and + the waveDataSize as calculated above is zero, then this is + a file governed by a dependency formula that was written + without the actual wave data. + """ + self.shape = (0,) + elif TYPE_TABLE.get(wave_header["type"], None) is not None: + assert self.data_size == self.count * self.dtype.itemsize, ( + self.data_size, + self.count, + self.dtype.itemsize, + self.dtype, + ) + else: + assert self.data_size >= 0, ( + bin_header["wfmSize"], + wave_header_structure.size, + ) + + def _get_size(self, bin_header, wave_header_size): + return bin_header["wfmSize"] - wave_header_size - 16 + + def _get_shape(self, bin_header, wave_header): + return (self.count,) + + def unpack(self, stream): + data_b = stream.read(self.data_size) + try: + data = np.ndarray( + shape=self.shape, + dtype=self.dtype, + buffer=data_b, + order="F", + ) + except: + LOG.error(f"could not reshape data from {self.shape} to {data_b}") + raise + return data + + +class DynamicWaveDataField5(DynamicWaveDataField1): + "Adds support for multidimensional data." + + def _get_size(self, bin_header, wave_header_size): + return bin_header["wfmSize"] - wave_header_size + + def _get_shape(self, bin_header, wave_header): + return [n for n in wave_header["nDim"] if n > 0] or (0,) + + +# End IGOR constants and typedefs from IgorBin.h + + +class DynamicStringField(StaticStringField): + _size_field = None + + def pre_unpack(self, parents, data): + size = self._get_size_data(parents, data) + if self._array_size_field: + self.counts = size + self.count = sum(self.counts) + else: + self.count = size + self.setup() + + def _get_size_data(self, parents, data): + wave_structure = parents[-1] + wave_data = self._get_structure_data(parents, data, wave_structure) + bin_header = wave_data["bin_header"] + return bin_header[self._size_field] + + +class DynamicWaveNoteField(DynamicStringField): + _size_field = "noteSize" + + +class DynamicDependencyFormulaField(DynamicStringField): + """Optional wave dependency formula + + Excerpted from TN003: + + A wave has a dependency formula if it has been bound by a + statement such as "wave0 := sin(x)". In this example, the + dependency formula is "sin(x)". The formula is stored with + no trailing null byte. + """ + + _size_field = "formulaSize" + # Except when it is stored with a trailing null byte :p. See, for + # example, test/data/mac-version3Dependent.ibw. + _null_terminated = True + + +class DynamicDataUnitsField(DynamicStringField): + """Optional extended data units data + + Excerpted from TN003: + + dataUnits - Present in versions 1, 2, 3, 5. The dataUnits field + stores the units for the data represented by the wave. It is a C + string terminated with a null character. This field supports + units of 0 to 3 bytes. In version 1, 2 and 3 files, longer units + can not be represented. In version 5 files, longer units can be + stored using the optional extended data units section of the + file. + """ + + _size_field = "dataEUnitsSize" + + +class DynamicDimensionUnitsField(DynamicStringField): + """Optional extended dimension units data + + Excerpted from TN003: + + xUnits - Present in versions 1, 2, 3. The xUnits field stores the + X units for a wave. It is a C string terminated with a null + character. This field supports units of 0 to 3 bytes. In + version 1, 2 and 3 files, longer units can not be represented. + + dimUnits - Present in version 5 only. This field is an array of 4 + strings, one for each possible wave dimension. Each string + supports units of 0 to 3 bytes. Longer units can be stored using + the optional extended dimension units section of the file. + """ + + _size_field = "dimEUnitsSize" + _array_size_field = True + + +class DynamicLabelsField(DynamicStringField): + """Optional dimension label data + + From TN003: + + If the wave has dimension labels for dimension d then the + dimLabelsSize[d] field of the BinHeader5 structure will be + non-zero. + + A wave will have dimension labels if a SetDimLabel command has + been executed on it. + + A 3 point 1D wave has 4 dimension labels. The first dimension + label is the label for the dimension as a whole. The next three + dimension labels are the labels for rows 0, 1, and 2. When Igor + writes dimension labels to disk, it writes each dimension label as + a C string (null-terminated) in a field of 32 bytes. + """ + + _size_field = "dimLabelsSize" + _array_size_field = True + + def post_unpack(self, parents, data): + wave_structure = parents[-1] + wave_data = self._get_structure_data(parents, data, wave_structure) + bin_header = wave_data["bin_header"] + d = wave_data[self.name] + dim_labels = [] + start = 0 + for size in bin_header[self._size_field]: + end = start + size + if end > start: + dim_data = d[start:end] + chunks = [] + for i in range(size // 32): + chunks.append(dim_data[32 * i : 32 * (i + 1)]) + labels = [b""] + for chunk in chunks: + labels[-1] = labels[-1] + b"".join(chunk) + if b"\x00" in chunk: + labels.append(b"") + labels.pop(-1) + start = end + else: + labels = [] + dim_labels.append(labels) + wave_data[self.name] = dim_labels + + +class DynamicStringIndicesDataField(DynamicField): + """String indices used for text waves only""" + + def pre_pack(self, parents, data): + raise NotImplementedError() + + def pre_unpack(self, parents, data): + wave_structure = parents[-1] + wave_data = self._get_structure_data(parents, data, wave_structure) + bin_header = wave_data["bin_header"] + wave_header = wave_data["wave_header"] + self.string_indices_size = bin_header["sIndicesSize"] + self.count = self.string_indices_size // 4 + if self.count: # make sure we're in a text wave + assert TYPE_TABLE[wave_header["type"]] is None, wave_header + self.setup() + + def post_unpack(self, parents, data): + if not self.count: + return + wave_structure = parents[-1] + wave_data = self._get_structure_data(parents, data, wave_structure) + wave_header = wave_data["wave_header"] + wdata = wave_data["wData"] + strings = [] + start = 0 + for i, offset in enumerate(wave_data["sIndices"]): + if offset > start: + chars = wdata[start:offset] + strings.append(b"".join(chars)) + start = offset + elif offset == start: + strings.append(b"") + else: + raise ValueError((offset, wave_data["sIndices"])) + wdata = np.array(strings) + shape = [n for n in wave_header["nDim"] if n > 0] or (0,) + try: + wdata = wdata.reshape(shape) + except ValueError: + LOG.error(f"could not reshape strings from {shape} to {wdata.shape}") + raise + wave_data["wData"] = wdata + + +class DynamicVersionField(DynamicField): + def pre_pack(self, parents, byte_order): + raise NotImplementedError() + + def post_unpack(self, parents, data): + wave_structure = parents[-1] + wave_data = self._get_structure_data(parents, data, wave_structure) + version = wave_data["version"] + if wave_structure.byte_order in "@=": + need_to_reorder_bytes = _need_to_reorder_bytes(version) + wave_structure.byte_order = byte_order(need_to_reorder_bytes) + LOG.debug( + "get byte order from version: {} (reorder? {})".format( + wave_structure.byte_order, need_to_reorder_bytes + ) + ) + else: + need_to_reorder_bytes = False + + old_format = wave_structure.fields[-1].format + if version == 1: + wave_structure.fields[-1].format = Wave1 + elif version == 2: + wave_structure.fields[-1].format = Wave2 + elif version == 3: + wave_structure.fields[-1].format = Wave3 + elif version == 5: + wave_structure.fields[-1].format = Wave5 + elif not need_to_reorder_bytes: + raise ValueError(f"invalid binary wave version: {version}") + + if wave_structure.fields[-1].format != old_format: + LOG.debug( + "change wave headers from {} to {}".format( + old_format, wave_structure.fields[-1].format + ) + ) + wave_structure.setup() + elif need_to_reorder_bytes: + wave_structure.setup() + + # we might need to unpack again with the new byte order + return need_to_reorder_bytes + + +class DynamicWaveField(DynamicField): + def post_unpack(self, parents, data): + return + raise NotImplementedError() # TODO + checksum_size = bin.size + wave.size + parents[-1] + if version == 5: + # Version 5 checksum does not include the wData field. + checksum_size -= 4 + c = checksum(b, parents[-1].byte_order, 0, checksum_size) + if c != 0: + raise ValueError( + ( + "This does not appear to be a valid Igor binary wave file. " + "Error in checksum: should be 0, is {}." + ).format(c) + ) + + +Wave1 = DynamicStructure( + name="Wave1", + fields=[ + Field(BinHeader1, "bin_header", help="Binary wave header"), + Field(WaveHeader2, "wave_header", help="Wave header"), + DynamicWaveDataField1( + "f", + "wData", + help="The start of the array of waveform data.", + count=0, + array=True, + ), + ], +) + +Wave2 = DynamicStructure( + name="Wave2", + fields=[ + Field(BinHeader2, "bin_header", help="Binary wave header"), + Field(WaveHeader2, "wave_header", help="Wave header"), + DynamicWaveDataField1( + "f", + "wData", + help="The start of the array of waveform data.", + count=0, + array=True, + ), + Field( + "x", + "padding", + help="16 bytes of padding in versions 2 and 3.", + count=16, + array=True, + ), + DynamicWaveNoteField( + "c", "note", help="Optional wave note data", count=0, array=True + ), + ], +) + +Wave3 = DynamicStructure( + name="Wave3", + fields=[ + Field(BinHeader3, "bin_header", help="Binary wave header"), + Field(WaveHeader2, "wave_header", help="Wave header"), + DynamicWaveDataField1( + "f", + "wData", + help="The start of the array of waveform data.", + count=0, + array=True, + ), + Field( + "x", + "padding", + help="16 bytes of padding in versions 2 and 3.", + count=16, + array=True, + ), + DynamicWaveNoteField( + "c", "note", help="Optional wave note data", count=0, array=True + ), + DynamicDependencyFormulaField( + "c", "formula", help="Optional wave dependency formula", count=0, array=True + ), + ], +) + +Wave5 = DynamicStructure( + name="Wave5", + fields=[ + Field(BinHeader5, "bin_header", help="Binary wave header"), + Field(WaveHeader5, "wave_header", help="Wave header"), + DynamicWaveDataField5( + "f", + "wData", + help="The start of the array of waveform data.", + count=0, + array=True, + ), + DynamicDependencyFormulaField( + "c", + "formula", + help="Optional wave dependency formula.", + count=0, + array=True, + ), + DynamicWaveNoteField( + "c", "note", help="Optional wave note data.", count=0, array=True + ), + DynamicDataUnitsField( + "c", + "data_units", + help="Optional extended data units data.", + count=0, + array=True, + ), + DynamicDimensionUnitsField( + "c", + "dimension_units", + help="Optional dimension label data", + count=0, + array=True, + ), + DynamicLabelsField( + "c", "labels", help="Optional dimension label data", count=0, array=True + ), + DynamicStringIndicesDataField( + "P", + "sIndices", + help="Dynamic string indices for text waves.", + count=0, + array=True, + ), + ], +) + +Wave = DynamicStructure( + name="Wave", + fields=[ + DynamicVersionField( + "h", "version", help="Version number for backwards compatibility." + ), + DynamicWaveField(Wave1, "wave", help="The rest of the wave data."), + ], +) + + +def load(filename): + if hasattr(filename, "read"): + f = filename # filename is actually a stream object + else: + f = open(filename, "rb") + try: + Wave.byte_order = "=" + Wave.setup() + data = Wave.unpack_stream(f) + finally: + if not hasattr(filename, "read"): + f.close() + + return data + + +def save(filename): + raise NotImplementedError diff --git a/src/igor/binarywave.pyi b/src/igor/binarywave.pyi new file mode 100644 index 0000000..ccfade9 --- /dev/null +++ b/src/igor/binarywave.pyi @@ -0,0 +1,133 @@ +from __future__ import annotations +from typing import BinaryIO, NoReturn +from . import LOG as LOG +from .struct import DynamicField, DynamicStructure, Structure +from _typeshed import Incomplete +from pathlib import Path +from numpy.typing import DTypeLike +from ._typing import IBW, BYTEORDER + +complexInt8: DTypeLike +complexInt16: DTypeLike +complexInt32: DTypeLike +complexUInt8: DTypeLike +complexUInt16: DTypeLike +complexUInt32: DTypeLike + + +class StaticStringField(DynamicField): + def __init__( + self, *args: str | Structure | int, **kwargs: str | int | bool + ) -> None: + ... + + def post_unpack(self, parents: type, data: bytes) -> None: + ... + + +class NullStaticStringField(StaticStringField): + ... + + +TYPE_TABLE: dict[int, DTypeLike] +MAXDIMS: int +BinHeader1: Structure +BinHeader2: Structure +BinHeader3: Structure +BinHeader5: Structure +MAX_WAVE_NAME2: int +MAX_WAVE_NAME5: int +MAX_UNIT_CHARS: int +WaveHeader2: DynamicStructure +WaveHeader5: DynamicStructure + + +class DynamicWaveDataField1(DynamicField): + def pre_pack(self, parents: type, data: bytes) -> None: + ... + + count: Incomplete + data_size: Incomplete + shape: Incomplete + dtype: Incomplete + + def pre_unpack(self, parents: type, data: bytes) -> None: + ... + + def unpack(self, stream: BinaryIO): + ... + + +class DynamicWaveDataField5(DynamicWaveDataField1): + ... + + +class DynamicStringField(StaticStringField): + counts: Incomplete + count: Incomplete + + def pre_unpack(self, parents: type, data: bytes) -> None: + ... + + +class DynamicWaveNoteField(DynamicStringField): + ... + + +class DynamicDependencyFormulaField(DynamicStringField): + ... + + +class DynamicDataUnitsField(DynamicStringField): + ... + + +class DynamicDimensionUnitsField(DynamicStringField): + ... + + +class DynamicLabelsField(DynamicStringField): + def post_unpack(self, parents: type, data: bytes) -> None: + ... + + +class DynamicStringIndicesDataField(DynamicField): + def pre_pack(self, parents: type, data: bytes) -> None: + ... + + string_indices_size: Incomplete + count: Incomplete + + def pre_unpack(self, parents: type, data: bytes) -> None: + ... + + def post_unpack(self, parents: type, data: bytes) -> None: + ... + + +class DynamicVersionField(DynamicField): + def pre_pack(self, parents: type, byte_order: BYTEORDER) -> NoReturn: + ... + + def post_unpack(self, parents: type, data: bytes): + ... + + +class DynamicWaveField(DynamicField): + def post_unpack(self, parents: type, data: bytes) -> None: + ... + + +Wave1: DynamicStructure +Wave2: DynamicStructure +Wave3: DynamicStructure +Wave5: DynamicStructure +Wave: DynamicStructure + + +def load(filename: str | Path) -> IBW: + ... + + +def save(filename: str) -> None: + ... diff --git a/src/igor/igorpy.py b/src/igor/igorpy.py new file mode 100644 index 0000000..c64e4d6 --- /dev/null +++ b/src/igor/igorpy.py @@ -0,0 +1,347 @@ +# This program is in the public domain +"""`igor.py` compatibility layer on top of the `igor` package. + +igor.load('filename') or igor.loads('data') loads the content of an igor file +into memory as a folder structure. + +Returns the root folder. + +Folders have name, path and children. +Children can be indexed by folder[i] or by folder['name']. +To see the whole tree, use: print folder.format() + +The usual igor folder types are given in the technical reports +PTN003.ifn and TN003.ifn. +""" + +# Memo no related file in igor2 + + +import io +import locale +import re +import sys + +import numpy as np + +from .binarywave import MAXDIMS +from .packed import load as packed_load +from .record.base import UnknownRecord +from .record.folder import FolderStartRecord +from .record.folder import FolderEndRecord +from .record.history import HistoryRecord +from .record.history import GetHistoryRecord +from .record.history import RecreationRecord +from .record.packedfile import PackedFileRecord +from .record.procedure import ProcedureRecord +from .record.wave import WaveRecord +from .record.variables import VariablesRecord + + +__version__ = "0.3.2" + + +ENCODING = locale.getpreferredencoding() or sys.getdefaultencoding() +PYKEYWORDS = { + "and", + "as", + "assert", + "break", + "class", + "continue", + "def", + "elif", + "else", + "except", + "exec", + "finally", + "for", + "global", + "if", + "import", + "in", + "is", + "lambda", + "or", + "pass", + "print", + "raise", + "return", + "try", + "with", + "yield", +} +PYID = re.compile(r"^[^\d\W]\w*$", re.UNICODE) + + +def valid_identifier(s): + """Check if a name is a valid identifier""" + return PYID.match(s) and s not in PYKEYWORDS + + +class IgorObject: + """Parent class for all objects the parser can return""" + + pass + + +class Variables(IgorObject): + """ + Contains system numeric variables (e.g., K0) and user numeric and string variables. + """ + + def __init__(self, record): + self.sysvar = record.variables["variables"]["sysVars"] + self.uservar = record.variables["variables"]["userVars"] + self.userstr = record.variables["variables"]["userStrs"] + self.depvar = record.variables["variables"].get("dependentVars", {}) + self.depstr = record.variables["variables"].get("dependentStrs", {}) + + def format(self, indent=0): + return " " * indent + "" % ( + len(self.sysvar), + len(self.uservar) + len(self.userstr), + len(self.depvar) + len(self.depstr), + ) + + +class History(IgorObject): + """ + Contains the experiment's history as plain text. + """ + + def __init__(self, data): + self.data = data + + def format(self, indent=0): + return " " * indent + "" + + +class Wave(IgorObject): + """ + Contains the data for a wave + """ + + def __init__(self, record): + d = record.wave["wave"] + self.name = d["wave_header"]["bname"].decode(ENCODING) + self.data = d["wData"] + self.fs = d["wave_header"]["fsValid"] + self.fstop = d["wave_header"]["topFullScale"] + self.fsbottom = d["wave_header"]["botFullScale"] + version = record.wave["version"] + if version in [1, 2, 3]: + dims = [d["wave_header"]["npnts"]] + [0] * (MAXDIMS - 1) + sfA = [d["wave_header"]["hsA"]] + [0] * (MAXDIMS - 1) + sfB = [d["wave_header"]["hsB"]] + [0] * (MAXDIMS - 1) + self.data_units = [d["wave_header"]["dataUnits"]] + self.axis_units = [d["wave_header"]["xUnits"]] + else: + dims = d["wave_header"]["nDim"] + sfA = d["wave_header"]["sfA"] + sfB = d["wave_header"]["sfB"] + # TODO find example with multiple data units + if version == 5: + self.data_units = [d["data_units"].decode(ENCODING)] + self.axis_units = [ + b"".join(d).decode(ENCODING) for d in d["wave_header"]["dimUnits"] + ] + else: + self.data_units = [d["data_units"].decode(ENCODING)] + self.axis_units = [d["dimension_units"].decode(ENCODING)] + + self.data_units.extend([""] * (MAXDIMS - len(self.data_units))) + self.data_units = tuple(self.data_units) + self.axis_units.extend([""] * (MAXDIMS - len(self.axis_units))) + self.axis_units = tuple(self.axis_units) + self.axis = [ + np.linspace(b, b + a * (c - 1), c) for a, b, c in zip(sfA, sfB, dims) + ] + self.formula = d.get("formula", "") + self.notes = d.get("note", "") + + def format(self, indent=0): + if isinstance(self.data, list): + type, size = "text", "%d" % len(self.data) + else: + type, size = "data", "x".join(str(d) for d in self.data.shape) + return " " * indent + "{} {} ({})".format(self.name, type, size) + + def __array__(self): + return self.data + + __repr__ = __str__ = lambda s: "" % s.format() + + +class Recreation(IgorObject): + """ + Contains the experiment's recreation procedures as plain text. + """ + + def __init__(self, data): + self.data = data + + def format(self, indent=0): + return " " * indent + "" + + +class Procedure(IgorObject): + """ + Contains the experiment's main procedure window text as plain text. + """ + + def __init__(self, data): + self.data = data + + def format(self, indent=0): + return " " * indent + "" + + +class GetHistory(IgorObject): + """ + Not a real record but rather, a message to go back and read the history text. + + The reason for GetHistory is that IGOR runs Recreation when it loads the + datafile. This puts entries in the history that shouldn't be there. The + GetHistory entry simply says that the Recreation has run, and the History + can be restored from the previously saved value. + """ + + def __init__(self, data): + self.data = data + + def format(self, indent=0): + return " " * indent + "" + + +class PackedFile(IgorObject): + """ + Contains the data for a procedure file or notebook in packed form. + """ + + def __init__(self, data): + self.data = data + + def format(self, indent=0): + return " " * indent + "" + + +class Unknown(IgorObject): + """ + Record type not documented in PTN003/TN003. + """ + + def __init__(self, data, type): + self.data = data + self.type = type + + def format(self, indent=0): + return " " * indent + "" % self.type + + +class Folder(IgorObject): + """ + Hierarchical record container. + """ + + def __init__(self, path): + self.name = path[-1] + self.path = path + self.children = [] + + def __getitem__(self, key): + if isinstance(key, int): + return self.children[key] + else: + for r in self.children: + if isinstance(r, (Folder, Wave)) and r.name == key: + return r + raise KeyError("Folder %s does not exist" % key) + + def __str__(self): + return "" % "/".join(self.path) + + __repr__ = __str__ + + def append(self, record): + """ + Add a record to the folder. + """ + self.children.append(record) + try: + # Record may not have a name, the name may be invalid, or it + # may already be in use. The noname case will be covered by + # record.name raising an attribute error. The others we need + # to test for explicitly. + if valid_identifier(record.name) and not hasattr(self, record.name): + setattr(self, record.name, record) + except AttributeError: + pass + + def format(self, indent=0): + parent = " " * indent + self.name + children = [r.format(indent=indent + 2) for r in self.children] + return "\n".join([parent] + children) + + +def loads(s, **kwargs): + """Load an igor file from string""" + stream = io.BytesIO(s) + return load(stream, **kwargs) + + +def load(filename, **kwargs): + """Load an igor file""" + try: + packed_experiment = packed_load( + filename, initial_byte_order=kwargs.pop("initial_byte_order", "=") + ) + except ValueError as e: + if e.args[0].startswith("not enough data for the next record header"): + raise OSError("invalid record header; bad pxp file?") + elif e.args[0].startswith("not enough data for the next record"): + raise OSError("final record too long; bad pxp file?") + raise + return _convert(packed_experiment, **kwargs) + + +def _convert(packed_experiment, ignore_unknown=True): + records, filesystem = packed_experiment + stack = [Folder(path=["root"])] + for record in records: + if isinstance(record, UnknownRecord): + if ignore_unknown: + continue + else: + r = Unknown(record.data, type=record.header["recordType"]) + elif isinstance(record, GetHistoryRecord): + r = GetHistory(record.text) + elif isinstance(record, HistoryRecord): + r = History(record.text) + elif isinstance(record, PackedFileRecord): + r = PackedFile(record.text) + elif isinstance(record, ProcedureRecord): + r = Procedure(record.text) + elif isinstance(record, RecreationRecord): + r = Recreation(record.text) + elif isinstance(record, VariablesRecord): + r = Variables(record) + elif isinstance(record, WaveRecord): + r = Wave(record) + else: + r = None + + if isinstance(record, FolderStartRecord): + path = stack[-1].path + [record.null_terminated_text.decode(ENCODING)] + folder = Folder(path) + stack[-1].append(folder) + stack.append(folder) + elif isinstance(record, FolderEndRecord): + stack.pop() + elif r is None: + raise NotImplementedError(record) + else: + stack[-1].append(r) + if len(stack) != 1: + raise OSError("FolderStart records do not match FolderEnd records") + return stack[0] diff --git a/src/igor/igorpy.pyi b/src/igor/igorpy.pyi new file mode 100644 index 0000000..61dd769 --- /dev/null +++ b/src/igor/igorpy.pyi @@ -0,0 +1,145 @@ +from __future__ import annotations +import re + +from _typeshed import Incomplete + +from .binarywave import MAXDIMS +from .packed import load +from .record.base import UnknownRecord +from .record.folder import FolderEndRecord, FolderStartRecord +from .record.history import GetHistoryRecord, HistoryRecord, RecreationRecord +from .record.packedfile import PackedFileRecord +from .record.procedure import ProcedureRecord +from .record.variables import VariablesRecord +from .record.wave import WaveRecord + +ENCODING: str +PYKEYWORDS: set[str] +PYID: re.Pattern + + +def valid_identifier(s): + ... + + +class IgorObject: + ... + + +class Variables(IgorObject): + sysvar: Incomplete + uservar: Incomplete + userstr: Incomplete + depvar: Incomplete + depstr: Incomplete + + def __init__(self, record) -> None: + ... + + def format(self, indent: int = ...): + ... + + +class History(IgorObject): + data: Incomplete + + def __init__(self, data) -> None: + ... + + def format(self, indent: int = ...): + ... + + +class Wave(IgorObject): + name: Incomplete + data: Incomplete + fs: Incomplete + fstop: Incomplete + fsbottom: Incomplete + data_units: Incomplete + axis_units: Incomplete + axis: Incomplete + formula: Incomplete + notes: Incomplete + + def __init__(self, record) -> None: + ... + + def format(self, indent: int = ...): + ... + + def __array__(self): + ... + + +class Recreation(IgorObject): + data: Incomplete + + def __init__(self, data) -> None: + ... + + def format(self, indent: int = ...): + ... + + +class Procedure(IgorObject): + data: Incomplete + + def __init__(self, data) -> None: + ... + + def format(self, indent: int = ...): + ... + + +class GetHistory(IgorObject): + data: Incomplete + + def __init__(self, data) -> None: + ... + + def format(self, indent: int = ...): + ... + + +class PackedFile(IgorObject): + data: Incomplete + + def __init__(self, data) -> None: + ... + + def format(self, indent: int = ...): + ... + + +class Unknown(IgorObject): + data: Incomplete + type: Incomplete + + def __init__(self, data, type) -> None: + ... + + def format(self, indent: int = ...): + ... + + +class Folder(IgorObject): + name: Incomplete + path: Incomplete + children: Incomplete + + def __init__(self, path) -> None: + ... + + def __getitem__(self, key): + ... + + def append(self, record) -> None: + ... + + def format(self, indent: int = ...): + ... + + +def loads(s, **kwargs): + ... diff --git a/igor/packed.py b/src/igor/packed.py similarity index 61% rename from igor/packed.py rename to src/igor/packed.py index 2035410..0606086 100644 --- a/igor/packed.py +++ b/src/igor/packed.py @@ -15,21 +15,24 @@ # You should have received a copy of the GNU Lesser General Public License # along with igor. If not, see . +# Memo essentially no diff with igor2 + + "Read IGOR Packed Experiment files files into records." -from . import LOG as _LOG -from .struct import Structure as _Structure -from .struct import Field as _Field +from . import LOG +from .struct import Structure +from .struct import Field from .util import byte_order as _byte_order from .util import need_to_reorder_bytes as _need_to_reorder_bytes from .util import _bytes -from .record import RECORD_TYPE as _RECORD_TYPE -from .record.base import UnknownRecord as _UnknownRecord -from .record.base import UnusedRecord as _UnusedRecord -from .record.folder import FolderStartRecord as _FolderStartRecord -from .record.folder import FolderEndRecord as _FolderEndRecord -from .record.variables import VariablesRecord as _VariablesRecord -from .record.wave import WaveRecord as _WaveRecord +from .record import RECORD_TYPE +from .record.base import UnknownRecord +from .record.base import UnusedRecord +from .record.folder import FolderStartRecord +from .record.folder import FolderEndRecord +from .record.variables import VariablesRecord +from .record.wave import WaveRecord # From PTN003: @@ -40,32 +43,39 @@ # files, you must skip any record with a record type that is not # listed above. -PackedFileRecordHeader = _Structure( - name='PackedFileRecordHeader', +PackedFileRecordHeader = Structure( + name="PackedFileRecordHeader", fields=[ - _Field('H', 'recordType', help='Record type plus superceded flag.'), - _Field('h', 'version', help='Version information depends on the type of record.'), - _Field('l', 'numDataBytes', help='Number of data bytes in the record following this record header.'), - ]) - -#CR_STR = '\x15' (\r) + Field("H", "recordType", help="Record type plus superseded flag."), + Field( + "h", "version", help="Version information depends on the type of record." + ), + Field( + "l", + "numDataBytes", + help="Number of data bytes in the record following this record header.", + ), + ], +) + +# CR_STR = '\x15' (\r) PACKEDRECTYPE_MASK = 0x7FFF # Record type = (recordType & PACKEDREC_TYPE_MASK) -SUPERCEDED_MASK = 0x8000 # Bit is set if the record is superceded by - # a later record in the packed file. +SUPERCEDED_MASK = 0x8000 # Bit is set if the record is superseded by +# a later record in the packed file. -def load(filename, strict=True, ignore_unknown=True, initial_byte_order='='): +def load(filename, *, strict=True, ignore_unknown=True, initial_byte_order="="): """ Probably better to actually infer the initial_byte_order, as can be done from the header. For now though we will let the user deal with this. """ - _LOG.debug('loading a packed experiment file from {}'.format(filename)) + LOG.debug(f"loading a packed experiment file from {filename}") records = [] - if hasattr(filename, 'read'): + if hasattr(filename, "read"): f = filename # filename is actually a stream object else: - f = open(filename, 'rb') + f = open(filename, "rb") byte_order = None try: while True: @@ -76,46 +86,53 @@ def load(filename, strict=True, ignore_unknown=True, initial_byte_order='='): break if len(b) < PackedFileRecordHeader.size: raise ValueError( - ('not enough data for the next record header ({} < {})' - ).format(len(b), PackedFileRecordHeader.size)) - _LOG.debug('reading a new packed experiment file record') + ("not enough data for the next record header ({} < {})").format( + len(b), PackedFileRecordHeader.size + ) + ) + LOG.debug("reading a new packed experiment file record") header = PackedFileRecordHeader.unpack_from(b) - if header['version'] and not byte_order: - need_to_reorder = _need_to_reorder_bytes(header['version']) + if header["version"] and not byte_order: + need_to_reorder = _need_to_reorder_bytes(header["version"]) byte_order = initial_byte_order = _byte_order(need_to_reorder) - _LOG.debug( - 'get byte order from version: {} (reorder? {})'.format( - byte_order, need_to_reorder)) + LOG.debug( + "get byte order from version: {} (reorder? {})".format( + byte_order, need_to_reorder + ) + ) if need_to_reorder: PackedFileRecordHeader.byte_order = byte_order PackedFileRecordHeader.setup() header = PackedFileRecordHeader.unpack_from(b) - _LOG.debug( - 'reordered version: {}'.format(header['version'])) - data = bytes(f.read(header['numDataBytes'])) - if len(data) < header['numDataBytes']: + LOG.debug("reordered version: {}".format(header["version"])) + data = bytes(f.read(header["numDataBytes"])) + if len(data) < header["numDataBytes"]: raise ValueError( - ('not enough data for the next record ({} < {})' - ).format(len(b), header['numDataBytes'])) - record_type = _RECORD_TYPE.get( - header['recordType'] & PACKEDRECTYPE_MASK, _UnknownRecord) - _LOG.debug('the new record has type {} ({}).'.format( - record_type, header['recordType'])) - if record_type in [_UnknownRecord, _UnusedRecord - ] and not ignore_unknown: - raise KeyError('unkown record type {}'.format( - header['recordType'])) + ("not enough data for the next record ({} < {})").format( + len(b), header["numDataBytes"] + ) + ) + record_type = RECORD_TYPE.get( + header["recordType"] & PACKEDRECTYPE_MASK, UnknownRecord + ) + LOG.debug( + "the new record has type {} ({}).".format( + record_type, header["recordType"] + ) + ) + if record_type in [UnknownRecord, UnusedRecord] and not ignore_unknown: + raise KeyError("unkwon record type {}".format(header["recordType"])) records.append(record_type(header, data, byte_order=byte_order)) finally: - _LOG.debug('finished loading {} records from {}'.format( - len(records), filename)) - if not hasattr(filename, 'read'): + LOG.debug(f"finished loading {len(records)} records from {filename}") + if not hasattr(filename, "read"): f.close() filesystem = _build_filesystem(records) return (records, filesystem) + def _build_filesystem(records): # From PTN003: """The name must be a valid Igor data folder name. See Object @@ -128,7 +145,7 @@ def _build_filesystem(records): folder end record.""" # From the Igor Manual, chapter 2, section 8, page II-123 # http://www.wavemetrics.net/doc/igorman/II-08%20Data%20Folders.pdf - """Like the Macintosh file system, Igor Pro's data folders use the + r"""Like the Macintosh file system, Igor Pro's data folders use the colon character (:) to separate components of a path to an object. This is analogous to Unix which uses / and Windows which uses \. (Reminder: Igor's data folders exist wholly in memory @@ -151,20 +168,20 @@ def _build_filesystem(records): " ' : ; """ - filesystem = {'root': {}} - dir_stack = [('root', filesystem['root'])] + filesystem = {"root": {}} + dir_stack = [("root", filesystem["root"])] for record in records: cwd = dir_stack[-1][-1] - if isinstance(record, _FolderStartRecord): + if isinstance(record, FolderStartRecord): name = record.null_terminated_text cwd[name] = {} dir_stack.append((name, cwd[name])) - elif isinstance(record, _FolderEndRecord): + elif isinstance(record, FolderEndRecord): dir_stack.pop() - elif isinstance(record, (_VariablesRecord, _WaveRecord)): - if isinstance(record, _VariablesRecord): - sys_vars = record.variables['variables']['sysVars'].keys() - for filename,value in record.namespace.items(): + elif isinstance(record, (VariablesRecord, WaveRecord)): + if isinstance(record, VariablesRecord): + sys_vars = record.variables["variables"]["sysVars"].keys() + for filename, value in record.namespace.items(): if len(dir_stack) > 1 and filename in sys_vars: # From PTN003: """When reading a packed file, any system @@ -175,23 +192,27 @@ def _build_filesystem(records): _check_filename(dir_stack, filename) cwd[filename] = value else: # WaveRecord - filename = record.wave['wave']['wave_header']['bname'] + filename = record.wave["wave"]["wave_header"]["bname"] _check_filename(dir_stack, filename) cwd[filename] = record return filesystem + def _check_filename(dir_stack, filename): cwd = dir_stack[-1][-1] if filename in cwd: - raise ValueError('collision on name {} in {}'.format( - filename, ':'.join(d for d,cwd in dir_stack))) + raise ValueError( + "collision on name {} in {}".format( + filename, ":".join(d for d, cwd in dir_stack) + ) + ) + def walk(filesystem, callback, dirpath=None): - """Walk a packed experiment filesystem, operating on each key,value pair. - """ + """Walk a packed experiment filesystem, operating on each key,value pair.""" if dirpath is None: dirpath = [] - for key,value in sorted((_bytes(k),v) for k,v in filesystem.items()): + for key, value in sorted((_bytes(k), v) for k, v in filesystem.items()): callback(dirpath, key, value) if isinstance(value, dict): - walk(filesystem=value, callback=callback, dirpath=dirpath+[key]) + walk(filesystem=value, callback=callback, dirpath=dirpath + [key]) diff --git a/src/igor/packed.pyi b/src/igor/packed.pyi new file mode 100644 index 0000000..f95f4f3 --- /dev/null +++ b/src/igor/packed.pyi @@ -0,0 +1,36 @@ +from __future__ import annotations +from . import LOG as LOG +from .record import RECORD_TYPE as RECORD_TYPE +from .record.base import UnknownRecord as UnknownRecord, UnusedRecord as UnusedRecord +from .record.folder import ( + FolderEndRecord as FolderEndRecord, + FolderStartRecord as FolderStartRecord, +) +from .record.variables import VariablesRecord as VariablesRecord +from .record.wave import WaveRecord as WaveRecord +from .struct import Field as Field, Structure as Structure +from _typeshed import Incomplete +from pathlib import Path +from typing import Callable +from typing import Any +from ._typing import RECORDS, BYTEORDER + +PackedFileRecordHeader: Incomplete +PACKEDRECTYPE_MASK: int +SUPERCEDED_MASK: int + +IGORDATAFOLDER = dict[ + str | bytes, WaveRecord | float | dict[str | bytes, IGORDATAFOLDER] +] + +def load( + filename: str | Path, + strict: bool = ..., + ignore_unknown: bool = ..., + initial_byte_order: BYTEORDER = ..., +) -> tuple[RECORDS, dict[str, IGORDATAFOLDER]]: ... +def walk( + filesystem: dict[str, IGORDATAFOLDER], + callback: Callable[[Any], Any], + dirpath: list[str | Path] | None = ..., +) -> None: ... diff --git a/igor/record/__init__.py b/src/igor/record/__init__.py similarity index 95% rename from igor/record/__init__.py rename to src/igor/record/__init__.py index eafebfb..e0b41d5 100644 --- a/igor/record/__init__.py +++ b/src/igor/record/__init__.py @@ -18,7 +18,7 @@ "Record parsers for IGOR's packed experiment files." -from .base import Record, UnknownRecord, UnusedRecord +from .base import UnusedRecord from .variables import VariablesRecord from .history import HistoryRecord, RecreationRecord, GetHistoryRecord from .wave import WaveRecord @@ -27,6 +27,8 @@ from .folder import FolderStartRecord, FolderEndRecord +# Memo No diff with igor2 + # From PackedFile.h RECORD_TYPE = { 0: UnusedRecord, @@ -40,4 +42,4 @@ 8: PackedFileRecord, 9: FolderStartRecord, 10: FolderEndRecord, - } +} diff --git a/src/igor/record/__init__.pyi b/src/igor/record/__init__.pyi new file mode 100644 index 0000000..2bae41f --- /dev/null +++ b/src/igor/record/__init__.pyi @@ -0,0 +1,3 @@ +from __future__ import annotations + +RECORD_TYPE: dict[int, type] diff --git a/igor/record/base.py b/src/igor/record/base.py similarity index 67% rename from igor/record/base.py rename to src/igor/record/base.py index 6b168cf..e875fc7 100644 --- a/igor/record/base.py +++ b/src/igor/record/base.py @@ -16,7 +16,10 @@ # along with igor. If not, see . -class Record (object): +# Memo No diff with igor2 + + +class Record: def __init__(self, header, data, byte_order=None): self.header = header self.data = data @@ -26,22 +29,22 @@ def __str__(self): return self.__repr__() def __repr__(self): - return '<{} {}>'.format(self.__class__.__name__, id(self)) + return f"<{self.__class__.__name__} {id(self)}>" -class UnknownRecord (Record): +class UnknownRecord(Record): def __repr__(self): - return '<{}-{} {}>'.format( - self.__class__.__name__, self.header['recordType'], id(self)) + return "<{}-{} {}>".format( + self.__class__.__name__, self.header["recordType"], id(self) + ) -class UnusedRecord (Record): +class UnusedRecord(Record): pass -class TextRecord (Record): +class TextRecord(Record): def __init__(self, *args, **kwargs): - super(TextRecord, self).__init__(*args, **kwargs) - self.text = bytes(self.data).replace( - b'\r\n', b'\n').replace(b'\r', b'\n') - self.null_terminated_text = self.text.split(b'\x00', 1)[0] + super().__init__(*args, **kwargs) + self.text = bytes(self.data).replace(b"\r\n", b"\n").replace(b"\r", b"\n") + self.null_terminated_text = self.text.split(b"\x00", 1)[0] diff --git a/src/igor/record/base.pyi b/src/igor/record/base.pyi new file mode 100644 index 0000000..6ecb374 --- /dev/null +++ b/src/igor/record/base.pyi @@ -0,0 +1,28 @@ +from __future__ import annotations + +from .._typing import BYTEORDER + +class Record: + header: dict[str, int] + data: bytes + byte_order: BYTEORDER | None + + def __init__( + self, + header: dict[str, int], + data: bytes, + byte_order: BYTEORDER = ..., + ) -> None: ... + +class UnknownRecord(Record): ... +class UnusedRecord(Record): ... + +class TextRecord(Record): + text: bytes + null_terminated_text: str + + def __init__( + self, + *args: dict[str, int] | bytes | BYTEORDER, + **kwargs: dict[str, int] | bytes | BYTEORDER, + ) -> None: ... diff --git a/igor/record/folder.py b/src/igor/record/folder.py similarity index 88% rename from igor/record/folder.py rename to src/igor/record/folder.py index caaeb54..7bf6d74 100644 --- a/igor/record/folder.py +++ b/src/igor/record/folder.py @@ -15,12 +15,14 @@ # You should have received a copy of the GNU Lesser General Public License # along with igor. If not, see . +# Memo No diff with igor2 + from .base import TextRecord -class FolderStartRecord (TextRecord): +class FolderStartRecord(TextRecord): pass -class FolderEndRecord (TextRecord): +class FolderEndRecord(TextRecord): pass diff --git a/src/igor/record/folder.pyi b/src/igor/record/folder.pyi new file mode 100644 index 0000000..094ce77 --- /dev/null +++ b/src/igor/record/folder.pyi @@ -0,0 +1,5 @@ +from __future__ import annotations +from .base import TextRecord + +class FolderStartRecord(TextRecord): ... +class FolderEndRecord(TextRecord): ... diff --git a/igor/record/history.py b/src/igor/record/history.py similarity index 85% rename from igor/record/history.py rename to src/igor/record/history.py index e5d2199..48d98d9 100644 --- a/igor/record/history.py +++ b/src/igor/record/history.py @@ -15,16 +15,18 @@ # You should have received a copy of the GNU Lesser General Public License # along with igor. If not, see . +# Memo No diff with igor2 + from .base import TextRecord -class HistoryRecord (TextRecord): +class HistoryRecord(TextRecord): pass -class RecreationRecord (TextRecord): +class RecreationRecord(TextRecord): pass -class GetHistoryRecord (TextRecord): +class GetHistoryRecord(TextRecord): pass diff --git a/src/igor/record/history.pyi b/src/igor/record/history.pyi new file mode 100644 index 0000000..6c55048 --- /dev/null +++ b/src/igor/record/history.pyi @@ -0,0 +1,6 @@ +from __future__ import annotations +from .base import TextRecord as TextRecord + +class HistoryRecord(TextRecord): ... +class RecreationRecord(TextRecord): ... +class GetHistoryRecord(TextRecord): ... diff --git a/igor/record/packedfile.py b/src/igor/record/packedfile.py similarity index 92% rename from igor/record/packedfile.py rename to src/igor/record/packedfile.py index b457f20..4f6b5c7 100644 --- a/igor/record/packedfile.py +++ b/src/igor/record/packedfile.py @@ -15,8 +15,10 @@ # You should have received a copy of the GNU Lesser General Public License # along with igor. If not, see . +# Memo No diff with igor2 + from .base import Record -class PackedFileRecord (Record): +class PackedFileRecord(Record): pass diff --git a/src/igor/record/packedfile.pyi b/src/igor/record/packedfile.pyi new file mode 100644 index 0000000..0607e56 --- /dev/null +++ b/src/igor/record/packedfile.pyi @@ -0,0 +1,5 @@ +from __future__ import annotations + +from .base import Record as Record + +class PackedFileRecord(Record): ... diff --git a/igor/record/procedure.py b/src/igor/record/procedure.py similarity index 92% rename from igor/record/procedure.py rename to src/igor/record/procedure.py index de00e6e..e70c00e 100644 --- a/igor/record/procedure.py +++ b/src/igor/record/procedure.py @@ -15,8 +15,10 @@ # You should have received a copy of the GNU Lesser General Public License # along with igor. If not, see . +# Memo No diff with igor2 + from .base import TextRecord -class ProcedureRecord (TextRecord): +class ProcedureRecord(TextRecord): pass diff --git a/src/igor/record/procedure.pyi b/src/igor/record/procedure.pyi new file mode 100644 index 0000000..cd07cd3 --- /dev/null +++ b/src/igor/record/procedure.pyi @@ -0,0 +1,4 @@ +from __future__ import annotations +from .base import TextRecord + +class ProcedureRecord(TextRecord): ... diff --git a/src/igor/record/variables.py b/src/igor/record/variables.py new file mode 100644 index 0000000..015763a --- /dev/null +++ b/src/igor/record/variables.py @@ -0,0 +1,375 @@ +# Copyright (C) 2012 W. Trevor King +# +# This file is part of igor. +# +# igor is free software: you can redistribute it and/or modify it under the +# terms of the GNU Lesser General Public License as published by the Free +# Software Foundation, either version 3 of the License, or (at your option) any +# later version. +# +# igor is distributed in the hope that it will be useful, but WITHOUT ANY +# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR +# A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more +# details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with igor. If not, see . + +# Memo essentially No diff with igor2 + +import io + +from .. import LOG +from ..binarywave import TYPE_TABLE +from ..binarywave import NullStaticStringField +from ..binarywave import DynamicStringField +from ..struct import Structure +from ..struct import DynamicStructure +from ..struct import Field +from ..struct import DynamicField +from ..util import byte_order as _byte_order +from ..util import need_to_reorder_bytes as _need_to_reorder_bytes +from .base import Record + + +class ListedStaticStringField(NullStaticStringField): + """Handle string conversions for multi-count dynamic parents. + + If a field belongs to a multi-count dynamic parent, the parent is + called multiple times to parse each count, and the field's + post-unpack hook gets called after the field is unpacked during + each iteration. This requires alternative logic for getting and + setting the string data. The actual string formatting code is not + affected. + """ + + def post_unpack(self, parents, data): + parent_structure = parents[-1] + parent_data = self._get_structure_data(parents, data, parent_structure) + d = self._normalize_string(parent_data[-1][self.name]) + parent_data[-1][self.name] = d + + +class ListedDynamicStrDataField(DynamicStringField, ListedStaticStringField): + _size_field = "strLen" + _null_terminated = False + + def _get_size_data(self, parents, data): + parent_structure = parents[-1] + parent_data = self._get_structure_data(parents, data, parent_structure) + return parent_data[-1][self._size_field] + + +class DynamicVarDataField(DynamicField): + def __init__(self, *args, **kwargs): + if "array" not in kwargs: + kwargs["array"] = True + super().__init__(*args, **kwargs) + + def pre_pack(self, parents, data): + raise NotImplementedError() + + def post_unpack(self, parents, data): + var_structure = parents[-1] + var_data = self._get_structure_data(parents, data, var_structure) + data = var_data[self.name] + d = {} + for i, value in enumerate(data): + key, value = self._normalize_item(i, value) + d[key] = value + var_data[self.name] = d + + def _normalize_item(self, index, value): + raise NotImplementedError() + + +class DynamicSysVarField(DynamicVarDataField): + def _normalize_item(self, index, value): + name = f"K{index}" + return (name, value) + + +class DynamicUserVarField(DynamicVarDataField): + def _normalize_item(self, index, value): + name = value["name"] + value = value["num"] + return (name, value) + + +class DynamicUserStrField(DynamicVarDataField): + def _normalize_item(self, index, value): + name = value["name"] + value = value["data"] + return (name, value) + + +class DynamicVarNumField(DynamicField): + def post_unpack(self, parents, data): + parent_structure = parents[-1] + parent_data = self._get_structure_data(parents, data, parent_structure) + d = self._normalize_numeric_variable(parent_data[-1][self.name]) + parent_data[-1][self.name] = d + + def _normalize_numeric_variable(self, num_var): + t = TYPE_TABLE[num_var["numType"]] + if num_var["numType"] % 2: # complex number + return t(complex(num_var["realPart"], num_var["imagPart"])) + else: + return t(num_var["realPart"]) + + +class DynamicFormulaField(DynamicStringField): + _size_field = "formulaLen" + _null_terminated = True + + +# From Variables.h +VarHeader1 = Structure( # `version` field pulled out into VariablesRecord + name="VarHeader1", + fields=[ + Field("h", "numSysVars", help="Number of system variables (K0, K1, ...)."), + Field( + "h", "numUserVars", help="Number of user numeric variables -- may be zero." + ), + Field( + "h", "numUserStrs", help="Number of user string variables -- may be zero." + ), + ], +) + +# From Variables.h +VarHeader2 = Structure( # `version` field pulled out into VariablesRecord + name="VarHeader2", + fields=[ + Field("h", "numSysVars", help="Number of system variables (K0, K1, ...)."), + Field( + "h", "numUserVars", help="Number of user numeric variables -- may be zero." + ), + Field( + "h", "numUserStrs", help="Number of user string variables -- may be zero." + ), + Field( + "h", + "numDependentVars", + help="Number of dependent numeric variables -- may be zero.", + ), + Field( + "h", + "numDependentStrs", + help="Number of dependent string variables -- may be zero.", + ), + ], +) + +# From Variables.h +UserStrVarRec1 = DynamicStructure( + name="UserStrVarRec1", + fields=[ + ListedStaticStringField( + "c", "name", help="Name of the string variable.", count=32 + ), + Field("h", "strLen", help="The real size of the following array."), + ListedDynamicStrDataField("c", "data"), + ], +) + +# From Variables.h +UserStrVarRec2 = DynamicStructure( + name="UserStrVarRec2", + fields=[ + ListedStaticStringField( + "c", "name", help="Name of the string variable.", count=32 + ), + Field("l", "strLen", help="The real size of the following array."), + Field("c", "data"), + ], +) + +# From Variables.h +VarNumRec = Structure( + name="VarNumRec", + fields=[ + Field("h", "numType", help="Type from binarywave.TYPE_TABLE"), + Field("d", "realPart", help="The real part of the number."), + Field("d", "imagPart", help="The imag part if the number is complex."), + Field("l", "reserved", help="Reserved - set to zero."), + ], +) + +# From Variables.h +UserNumVarRec = DynamicStructure( + name="UserNumVarRec", + fields=[ + ListedStaticStringField( + "c", "name", help="Name of the string variable.", count=32 + ), + Field("h", "type", help="0 = string, 1 = numeric."), + DynamicVarNumField( + VarNumRec, + "num", + help="Type and value of the variable if it is numeric. Not used for string.", + ), + ], +) + +# From Variables.h +UserDependentVarRec = DynamicStructure( + name="UserDependentVarRec", + fields=[ + ListedStaticStringField( + "c", "name", help="Name of the string variable.", count=32 + ), + Field("h", "type", help="0 = string, 1 = numeric."), + Field( + VarNumRec, + "num", + help="Type and value of the variable if it is numeric. Not used for string.", + ), + Field("h", "formulaLen", help="The length of the dependency formula."), + DynamicFormulaField( + "c", + "formula", + help="Start of the dependency formula. A C string including null terminator.", + ), + ], +) + + +class DynamicVarHeaderField(DynamicField): + def pre_pack(self, parents, data): + raise NotImplementedError() + + def post_unpack(self, parents, data): + var_structure = parents[-1] + var_data = self._get_structure_data(parents, data, var_structure) + data = var_data["var_header"] + sys_vars_field = var_structure.get_field("sysVars") + sys_vars_field.count = data["numSysVars"] + sys_vars_field.setup() + user_vars_field = var_structure.get_field("userVars") + user_vars_field.count = data["numUserVars"] + user_vars_field.setup() + user_strs_field = var_structure.get_field("userStrs") + user_strs_field.count = data["numUserStrs"] + user_strs_field.setup() + if "numDependentVars" in data: + dependent_vars_field = var_structure.get_field("dependentVars") + dependent_vars_field.count = data["numDependentVars"] + dependent_vars_field.setup() + dependent_strs_field = var_structure.get_field("dependentStrs") + dependent_strs_field.count = data["numDependentStrs"] + dependent_strs_field.setup() + var_structure.setup() + + +Variables1 = DynamicStructure( + name="Variables1", + fields=[ + DynamicVarHeaderField(VarHeader1, "var_header", help="Variables header"), + DynamicSysVarField("f", "sysVars", help="System variables", count=0), + DynamicUserVarField( + UserNumVarRec, "userVars", help="User numeric variables", count=0 + ), + DynamicUserStrField( + UserStrVarRec1, "userStrs", help="User string variables", count=0 + ), + ], +) + + +Variables2 = DynamicStructure( + name="Variables2", + fields=[ + DynamicVarHeaderField(VarHeader2, "var_header", help="Variables header"), + DynamicSysVarField("f", "sysVars", help="System variables", count=0), + DynamicUserVarField( + UserNumVarRec, "userVars", help="User numeric variables", count=0 + ), + DynamicUserStrField( + UserStrVarRec2, "userStrs", help="User string variables", count=0 + ), + Field( + UserDependentVarRec, + "dependentVars", + help="Dependent numeric variables.", + count=0, + array=True, + ), + Field( + UserDependentVarRec, + "dependentStrs", + help="Dependent string variables.", + count=0, + array=True, + ), + ], +) + + +class DynamicVersionField(DynamicField): + def pre_pack(self, parents, byte_order): + raise NotImplementedError() + + def post_unpack(self, parents, data): + variables_structure = parents[-1] + variables_data = self._get_structure_data(parents, data, variables_structure) + version = variables_data["version"] + if variables_structure.byte_order in "@=": + need_to_reorder_bytes = _need_to_reorder_bytes(version) + variables_structure.byte_order = _byte_order(need_to_reorder_bytes) + LOG.debug( + "get byte order from version: {} (reorder? {})".format( + variables_structure.byte_order, need_to_reorder_bytes + ) + ) + else: + need_to_reorder_bytes = False + + old_format = variables_structure.fields[-1].format + if version == 1: + variables_structure.fields[-1].format = Variables1 + elif version == 2: + variables_structure.fields[-1].format = Variables2 + elif not need_to_reorder_bytes: + raise ValueError(f"invalid variables record version: {version}") + + if variables_structure.fields[-1].format != old_format: + LOG.debug( + "change variables record from {} to {}".format( + old_format, variables_structure.fields[-1].format + ) + ) + variables_structure.setup() + elif need_to_reorder_bytes: + variables_structure.setup() + + # we might need to unpack again with the new byte order + return need_to_reorder_bytes + + +VariablesRecordStructure = DynamicStructure( + name="VariablesRecord", + fields=[ + DynamicVersionField("h", "version", help="Version number for this header."), + Field(Variables1, "variables", help="The rest of the variables data."), + ], +) + + +class VariablesRecord(Record): + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + # self.header['version'] # record version always 0? + VariablesRecordStructure.byte_order = "=" + VariablesRecordStructure.setup() + stream = io.BytesIO(bytes(self.data)) + self.variables = VariablesRecordStructure.unpack_stream(stream) + self.namespace = {} + for key, value in self.variables["variables"].items(): + if key not in ["var_header"]: + LOG.debug( + "update namespace {} with {} for {}".format( + self.namespace, value, key + ) + ) + self.namespace.update(value) diff --git a/src/igor/record/variables.pyi b/src/igor/record/variables.pyi new file mode 100644 index 0000000..e3d5e67 --- /dev/null +++ b/src/igor/record/variables.pyi @@ -0,0 +1,60 @@ +from __future__ import annotations +from ..binarywave import DynamicStringField, NullStaticStringField +from ..struct import ( + DynamicField, + DynamicStructure, + Structure, +) +from .base import Record +from _typeshed import Incomplete +from .._typing import BYTEORDER + +class ListedStaticStringField(NullStaticStringField): + def post_unpack(self, parents: type, data: bytes) -> None: ... + +class ListedDynamicStrDataField(DynamicStringField, ListedStaticStringField): ... + +class DynamicVarDataField(DynamicField): + def __init__(self, *args: Incomplete, **kwargs: Incomplete) -> None: ... + def pre_pack(self, parents: Incomplete, data: Incomplete) -> None: ... + def post_unpack(self, parents: Incomplete, data: Incomplete) -> None: ... + +class DynamicSysVarField(DynamicVarDataField): ... +class DynamicUserVarField(DynamicVarDataField): ... +class DynamicUserStrField(DynamicVarDataField): ... + +class DynamicVarNumField(DynamicField): + def post_unpack(self, parents: type, data: bytes) -> None: ... + +class DynamicFormulaField(DynamicStringField): ... + +VarHeader1: Structure +VarHeader2: Structure +UserStrVarRec1: DynamicStructure +UserStrVarRec2: DynamicStructure +VarNumRec: Structure +UserNumVarRec: DynamicStructure +UserDependentVarRec: DynamicStructure + +class DynamicVarHeaderField(DynamicField): + def pre_pack(self, parents: Incomplete, data: Incomplete) -> None: ... + def post_unpack(self, parents: Incomplete, data: Incomplete) -> None: ... + +Variables1: DynamicVarHeaderField +Variables2: DynamicStructure + +class DynamicVersionField(DynamicField): + def pre_pack(self, parents: Incomplete, byte_order: BYTEORDER) -> None: ... + def post_unpack(self, parents: type, data: bytes) -> None: ... + +VariablesRecordStructure: DynamicStructure + +class VariablesRecord(Record): + variables: dict[str, float | dict[str, float | dict[str, float]]] + namespace: dict[str | bytes, float] + + def __init__( + self, + *args: dict[str, int] | bytes | BYTEORDER, + **kwargs: dict[str, int] | bytes | BYTEORDER, + ) -> None: ... diff --git a/igor/record/wave.py b/src/igor/record/wave.py similarity index 77% rename from igor/record/wave.py rename to src/igor/record/wave.py index 49ed20e..13858dd 100644 --- a/igor/record/wave.py +++ b/src/igor/record/wave.py @@ -15,16 +15,18 @@ # You should have received a copy of the GNU Lesser General Public License # along with igor. If not, see . -from io import BytesIO as _BytesIO +# Memo No diff with igor2 -from ..binarywave import load as _loadibw -from . import Record +from io import BytesIO +from ..binarywave import load as loadibw +from .base import Record -class WaveRecord (Record): + +class WaveRecord(Record): def __init__(self, *args, **kwargs): - super(WaveRecord, self).__init__(*args, **kwargs) - self.wave = _loadibw(_BytesIO(bytes(self.data))) + super().__init__(*args, **kwargs) + self.wave = loadibw(BytesIO(bytes(self.data))) def __str__(self): return str(self.wave) diff --git a/src/igor/record/wave.pyi b/src/igor/record/wave.pyi new file mode 100644 index 0000000..42f61e9 --- /dev/null +++ b/src/igor/record/wave.pyi @@ -0,0 +1,12 @@ +from __future__ import annotations +from .base import Record as Record +from .._typing import IBW, BYTEORDER + +class WaveRecord(Record): + wave: IBW + + def __init__( + self, + *args: dict[str, int] | bytes | BYTEORDER, + **kwargs: dict[str, int] | bytes | BYTEORDER, + ) -> None: ... diff --git a/src/igor/script.py b/src/igor/script.py new file mode 100644 index 0000000..dc58156 --- /dev/null +++ b/src/igor/script.py @@ -0,0 +1,95 @@ +# Copyright (C) 2012-2016 W. Trevor King +# +# This file is part of igor. +# +# igor is free software: you can redistribute it and/or modify it under the +# terms of the GNU Lesser General Public License as published by the Free +# Software Foundation, either version 3 of the License, or (at your option) any +# later version. +# +# igor is distributed in the hope that it will be useful, but WITHOUT ANY +# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR +# A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more +# details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with igor. If not, see . + +# Memo In igor2 there is not the file named script.py + + +"Common code for scripts distributed with the `igor` package." + +import argparse +import logging +import sys + +import matplotlib.pyplot as plt + +from . import __version__ +from . import LOG + + +class Script: + log_levels = [logging.ERROR, logging.WARNING, logging.INFO, logging.DEBUG] + + def __init__(self, description=None, filetype="IGOR Binary Wave (.ibw) file"): + self.parser = argparse.ArgumentParser(description=description) + self.parser.add_argument( + "--version", action="version", version=f"%(prog)s {__version__}" + ) + self.parser.add_argument( + "-f", + "--infile", + metavar="FILE", + default="-", + help=f"input {filetype}", + ) + self.parser.add_argument( + "-o", "--outfile", metavar="FILE", default="-", help="file for ASCII output" + ) + self.parser.add_argument( + "-p", + "--plot", + action="store_const", + const=True, + help="use Matplotlib to plot any IGOR waves", + ) + self.parser.add_argument( + "-V", "--verbose", action="count", default=0, help="increment verbosity" + ) + self._num_plots = 0 + + def run(self, *args, **kwargs): + args = self.parser.parse_args(*args, **kwargs) + if args.infile == "-": + args.infile = sys.stdin + if args.outfile == "-": + args.outfile = sys.stdout + if args.verbose > 1: + log_level = self.log_levels[min(args.verbose - 1, len(self.log_levels) - 1)] + LOG.setLevel(log_level) + self._run(args) + self.display_plots() + + def _run(self, args): + raise NotImplementedError() + + def plot_wave(self, args, wave, title=None): + if not args.plot: + return # no-op + if title is None: + title = wave["wave"]["wave_header"]["bname"] + figure = plt.figure() + axes = figure.add_subplot(1, 1, 1) + axes.set_title(title) + try: + axes.plot(wave["wave"]["wData"], "r.") + except ValueError as error: + LOG.error(f"error plotting {title}: {error}") + pass + self._num_plots += 1 + + def display_plots(self): + if self._num_plots: + plt.show() diff --git a/src/igor/script.pyi b/src/igor/script.pyi new file mode 100644 index 0000000..6828c96 --- /dev/null +++ b/src/igor/script.pyi @@ -0,0 +1,22 @@ +from __future__ import annotations +from . import LOG as LOG +from _typeshed import Incomplete + + +class Script: + log_levels: Incomplete + parser: Incomplete + + def __init__( + self, description: Incomplete | None = ..., filetype: str = ... + ) -> None: + ... + + def run(self, *args, **kwargs) -> None: + ... + + def plot_wave(self, args, wave, title: Incomplete | None = ...) -> None: + ... + + def display_plots(self) -> None: + ... diff --git a/igor/struct.py b/src/igor/struct.py similarity index 77% rename from igor/struct.py rename to src/igor/struct.py index a50ede5..7512da0 100644 --- a/igor/struct.py +++ b/src/igor/struct.py @@ -15,6 +15,8 @@ # You should have received a copy of the GNU Lesser General Public License # along with igor. If not, see . +# Memo Diffierence (small but significant) + """Structure and Field classes for declaring structures There are a few formats that can be used to represent the same data, a @@ -22,19 +24,18 @@ format with each field in a single Python list, and a nested format with each field in a hierarchy of Python dictionaries. """ +from __future__ import annotations +import io +import logging +import pprint +import struct +import numpy as np -from __future__ import absolute_import -import io as _io -import logging as _logging -import pprint as _pprint -import struct as _struct - -import numpy as _numpy +from . import __version__ +from . import LOG -from . import LOG as _LOG - -class Field (object): +class Field: """Represent a Structure field. The format argument can be a format character from the ``struct`` @@ -148,12 +149,14 @@ class Field (object): -------- Structure """ - def __init__(self, format, name, default=None, help=None, count=1, - array=False): + + def __init__( + self, format, name, default=None, help=None, count: int = 1, array: bool = False + ) -> None: self.format = format self.name = name - self.default = default - self.help = help + self.default = default # Seems not to be used + self.help = help # Seems not to be used (?) self.count = count self.array = array self.setup() @@ -161,20 +164,21 @@ def __init__(self, format, name, default=None, help=None, count=1, def setup(self): """Setup any dynamic properties of a field. - Use this method to recalculate dynamic properities after + Use this method to recalculate dynamic properties after changing the basic properties set during initialization. """ - _LOG.debug('setup {}'.format(self)) - self.item_count = _numpy.prod(self.count) # number of item repeats + LOG.debug(f"setup {self}") + self.item_count = np.prod(self.count) # number of item repeats if not self.array and self.item_count != 1: raise ValueError( - '{} must be an array field to have a count of {}'.format( - self, self.count)) + "{} must be an array field to have a count of {}".format( + self, self.count + ) + ) if isinstance(self.format, Structure): - self.structure_count = sum( - f.arg_count for f in self.format.fields) + self.structure_count = sum(f.arg_count for f in self.format.fields) self.arg_count = self.item_count * self.structure_count - elif self.format == 'x': + elif self.format == "x": self.arg_count = 0 # no data in padding bytes else: self.arg_count = self.item_count # struct.Struct format args @@ -183,8 +187,7 @@ def __str__(self): return self.__repr__() def __repr__(self): - return '<{} {} {}>'.format( - self.__class__.__name__, self.name, id(self)) + return f"<{self.__class__.__name__} {self.name} {id(self)}>" def indexes(self): """Iterate through indexes to a possibly multi-dimensional array""" @@ -197,7 +200,7 @@ def indexes(self): else: for i in range(self.item_count): index = [] - for j,c in enumerate(reversed(self.count)): + for j, c in enumerate(reversed(self.count)): index.insert(0, i % c) i //= c yield index @@ -211,18 +214,19 @@ def pack_data(self, data=None): if self.array: if data is None: data = [] - if hasattr(data, 'flat'): # take advantage of numpy's ndarray.flat + if hasattr(data, "flat"): # take advantage of numpy's ndarray.flat items = 0 for item in data.flat: items += 1 - for arg in self.pack_item(item): - yield arg + yield from self.pack_item(item) if items < self.item_count: + raise NotADirectoryError("Very bad state") + """ if f.default is None: - raise ValueError( - 'no default for {}.{}'.format(self, f)) + raise ValueError("no default for {}.{}".format(self, f)) for i in range(self.item_count - items): yield f.default + """ else: for index in self.indexes(): try: @@ -234,39 +238,35 @@ def pack_data(self, data=None): item = item[i] except IndexError: item = None - for arg in self.pack_item(item): - yield arg + yield from self.pack_item(item) else: - for arg in self.pack_item(data): - yield arg + yield from self.pack_item(data) def pack_item(self, item=None): - """Linearize a single count of the field's data to a flat iterable - """ + """Linearize a single count of the field's data to a flat iterable""" if isinstance(self.format, Structure): - for i in self.format._pack_item(item): - yield i + yield from self.format._pack_item(item) elif item is None: if self.default is None: - raise ValueError('no default for {}'.format(self)) + raise ValueError(f"no default for {self}") yield self.default else: yield item def unpack_data(self, data): """Inverse of .pack_data""" - _LOG.debug('unpack {} for {} {}'.format(data, self, self.format)) + LOG.debug(f"unpack {data} for {self} {self.format}") iterator = iter(data) try: items = [next(iterator) for i in range(self.arg_count)] except StopIteration: - raise ValueError('not enough data to unpack {}'.format(self)) + raise ValueError(f"not enough data to unpack {self}") try: next(iterator) except StopIteration: pass else: - raise ValueError('too much data to unpack {}'.format(self)) + raise ValueError(f"too much data to unpack {self}") if isinstance(self.format, Structure): # break into per-structure clumps s = self.structure_count @@ -287,11 +287,10 @@ def unpack_data(self, data): except TypeError: pass else: - raise NotImplementedError('reshape Structure field') + raise NotImplementedError("reshape Structure field") else: - unpacked = _numpy.array(unpacked) - _LOG.debug('reshape {} data from {} to {}'.format( - self, unpacked.shape, count)) + unpacked = np.array(unpacked) + LOG.debug(f"reshape {self} data from {unpacked.shape} to {count}") unpacked = unpacked.reshape(count) return unpacked @@ -304,8 +303,8 @@ def unpack_item(self, item): return item[0] -class DynamicField (Field): - """Represent a DynamicStructure field with a dynamic definition. +class DynamicField(Field): + r"""Represent a DynamicStructure field with a dynamic definition. Adds the methods ``.pre_pack``, ``pre_unpack``, and ``post_unpack``, all of which are called when a ``DynamicField`` @@ -321,6 +320,7 @@ class DynamicField (Field): -------- Field, DynamicStructure """ + def pre_pack(self, parents, data): "Prepare to pack." pass @@ -334,8 +334,7 @@ def post_unpack(self, parents, data): pass def _get_structure_data(self, parents, data, structure): - """Extract the data belonging to a particular ancestor structure. - """ + """Extract the data belonging to a particular ancestor structure.""" d = data s = parents[0] if s == structure: @@ -352,7 +351,7 @@ def _get_structure_data(self, parents, data, structure): return d -class Structure (_struct.Struct): +class Structure(struct.Struct): r"""Represent a C structure. A convenient wrapper around struct.Struct that uses Fields and @@ -479,9 +478,9 @@ class Structure (_struct.Struct): >>> b2 == b True """ - _byte_order_symbols = '@=<>!' + _byte_order_symbols = "@=<>!" - def __init__(self, name, fields, byte_order='@'): + def __init__(self, name, fields, byte_order="@"): # '=' for native byte order, standard size and alignment # See http://docs.python.org/library/struct for details self.name = name @@ -493,8 +492,7 @@ def __str__(self): return self.name def __repr__(self): - return '<{} {} {}>'.format( - self.__class__.__name__, self.name, id(self)) + return f"<{self.__class__.__name__} {self.name} {id(self)}>" def setup(self): """Setup any dynamic properties of a structure. @@ -502,44 +500,40 @@ def setup(self): Use this method to recalculate dynamic properities after changing the basic properties set during initialization. """ - _LOG.debug('setup {!r}'.format(self)) + LOG.debug(f"setup {self!r}") self.set_byte_order(self.byte_order) self.get_format() def set_byte_order(self, byte_order): - """Allow changing the format byte_order on the fly. - """ - _LOG.debug('set byte order for {!r} to {}'.format(self, byte_order)) + """Allow changing the format byte_order on the fly.""" + LOG.debug(f"set byte order for {self!r} to {byte_order}") self.byte_order = byte_order for field in self.fields: if isinstance(field.format, Structure): field.format.set_byte_order(byte_order) - def get_format(self): - format = self.byte_order + ''.join(self.sub_format()) + def get_format(self) -> str: + format = self.byte_order + "".join(self.sub_format()) # P format only allowed for native byte ordering # Convert P to I for ILP32 compatibility when running on a LP64. - format = format.replace('P', 'I') + format = format.replace("P", "I") try: - super(Structure, self).__init__(format=format) - except _struct.error as e: + super().__init__(format=format) + except struct.error as e: raise ValueError((e, format)) return format def sub_format(self): - _LOG.debug('calculate sub-format for {!r}'.format(self)) + LOG.debug(f"calculate sub-format for {self!r}") for field in self.fields: if isinstance(field.format, Structure): - field_format = list( - field.format.sub_format()) * field.item_count + field_format = list(field.format.sub_format()) * field.item_count else: - field_format = [field.format]*field.item_count - for fmt in field_format: - yield fmt + field_format = [field.format] * field.item_count + yield from field_format - def _pack_item(self, item=None): - """Linearize a single count of the structure's data to a flat iterable - """ + def _pack_item(self, item: dict | None = None): + """Linearize a single count of the structure's data to a flat iterable""" if item is None: item = {} for f in self.fields: @@ -549,8 +543,7 @@ def _pack_item(self, item=None): raise ValueError((f.name, item)) except KeyError: data = None - for arg in f.pack_data(data): - yield arg + yield from f.pack_data(data) def _unpack_item(self, args): """Inverse of ._unpack_item""" @@ -560,57 +553,57 @@ def _unpack_item(self, args): try: items = [next(iterator) for i in range(f.arg_count)] except StopIteration: - raise ValueError('not enough data to unpack {}.{}'.format( - self, f)) + raise ValueError(f"not enough data to unpack {self}.{f}") data[f.name] = f.unpack_data(items) try: next(iterator) except StopIteration: pass else: - raise ValueError('too much data to unpack {}'.format(self)) + raise ValueError(f"too much data to unpack {self}") return data def pack(self, data): args = list(self._pack_item(data)) try: - return super(Structure, self).pack(*args) + return super().pack(*args) except: raise ValueError(self.format) - def pack_into(self, buffer, offset=0, data={}): + def pack_into( + self, buffer, offset=0, data={} + ): # check !! should data:dict|None = None? args = list(self._pack_item(data)) - return super(Structure, self).pack_into( - buffer, offset, *args) + return super().pack_into(buffer, offset, *args) def unpack(self, *args, **kwargs): - args = super(Structure, self).unpack(*args, **kwargs) + args = super().unpack(*args, **kwargs) return self._unpack_item(args) def unpack_from(self, buffer, offset=0, *args, **kwargs): - _LOG.debug( - 'unpack {!r} for {!r} ({}, offset={}) with {} ({})'.format( - buffer, self, len(buffer), offset, self.format, self.size)) - args = super(Structure, self).unpack_from( - buffer, offset, *args, **kwargs) + LOG.debug( + "unpack {!r} for {!r} ({}, offset={}) with {} ({})".format( + buffer, self, len(buffer), offset, self.format, self.size + ) + ) + args = super().unpack_from(buffer, offset, *args, **kwargs) return self._unpack_item(args) def get_field(self, name): return [f for f in self.fields if f.name == name][0] -class DebuggingStream (object): +class DebuggingStream: def __init__(self, stream): self.stream = stream def read(self, size): data = self.stream.read(size) - _LOG.debug('read {} from {}: ({}) {!r}'.format( - size, self.stream, len(data), data)) + LOG.debug(f"read {size} from {self.stream}: ({len(data)}) {data!r}") return data -class DynamicStructure (Structure): +class DynamicStructure(Structure): r"""Represent a C structure field with a dynamic definition. Any dynamic fields have their ``.pre_pack`` called before any @@ -698,54 +691,56 @@ class DynamicStructure (Structure): for ``Structure``, because we must make multiple calls to ``struct.Struct.unpack`` to unpack the data. """ - #def __init__(self, *args, **kwargs): + # def __init__(self, *args, **kwargs): # pass #self.parent = .. - def _pre_pack(self, parents=None, data=None): + def _pre_pack(self, parents: list | None = None, data=None): if parents is None: parents = [self] else: parents = parents + [self] for f in self.fields: - if hasattr(f, 'pre_pack'): - _LOG.debug('pre-pack {}'.format(f)) + if hasattr(f, "pre_pack"): + LOG.debug(f"pre-pack {f}") f.pre_pack(parents=parents, data=data) if isinstance(f.format, DynamicStructure): - _LOG.debug('pre-pack {!r}'.format(f.format)) + LOG.debug(f"pre-pack {f.format!r}") f._pre_pack(parents=parents, data=data) def pack(self, data): self._pre_pack(data=data) self.setup() - return super(DynamicStructure, self).pack(data) + return super().pack(data) def pack_into(self, buffer, offset=0, data={}): self._pre_pack(data=data) self.setup() - return super(DynamicStructure, self).pack_into( - buffer=buffer, offset=offset, data=data) + return super().pack_into(buffer=buffer, offset=offset, data=data) - def unpack_stream(self, stream, parents=None, data=None, d=None): + def unpack_stream(self, stream, parents=None, data: dict | None = None, d=None): # `d` is the working data directory if data is None: parents = [self] data = d = {} - if _LOG.level <= _logging.DEBUG: + if LOG.level <= logging.DEBUG: stream = DebuggingStream(stream) else: parents = parents + [self] for f in self.fields: - _LOG.debug('parsing {!r}.{} (count={}, item_count={})'.format( - self, f, f.count, f.item_count)) - if _LOG.level <= _logging.DEBUG: - _LOG.debug('data:\n{}'.format(_pprint.pformat(data))) - if hasattr(f, 'pre_unpack'): - _LOG.debug('pre-unpack {}'.format(f)) + LOG.debug( + "parsing {!r}.{} (count={}, item_count={})".format( + self, f, f.count, f.item_count + ) + ) + if LOG.level <= logging.DEBUG: + LOG.debug(f"data:\n{pprint.pformat(data)}") + if hasattr(f, "pre_unpack"): + LOG.debug(f"pre-unpack {f}") f.pre_unpack(parents=parents, data=data) - if hasattr(f, 'unpack'): # override default unpacking - _LOG.debug('override unpack for {}'.format(f)) + if hasattr(f, "unpack"): # override default unpacking + LOG.debug(f"override unpack for {f}") d[f.name] = f.unpack(stream) continue @@ -761,23 +756,26 @@ def unpack_stream(self, stream, parents=None, data=None, d=None): x = {} d[f.name].append(x) f.format.unpack_stream( - stream, parents=parents, data=data, d=x) + stream, parents=parents, data=data, d=x + ) else: assert f.item_count == 1, (f, f.count) d[f.name] = {} f.format.unpack_stream( - stream, parents=parents, data=data, d=d[f.name]) - if hasattr(f, 'post_unpack'): - _LOG.debug('post-unpack {}'.format(f)) + stream, parents=parents, data=data, d=d[f.name] + ) + if hasattr(f, "post_unpack"): + LOG.debug(f"post-unpack {f}") repeat = f.post_unpack(parents=parents, data=data) if repeat: raise NotImplementedError( - 'cannot repeat unpack for dynamic structures') + "cannot repeat unpack for dynamic structures" + ) continue if isinstance(f.format, Structure): - _LOG.debug('parsing {} bytes for {}'.format( - f.format.size, f.format.format)) + LOG.debug(f"parsing {f.format.size} bytes for {f.format.format}") bs = [stream.read(f.format.size) for i in range(f.item_count)] + def unpack(): f.format.set_byte_order(self.byte_order) f.setup() @@ -787,50 +785,50 @@ def unpack(): assert len(x) == 1, (f, f.count, x) x = x[0] return x + else: - field_format = self.byte_order + f.format*f.item_count - field_format = field_format.replace('P', 'I') + field_format = self.byte_order + f.format * f.item_count + field_format = field_format.replace("P", "I") try: - size = _struct.calcsize(field_format) - except _struct.error as e: - _LOG.error(e) - _LOG.error('{}.{}: {}'.format(self, f, field_format)) + size = struct.calcsize(field_format) + except struct.error as e: + LOG.error(e) + LOG.error(f"{self}.{f}: {field_format}") raise - _LOG.debug('parsing {} bytes for preliminary {}'.format( - size, field_format)) + LOG.debug(f"parsing {size} bytes for preliminary {field_format}") raw = stream.read(size) if len(raw) < size: raise ValueError( - 'not enough data to unpack {}.{} ({} < {})'.format( - self, f, len(raw), size)) + "not enough data to unpack {}.{} ({} < {})".format( + self, f, len(raw), size + ) + ) + def unpack(): - field_format = self.byte_order + f.format*f.item_count - field_format = field_format.replace('P', 'I') - _LOG.debug('parse previous bytes using {}'.format( - field_format)) - struct = _struct.Struct(field_format) - items = struct.unpack(raw) + field_format = self.byte_order + f.format * f.item_count + field_format = field_format.replace("P", "I") + LOG.debug(f"parse previous bytes using {field_format}") + items = struct.Struct(field_format).unpack(raw) return f.unpack_data(items) # unpacking loop repeat = True while repeat: d[f.name] = unpack() - if hasattr(f, 'post_unpack'): - _LOG.debug('post-unpack {}'.format(f)) + if hasattr(f, "post_unpack"): + LOG.debug(f"post-unpack {f}") repeat = f.post_unpack(parents=parents, data=data) else: repeat = False if repeat: - _LOG.debug('repeat unpack for {}'.format(f)) + LOG.debug(f"repeat unpack for {f}") return data def unpack(self, string): - stream = _io.BytesIO(string) + stream = io.BytesIO(string) return self.unpack_stream(stream) def unpack_from(self, buffer, offset=0, *args, **kwargs): - args = super(Structure, self).unpack_from( - buffer, offset, *args, **kwargs) + args = super(Structure, self).unpack_from(buffer, offset, *args, **kwargs) return self._unpack_item(args) diff --git a/src/igor/struct.pyi b/src/igor/struct.pyi new file mode 100644 index 0000000..e184da9 --- /dev/null +++ b/src/igor/struct.pyi @@ -0,0 +1,167 @@ +from __future__ import annotations + +import struct +from collections.abc import Generator +from typing import BinaryIO + +import numpy as np +from _typeshed import Incomplete +from numpy.typing import NDArray + +from igor.binarywave import ( + DynamicDataUnitsField, + DynamicDependencyFormulaField, + DynamicDimensionUnitsField, + DynamicLabelsField, + DynamicStringIndicesDataField, + DynamicWaveDataField1, + DynamicWaveDataField5, + DynamicWaveNoteField, + NullStaticStringField, +) + +from ._typing import BYTEORDER, IBW + + +class Field: + format: str | Structure + name: str + default: int + help: str + count: int + array: bool + + def __init__( + self, + format: str | Structure, + name: str, + default: int | None = ..., + help: str | None = ..., + count: int = ..., + array: bool = ..., + ) -> None: + ... + + item_count: int + structure_count: int + arg_count: int + + def setup(self) -> None: + ... + + def indexes(self) -> Generator[int, None, None]: + ... + + def pack_data( + self, + data: list[dict[str, complex | NDArray[np.complex_]]] + | dict[str, complex | NDArray[np.complex_]] + | None = ..., + ) -> Generator[Incomplete, None, None]: + ... + + def pack_item( + self, item: Incomplete | None = ... + ) -> Generator[Incomplete, None, None]: + ... + + def unpack_data(self, data: bytes): + ... + + def unpack_item(self, item): + ... + + +class DynamicField(Field): + def pre_pack(self, parents: type, data: bytes) -> None: + ... + + def pre_unpack(self, parents: type, data: bytes) -> None: + ... + + def post_unpack(self, parents: type, data: bytes) -> None: + ... + + +class Structure(struct.Struct): + name: str + fields: list[ + Field + | NullStaticStringField + | DynamicWaveDataField1 + | DynamicWaveNoteField + | DynamicDependencyFormulaField + | DynamicWaveDataField5 + | DynamicDependencyFormulaField + | DynamicWaveNoteField + | DynamicDataUnitsField + | DynamicDimensionUnitsField + | DynamicLabelsField + | DynamicStringIndicesDataField + ] + byte_order: BYTEORDER + + def __init__( + self, name: str, fields: list[Field], byte_order: BYTEORDER = ... + ) -> None: + ... + + def setup(self) -> None: + ... + + def set_byte_order(self, byte_order: BYTEORDER) -> None: + ... + + def get_format(self) -> str: + ... + + def sub_format(self) -> Generator[Incomplete, None, None]: + ... + + def pack(self, data: dict): + ... + + def pack_into(self, buffer, offset: int = ..., data: bytes = ...): + ... + + def unpack(self, *args, **kwargs): + ... + + def unpack_from(self, buffer, offset: int = ..., *args, **kwargs): + ... + + def get_field(self, name: str): + ... + + +class DebuggingStream: + stream: BinaryIO + + def __init__(self, stream: BinaryIO) -> None: + ... + + def read(self, size: int): + ... + + +class DynamicStructure(Structure): + def pack(self, data: dict) -> bytearray: + ... + + def pack_into(self, buffer, offset: int = ..., data: bytes = ...) -> None: + ... + + def unpack_stream( + self, + stream: BinaryIO, + parents: type | None = ..., + data: dict | None = ..., + d: Incomplete | None = ..., + ) -> IBW: + ... + + def unpack(self, string): + ... + + def unpack_from(self, buffer, offset: int = ..., *args, **kwargs): + ... diff --git a/igor/util.py b/src/igor/util.py similarity index 73% rename from igor/util.py rename to src/igor/util.py index ecc783a..8dbdba4 100644 --- a/igor/util.py +++ b/src/igor/util.py @@ -15,11 +15,13 @@ # You should have received a copy of the GNU Lesser General Public License # along with igor. If not, see . -"Utility functions for handling buffers" +# Memo: No essential diff with igor2 -import sys as _sys +"Utility functions for handling buffers" +from __future__ import annotations +import sys as sys -import numpy as _numpy +import numpy as np def _ord(byte): @@ -29,10 +31,8 @@ def _ord(byte): >>> [_ord(b) for b in buffer] [0, 1, 2] """ - if _sys.version_info >= (3,): - return byte - else: - return ord(byte) + return byte + def hex_bytes(buffer, spaces=None): r"""Pretty-printing for binary buffers. @@ -48,14 +48,15 @@ def hex_bytes(buffer, spaces=None): >>> hex_bytes(b'\x00\x01\x02\x03\x04\x05\x06', spaces=3) '000102 030405 06' """ - hex_bytes = ['{:02x}'.format(_ord(x)) for x in buffer] + hex_bytes = [f"{_ord(x):02x}" for x in buffer] if spaces is None: - return ''.join(hex_bytes) - elif spaces is 1: - return ' '.join(hex_bytes) - for i in range(len(hex_bytes)//spaces): - hex_bytes.insert((spaces+1)*(i+1)-1, ' ') - return ''.join(hex_bytes) + return "".join(hex_bytes) + elif spaces == 1: + return " ".join(hex_bytes) + for i in range(len(hex_bytes) // spaces): + hex_bytes.insert((spaces + 1) * (i + 1) - 1, " ") + return "".join(hex_bytes) + def assert_null(buffer, strict=True): r"""Ensure an input buffer is entirely zero. @@ -78,17 +79,18 @@ def assert_null(buffer, strict=True): if strict: raise ValueError(hex_string) else: - _sys.stderr.write( - 'warning: post-data padding not zero: {}\n'.format(hex_string)) + sys.stderr.write(f"warning: post-data padding not zero: {hex_string}\n") + # From ReadWave.c def byte_order(needToReorderBytes): - little_endian = _sys.byteorder == 'little' + little_endian = sys.byteorder == "little" if needToReorderBytes: little_endian = not little_endian if little_endian: - return '<' # little-endian - return '>' # big-endian + return "<" # little-endian + return ">" # big-endian + # From ReadWave.c def need_to_reorder_bytes(version): @@ -98,20 +100,23 @@ def need_to_reorder_bytes(version): # reordered. return version & 0xFF == 0 + # From ReadWave.c def checksum(buffer, byte_order, oldcksum, numbytes): - x = _numpy.ndarray( - (numbytes/2,), # 2 bytes to a short -- ignore trailing odd byte - dtype=_numpy.dtype(byte_order+'h'), - buffer=buffer) + x = np.ndarray( + (numbytes / 2,), # 2 bytes to a short -- ignore trailing odd byte + dtype=np.dtype(byte_order + "h"), + buffer=buffer, + ) oldcksum += x.sum() if oldcksum > 2**31: # fake the C implementation's int rollover oldcksum %= 2**32 if oldcksum > 2**31: oldcksum -= 2**31 - return oldcksum & 0xffff + return oldcksum & 0xFFFF -def _bytes(obj, encoding='utf-8'): + +def _bytes(obj, encoding="utf-8"): """Convert bytes or strings into bytes >>> _bytes(b'123') @@ -119,10 +124,6 @@ def _bytes(obj, encoding='utf-8'): >>> _bytes('123') '123' """ - if _sys.version_info >= (3,): - if isinstance(obj, bytes): - return obj - else: - return bytes(obj, encoding) - else: - return bytes(obj) + if isinstance(obj, bytes): + return obj + return bytes(obj, encoding) diff --git a/src/igor/util.pyi b/src/igor/util.pyi new file mode 100644 index 0000000..1b82e10 --- /dev/null +++ b/src/igor/util.pyi @@ -0,0 +1,25 @@ +from __future__ import annotations +from _typeshed import Incomplete +from ._typing import BYTEORDER + + +def hex_bytes(buffer: Incomplete, spaces: int | None = ...) -> str: + ... + + +def assert_null(buffer: Incomplete, strict: bool = ...) -> None: + ... + + +def byte_order(needToReorderBytes: bool) -> BYTEORDER: + ... + + +def need_to_reorder_bytes(version: int) -> bool: + ... + + +def checksum( + buffer: Incomplete, byte_order: BYTEORDER, oldcksum: int, numbytes: int +) -> int: + ... diff --git a/test/data/ID_003.ibw b/test/data/ID_003.ibw new file mode 100644 index 0000000..33a4af4 Binary files /dev/null and b/test/data/ID_003.ibw differ diff --git a/test/helper.py b/test/helper.py new file mode 100644 index 0000000..8f88bdf --- /dev/null +++ b/test/helper.py @@ -0,0 +1,30 @@ +import pathlib +from pprint import pformat + +from igor.binarywave import load as loadibw + + +data_dir = pathlib.Path(__file__).parent / "data" + + +def assert_equal_dump_no_whitespace_no_byte(data_a, data_b): + a = data_a.replace(" ", "").replace("b'", "'").replace("\n", "") + b = data_b.replace(" ", "").replace("b'", "'").replace("\n", "") + assert a == b + + +def dumpibw(filename): + path = data_dir / filename + data = loadibw(path) + return format_data(data) + + +def format_data(data): + lines = pformat(data).splitlines() + return "\n".join([line.rstrip() for line in lines]) + + +def walk_callback(dirpath, key, value): + return "walk callback on ({}, {}, {})".format( + dirpath, key, "{...}" if isinstance(value, dict) else value + ) diff --git a/test/test_ibw.py b/test/test_ibw.py new file mode 100644 index 0000000..1eb1163 --- /dev/null +++ b/test/test_ibw.py @@ -0,0 +1,631 @@ +from helper import assert_equal_dump_no_whitespace_no_byte, dumpibw + + +def test_ibw01(): + act = dumpibw("mac-double.ibw") # doctest: +REPORT_UDIFF + + exp = """{'version': 2, + 'wave': {'bin_header': {'checksum': 25137, + 'noteSize': 0, + 'pictSize': 0, + 'wfmSize': 166}, + 'note': '', + 'padding': array([], dtype=float64), + 'wData': array([ 5., 4., 3., 2., 1.]), + 'wave_header': {'aModified': 0, + 'bname': 'double', + 'botFullScale': 0.0, + 'creationDate': 3001587842, + 'dataUnits': array(['', '', '', ''], + dtype='|S1'), + 'depID': 0, + 'fileName': 0, + 'formula': 0, + 'fsValid': 0, + 'hsA': 1.0, + 'hsB': 0.0, + 'kindBits': '\\x00', + 'modDate': 3001587842, + 'next': 0, + 'npnts': 5, + 'srcFldr': 0, + 'swModified': 0, + 'topFullScale': 0.0, + 'type': 4, + 'useBits': '\\x00', + 'wModified': 0, + 'wUnused': array(['', ''], + dtype='|S1'), + 'waveNoteH': 0, + 'whVersion': 0, + 'xUnits': array(['', '', '', ''], + dtype='|S1')}}}""" + + assert_equal_dump_no_whitespace_no_byte(act, exp) + + +def test_ibw_02(): + act = dumpibw("mac-textWave.ibw") + + exp = """ + {'version': 5, + 'wave': {'bin_header': {'checksum': 5554, + 'dataEUnitsSize': 0, + 'dimEUnitsSize': array([0, 0, 0, 0]), + 'dimLabelsSize': array([0, 0, 0, 0]), + 'formulaSize': 0, + 'noteSize': 0, + 'optionsSize1': 0, + 'optionsSize2': 0, + 'sIndicesSize': 20, + 'wfmSize': 338}, + 'data_units': '', + 'dimension_units': '', + 'formula': '', + 'labels': [[], [], [], []], + 'note': '', + 'sIndices': array([ 4, 7, 8, 14, 18]), + 'wData': array(['Mary', 'had', 'a', 'little', 'lamb'], + dtype='|S6'), + 'wave_header': {'aModified': 0, + 'bname': 'text0', + 'botFullScale': 0.0, + 'creationDate': 3001571199, + 'dFolder': 69554896, + 'dLock': 0, + 'dataEUnits': 0, + 'dataUnits': array(['', '', '', ''], + dtype='|S1'), + 'depID': 22, + 'dimEUnits': array([0, 0, 0, 0]), + 'dimLabels': array([0, 0, 0, 0]), + 'dimUnits': array([['', '', '', ''], + ['', '', '', ''], + ['', '', '', ''], + ['', '', '', '']], + dtype='|S1'), + 'fileName': 0, + 'formula': 0, + 'fsValid': 0, + 'kindBits': '\\x00', + 'modDate': 3001571215, + 'nDim': array([5, 0, 0, 0]), + 'next': 0, + 'npnts': 5, + 'sIndices': 69557296, + 'sfA': array([ 1., 1., 1., 1.]), + 'sfB': array([ 0., 0., 0., 0.]), + 'srcFldr': 0, + 'swModified': 1, + 'topFullScale': 0.0, + 'type': 0, + 'useBits': '\\x00', + 'wModified': 0, + 'waveNoteH': 0, + 'whUnused': array([0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0]), + 'whVersion': 1, + 'whpad1': array(['', '', '', '', '', ''], + dtype='|S1'), + 'whpad2': 0, + 'whpad3': 0, + 'whpad4': 0}}} + """ + + assert_equal_dump_no_whitespace_no_byte(act, exp) + + +def test_ibw_03(): + act = dumpibw("mac-version2.ibw") + exp = """ + {'version': 2, + 'wave': {'bin_header': {'checksum': -16803, + 'noteSize': 15, + 'pictSize': 0, + 'wfmSize': 146}, + 'note': 'This is a test.', + 'padding': array([], dtype=float64), + 'wData': array([ 5., 4., 3., 2., 1.], dtype=float32), + 'wave_header': {'aModified': 0, + 'bname': 'version2', + 'botFullScale': 0.0, + 'creationDate': 3001251979, + 'dataUnits': array(['', '', '', ''], + dtype='|S1'), + 'depID': 0, + 'fileName': 0, + 'formula': 0, + 'fsValid': 0, + 'hsA': 1.0, + 'hsB': 0.0, + 'kindBits': '\\x00', + 'modDate': 3001573594, + 'next': 0, + 'npnts': 5, + 'srcFldr': 0, + 'swModified': 0, + 'topFullScale': 0.0, + 'type': 2, + 'useBits': '\\x00', + 'wModified': 0, + 'wUnused': array(['', ''], + dtype='|S1'), + 'waveNoteH': 0, + 'whVersion': 0, + 'xUnits': array(['', '', '', ''], + dtype='|S1')}}} + """ + assert_equal_dump_no_whitespace_no_byte(act, exp) + + +def test_ibw_(): + act = dumpibw("mac-version3Dependent.ibw") + exp = """ + {'version': 3, + 'wave': {'bin_header': {'checksum': -32334, + 'formulaSize': 4, + 'noteSize': 0, + 'pictSize': 0, + 'wfmSize': 126}, + 'formula': ' K0', + 'note': '', + 'padding': array([], dtype=float64), + 'wData': array([], dtype=float32), + 'wave_header': {'aModified': 3, + 'bname': 'version3Dependent', + 'botFullScale': 0.0, + 'creationDate': 0, + 'dataUnits': array(['', '', '', ''], + dtype='|S1'), + 'depID': 23, + 'fileName': 0, + 'formula': 103408364, + 'fsValid': 0, + 'hsA': 1.0, + 'hsB': 0.0, + 'kindBits': '\\x00', + 'modDate': 3001672861, + 'next': 0, + 'npnts': 10, + 'srcFldr': 0, + 'swModified': 1, + 'topFullScale': 0.0, + 'type': 2, + 'useBits': '\\x00', + 'wModified': 1, + 'wUnused': array(['', ''], + dtype='|S1'), + 'waveNoteH': 0, + 'whVersion': 0, + 'xUnits': array(['', '', '', ''], + dtype='|S1')}}} + """ + assert_equal_dump_no_whitespace_no_byte(act, exp) + + +def test_ibw_04(): + act = dumpibw("mac-version5.ibw") + exp = """ + {'version': 5, + 'wave': {'bin_header': {'checksum': -12033, + 'dataEUnitsSize': 0, + 'dimEUnitsSize': array([0, 0, 0, 0]), + 'dimLabelsSize': array([64, 0, 0, 0]), + 'formulaSize': 0, + 'noteSize': 15, + 'optionsSize1': 0, + 'optionsSize2': 0, + 'sIndicesSize': 0, + 'wfmSize': 340}, + 'data_units': '', + 'dimension_units': '', + 'formula': '', + 'labels': [['', 'Column0'], [], [], []], + 'note': 'This is a test.', + 'sIndices': array([], dtype=float64), + 'wData': array([ 5., 4., 3., 2., 1.], dtype=float32), + 'wave_header': {'aModified': 0, + 'bname': 'version5', + 'botFullScale': 0.0, + 'creationDate': 3001252180, + 'dFolder': 69554896, + 'dLock': 0, + 'dataEUnits': 0, + 'dataUnits': array(['', '', '', ''], + dtype='|S1'), + 'depID': 27, + 'dimEUnits': array([0, 0, 0, 0]), + 'dimLabels': array([69554136, 0, 0, 0]), + 'dimUnits': array([['', '', '', ''], + ['', '', '', ''], + ['', '', '', ''], + ['', '', '', '']], + dtype='|S1'), + 'fileName': 69554292, + 'formula': 0, + 'fsValid': 0, + 'kindBits': '\\x00', + 'modDate': 3001573601, + 'nDim': array([5, 0, 0, 0]), + 'next': 69555212, + 'npnts': 5, + 'sIndices': 0, + 'sfA': array([ 1., 1., 1., 1.]), + 'sfB': array([ 0., 0., 0., 0.]), + 'srcFldr': -32349, + 'swModified': 1, + 'topFullScale': 0.0, + 'type': 2, + 'useBits': '\\x00', + 'wModified': 0, + 'waveNoteH': 69554032, + 'whUnused': array([0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0]), + 'whVersion': 1, + 'whpad1': array(['', '', '', '', '', ''], + dtype='|S1'), + 'whpad2': 0, + 'whpad3': 0, + 'whpad4': 0}}} + """ + assert_equal_dump_no_whitespace_no_byte(act, exp) + + +def test_ibw_05(): + act = dumpibw("mac-zeroPointWave.ibw") + exp = """ + {'version': 5, + 'wave': {'bin_header': {'checksum': -15649, + 'dataEUnitsSize': 0, + 'dimEUnitsSize': array([0, 0, 0, 0]), + 'dimLabelsSize': array([0, 0, 0, 0]), + 'formulaSize': 0, + 'noteSize': 0, + 'optionsSize1': 0, + 'optionsSize2': 0, + 'sIndicesSize': 0, + 'wfmSize': 320}, + 'data_units': '', + 'dimension_units': '', + 'formula': '', + 'labels': [[], [], [], []], + 'note': '', + 'sIndices': array([], dtype=float64), + 'wData': array([], dtype=float32), + 'wave_header': {'aModified': 3, + 'bname': 'zeroWave', + 'botFullScale': 0.0, + 'creationDate': 3001573964, + 'dFolder': 69554896, + 'dLock': 0, + 'dataEUnits': 0, + 'dataUnits': array(['', '', '', ''], + dtype='|S1'), + 'depID': 29, + 'dimEUnits': array([0, 0, 0, 0]), + 'dimLabels': array([0, 0, 0, 0]), + 'dimUnits': array([['', '', '', ''], + ['', '', '', ''], + ['', '', '', ''], + ['', '', '', '']], + dtype='|S1'), + 'fileName': 0, + 'formula': 0, + 'fsValid': 0, + 'kindBits': '\\x00', + 'modDate': 3001573964, + 'nDim': array([0, 0, 0, 0]), + 'next': 0, + 'npnts': 0, + 'sIndices': 0, + 'sfA': array([ 1., 1., 1., 1.]), + 'sfB': array([ 0., 0., 0., 0.]), + 'srcFldr': 0, + 'swModified': 1, + 'topFullScale': 0.0, + 'type': 2, + 'useBits': '\\x00', + 'wModified': 1, + 'waveNoteH': 0, + 'whUnused': array([0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0]), + 'whVersion': 1, + 'whpad1': array(['', '', '', '', '', ''], + dtype='|S1'), + 'whpad2': 0, + 'whpad3': 0, + 'whpad4': 0}}} + """ + assert_equal_dump_no_whitespace_no_byte(act, exp) + + +def test_ibw_06(): + act = dumpibw("win-double.ibw") + exp = """ + {'version': 2, + 'wave': {'bin_header': {'checksum': 28962, + 'noteSize': 0, + 'pictSize': 0, + 'wfmSize': 166}, + 'note': '', + 'padding': array([], dtype=float64), + 'wData': array([ 5., 4., 3., 2., 1.]), + 'wave_header': {'aModified': 0, + 'bname': 'double', + 'botFullScale': 0.0, + 'creationDate': 3001587842, + 'dataUnits': array(['', '', '', ''], + dtype='|S1'), + 'depID': 0, + 'fileName': 0, + 'formula': 0, + 'fsValid': 0, + 'hsA': 1.0, + 'hsB': 0.0, + 'kindBits': '\\x00', + 'modDate': 3001587842, + 'next': 0, + 'npnts': 5, + 'srcFldr': 0, + 'swModified': 0, + 'topFullScale': 0.0, + 'type': 4, + 'useBits': '\\x00', + 'wModified': 0, + 'wUnused': array(['', ''], + dtype='|S1'), + 'waveNoteH': 0, + 'whVersion': 0, + 'xUnits': array(['', '', '', ''], + dtype='|S1')}}} + """ + assert_equal_dump_no_whitespace_no_byte(act, exp) + + +def test_ibw_07(): + act = dumpibw("win-textWave.ibw") + exp = """ + {'version': 5, + 'wave': {'bin_header': {'checksum': 184, + 'dataEUnitsSize': 0, + 'dimEUnitsSize': array([0, 0, 0, 0]), + 'dimLabelsSize': array([0, 0, 0, 0]), + 'formulaSize': 0, + 'noteSize': 0, + 'optionsSize1': 0, + 'optionsSize2': 0, + 'sIndicesSize': 20, + 'wfmSize': 338}, + 'data_units': '', + 'dimension_units': '', + 'formula': '', + 'labels': [[], [], [], []], + 'note': '', + 'sIndices': array([ 4, 7, 8, 14, 18]), + 'wData': array(['Mary', 'had', 'a', 'little', 'lamb'], + dtype='|S6'), + 'wave_header': {'aModified': 0, + 'bname': 'text0', + 'botFullScale': 0.0, + 'creationDate': 3001571199, + 'dFolder': 8108612, + 'dLock': 0, + 'dataEUnits': 0, + 'dataUnits': array(['', '', '', ''], + dtype='|S1'), + 'depID': 32, + 'dimEUnits': array([0, 0, 0, 0]), + 'dimLabels': array([0, 0, 0, 0]), + 'dimUnits': array([['', '', '', ''], + ['', '', '', ''], + ['', '', '', ''], + ['', '', '', '']], + dtype='|S1'), + 'fileName': 7814472, + 'formula': 0, + 'fsValid': 0, + 'kindBits': '\\x00', + 'modDate': 3001571215, + 'nDim': array([5, 0, 0, 0]), + 'next': 0, + 'npnts': 5, + 'sIndices': 8133100, + 'sfA': array([ 1., 1., 1., 1.]), + 'sfB': array([ 0., 0., 0., 0.]), + 'srcFldr': -1007, + 'swModified': 0, + 'topFullScale': 0.0, + 'type': 0, + 'useBits': '\\x00', + 'wModified': 1, + 'waveNoteH': 0, + 'whUnused': array([0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0]), + 'whVersion': 1, + 'whpad1': array(['', '', '', '', '', ''], + dtype='|S1'), + 'whpad2': 0, + 'whpad3': 0, + 'whpad4': 0}}} + """ + assert_equal_dump_no_whitespace_no_byte(act, exp) + + +def test_ibw_08(): + act = dumpibw("win-version2.ibw") + exp = """ + {'version': 2, + 'wave': {'bin_header': {'checksum': 1047, + 'noteSize': 15, + 'pictSize': 0, + 'wfmSize': 146}, + 'note': 'This is a test.', + 'padding': array([], dtype=float64), + 'wData': array([ 5., 4., 3., 2., 1.], dtype=float32), + 'wave_header': {'aModified': 0, + 'bname': 'version2', + 'botFullScale': 0.0, + 'creationDate': 3001251979, + 'dataUnits': array(['', '', '', ''], + dtype='|S1'), + 'depID': 0, + 'fileName': 0, + 'formula': 0, + 'fsValid': 0, + 'hsA': 1.0, + 'hsB': 0.0, + 'kindBits': '\\x00', + 'modDate': 3001573594, + 'next': 0, + 'npnts': 5, + 'srcFldr': 0, + 'swModified': 0, + 'topFullScale': 0.0, + 'type': 2, + 'useBits': '\\x00', + 'wModified': 0, + 'wUnused': array(['', ''], + dtype='|S1'), + 'waveNoteH': 0, + 'whVersion': 0, + 'xUnits': array(['', '', '', ''], + dtype='|S1')}}} + """ + assert_equal_dump_no_whitespace_no_byte(act, exp) + + +def test_ibw_09(): + act = dumpibw("win-version5.ibw") + exp = """ + {'version': 5, + 'wave': {'bin_header': {'checksum': 13214, + 'dataEUnitsSize': 0, + 'dimEUnitsSize': array([0, 0, 0, 0]), + 'dimLabelsSize': array([64, 0, 0, 0]), + 'formulaSize': 0, + 'noteSize': 15, + 'optionsSize1': 0, + 'optionsSize2': 0, + 'sIndicesSize': 0, + 'wfmSize': 340}, + 'data_units': '', + 'dimension_units': '', + 'formula': '', + 'labels': [['', 'Column0'], [], [], []], + 'note': 'This is a test.', + 'sIndices': array([], dtype=float64), + 'wData': array([ 5., 4., 3., 2., 1.], dtype=float32), + 'wave_header': {'aModified': 0, + 'bname': 'version5', + 'botFullScale': 0.0, + 'creationDate': 3001252180, + 'dFolder': 8108612, + 'dLock': 0, + 'dataEUnits': 0, + 'dataUnits': array(['', '', '', ''], + dtype='|S1'), + 'depID': 30, + 'dimEUnits': array([0, 0, 0, 0]), + 'dimLabels': array([8138784, 0, 0, 0]), + 'dimUnits': array([['', '', '', ''], + ['', '', '', ''], + ['', '', '', ''], + ['', '', '', '']], + dtype='|S1'), + 'fileName': 8131824, + 'formula': 0, + 'fsValid': 0, + 'kindBits': '\\x00', + 'modDate': 3001573601, + 'nDim': array([5, 0, 0, 0]), + 'next': 8125236, + 'npnts': 5, + 'sIndices': 0, + 'sfA': array([ 1., 1., 1., 1.]), + 'sfB': array([ 0., 0., 0., 0.]), + 'srcFldr': -1007, + 'swModified': 0, + 'topFullScale': 0.0, + 'type': 2, + 'useBits': '\\x00', + 'wModified': 1, + 'waveNoteH': 8131596, + 'whUnused': array([0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0]), + 'whVersion': 1, + 'whpad1': array(['', '', '', '', '', ''], + dtype='|S1'), + 'whpad2': 0, + 'whpad3': 0, + 'whpad4': 0}}} + """ + assert_equal_dump_no_whitespace_no_byte(act, exp) + + +def test_ibw_10(): + act = dumpibw("win-zeroPointWave.ibw") + + exp = """ + {'version': 5, + 'wave': {'bin_header': {'checksum': 27541, + 'dataEUnitsSize': 0, + 'dimEUnitsSize': array([0, 0, 0, 0]), + 'dimLabelsSize': array([0, 0, 0, 0]), + 'formulaSize': 0, + 'noteSize': 0, + 'optionsSize1': 0, + 'optionsSize2': 0, + 'sIndicesSize': 0, + 'wfmSize': 320}, + 'data_units': '', + 'dimension_units': '', + 'formula': '', + 'labels': [[], [], [], []], + 'note': '', + 'sIndices': array([], dtype=float64), + 'wData': array([], dtype=float32), + 'wave_header': {'aModified': 3, + 'bname': 'zeroWave', + 'botFullScale': 0.0, + 'creationDate': 3001573964, + 'dFolder': 8108612, + 'dLock': 0, + 'dataEUnits': 0, + 'dataUnits': array(['', '', '', ''], + dtype='|S1'), + 'depID': 31, + 'dimEUnits': array([0, 0, 0, 0]), + 'dimLabels': array([0, 0, 0, 0]), + 'dimUnits': array([['', '', '', ''], + ['', '', '', ''], + ['', '', '', ''], + ['', '', '', '']], + dtype='|S1'), + 'fileName': 8125252, + 'formula': 0, + 'fsValid': 0, + 'kindBits': '\\x00', + 'modDate': 3001573964, + 'nDim': array([0, 0, 0, 0]), + 'next': 8133140, + 'npnts': 0, + 'sIndices': 0, + 'sfA': array([ 1., 1., 1., 1.]), + 'sfB': array([ 0., 0., 0., 0.]), + 'srcFldr': -1007, + 'swModified': 0, + 'topFullScale': 0.0, + 'type': 2, + 'useBits': '\\x00', + 'wModified': 1, + 'waveNoteH': 0, + 'whUnused': array([0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0]), + 'whVersion': 1, + 'whpad1': array(['', '', '', '', '', ''], + dtype='|S1'), + 'whpad2': 0, + 'whpad3': 0, + 'whpad4': 0}}} + """ + assert_equal_dump_no_whitespace_no_byte(act, exp) diff --git a/test/test_pxp.py b/test/test_pxp.py new file mode 100644 index 0000000..95a663a --- /dev/null +++ b/test/test_pxp.py @@ -0,0 +1,433 @@ +import numpy as np + +from igor.packed import load as loadpxp + +from helper import data_dir + + +def tostr(data): + if isinstance(data, bytes): + data = data.decode("utf-8") + return data + + +def test_pxp(): + data = loadpxp(data_dir / "polar-graphs-demo.pxp") + records = data[0] + assert len(records) == 51 + + assert records[30].variables == { + "version": 1, + "variables": { + "var_header": {"numSysVars": 21, "numUserVars": 0, "numUserStrs": 0}, + "sysVars": { + "K0": 0.0, + "K1": 0.0, + "K2": 0.0, + "K3": 0.0, + "K4": 0.0, + "K5": 0.0, + "K6": 0.0, + "K7": 0.0, + "K8": 0.0, + "K9": 0.0, + "K10": 0.0, + "K11": 0.0, + "K12": 0.0, + "K13": 0.0, + "K14": 0.0, + "K15": 0.0, + "K16": 0.0, + "K17": 0.0, + "K18": 0.0, + "K19": 0.0, + "K20": 128.0, + }, + "userVars": {}, + "userStrs": {}, + }, + } + + wave = data[1]["root"] + wkeys = [tostr(st) for st in wave.keys()] + assert sorted(wkeys) == sorted( + [ + "K0", + "K1", + "K2", + "K3", + "K4", + "K5", + "K6", + "K7", + "K8", + "K9", + "K10", + "K11", + "K12", + "K13", + "K14", + "K15", + "K16", + "K17", + "K18", + "K19", + "K20", + "radiusData", + "angleData", + "W_plrX5", + "W_plrY5", + "angleQ1", + "radiusQ1", + "W_plrX6", + "W_plrY6", + "Packages", + ] + ) + + assert np.allclose( + records[32].wave["wave"]["wData"], + np.array( + [ + 0.3, + 0.5448544, + 0.77480197, + 0.9758435, + 1.1357394, + 1.2447554, + 1.2962544, + 1.287101, + 1.2178528, + 1.0927255, + 0.91933674, + 0.7082426, + 0.47229454, + 0.22585714, + -0.01606643, + -0.23874778, + -0.42862982, + -0.574153, + -0.6664573, + -0.6999235, + -0.6725141, + -0.5858976, + -0.44534767, + -0.25942117, + -0.03943586, + 0.20121357, + 0.44787762, + 0.6855388, + 0.8997279, + 1.0774051, + 1.2077546, + 1.2828392, + 1.2980883, + 1.2525737, + 1.1490659, + 0.99386656, + 0.7964253, + 0.5687607, + 0.32473388, + 0.07920124, + -0.15288824, + -0.35740662, + -0.5219018, + -0.636359, + -0.69381076, + -0.69075894, + -0.62739, + -0.5075599, + -0.3385666, + -0.13069656, + 0.10339352, + 0.34945396, + 0.5925036, + 0.8177455, + 1.0114669, + 1.1618733, + 1.2598093, + 1.2993116, + 1.277976, + 1.1971004, + 1.061609, + 0.8797508, + 0.6625979, + 0.4233691, + 0.17663053, + -0.06259823, + -0.2797519, + -0.46160996, + -0.597101, + -0.6779761, + -0.6993116, + -0.6598092, + -0.56187314, + -0.41146588, + -0.21774435, + 0.00749773, + 0.25054744, + 0.49660596, + 0.7306987, + 0.9385669, + 1.1075606, + 1.2273898, + 1.2907591, + 1.2938106, + 1.2363585, + 1.1219027, + 0.95740634, + 0.7528879, + 0.5207975, + 0.2752648, + 0.03123802, + -0.19642642, + -0.39386547, + -0.54906607, + -0.6525743, + -0.6980884, + -0.682839, + -0.607754, + -0.47740453, + -0.29972947, + -0.08553842, + 0.15212469, + 0.39878684, + 0.6394367, + 0.85942155, + 1.0453486, + 1.1858985, + 1.2725141, + 1.2999234, + 1.2664578, + 1.1741526, + 1.0286293, + 0.83874667, + 0.6160649, + 0.37414294, + 0.12770344, + -0.1082412, + -0.3193372, + -0.49272597, + -0.6178533, + -0.6871013, + -0.6962544, + -0.6447547, + -0.5357403, + -0.37584305, + -0.17479956, + 0.05514668, + 0.30000135, + ], + dtype=np.float32, + ), + ) + + assert np.allclose( + records[33].wave["wave"]["wData"], + np.array( + [ + 0.0, + 0.0494739, + 0.0989478, + 0.1484217, + 0.1978956, + 0.24736951, + 0.29684341, + 0.34631732, + 0.3957912, + 0.44526511, + 0.49473903, + 0.54421294, + 0.59368682, + 0.6431607, + 0.69263464, + 0.74210852, + 0.79158241, + 0.84105635, + 0.89053023, + 0.94000411, + 0.98947805, + 1.03895199, + 1.08842587, + 1.13789964, + 1.18737364, + 1.23684752, + 1.2863214, + 1.3357954, + 1.38526928, + 1.43474305, + 1.48421705, + 1.53369093, + 1.58316481, + 1.63263881, + 1.68211269, + 1.73158658, + 1.78106046, + 1.83053434, + 1.88000822, + 1.92948222, + 1.9789561, + 2.02842999, + 2.07790399, + 2.12737775, + 2.17685175, + 2.22632551, + 2.27579927, + 2.32527351, + 2.37474728, + 2.42422128, + 2.47369504, + 2.52316904, + 2.5726428, + 2.6221168, + 2.67159081, + 2.72106457, + 2.77053857, + 2.82001233, + 2.86948609, + 2.91896009, + 2.9684341, + 3.0179081, + 3.06738186, + 3.11685586, + 3.16632962, + 3.21580338, + 3.26527762, + 3.31475139, + 3.36422539, + 3.41369915, + 3.46317315, + 3.51264691, + 3.56212091, + 3.61159492, + 3.66106868, + 3.71054268, + 3.76001644, + 3.8094902, + 3.85896444, + 3.90843821, + 3.95791221, + 4.00738621, + 4.05685997, + 4.10633373, + 4.15580797, + 4.20528126, + 4.2547555, + 4.30422926, + 4.3537035, + 4.40317726, + 4.45265102, + 4.50212526, + 4.55159855, + 4.60107279, + 4.65054703, + 4.70002079, + 4.74949455, + 4.79896832, + 4.84844255, + 4.89791584, + 4.94739008, + 4.99686432, + 5.04633808, + 5.09581184, + 5.14528561, + 5.19475985, + 5.24423361, + 5.29370737, + 5.34318161, + 5.3926549, + 5.44212914, + 5.4916029, + 5.54107714, + 5.5905509, + 5.64002466, + 5.6894989, + 5.73897219, + 5.78844643, + 5.83792019, + 5.88739443, + 5.93686819, + 5.98634195, + 6.03581619, + 6.08528948, + 6.13476372, + 6.18423796, + 6.23371172, + 6.28318548, + ], + dtype=np.float32, + ), + ) + + assert np.allclose( + records[39].wave["wave"]["wData"], + np.array( + [ + 8.19404411, + 8.88563347, + 9.70543861, + 10.17177773, + 10.11173058, + 9.73756695, + 9.25513077, + 8.8788929, + 9.16085339, + 10.56489944, + 12.75579453, + 14.90572262, + 16.46352959, + 17.33401871, + 17.68511391, + 17.74635315, + 17.70048141, + 17.79942513, + 18.36241531, + 19.38741684, + 20.41767311, + 21.02259827, + 21.09260368, + 20.4905529, + 18.95538521, + 16.9299469, + 15.94969368, + 17.14490509, + 19.78741264, + 22.33615875, + 23.96352196, + 24.04369545, + 21.92454147, + 18.79150391, + 17.77407646, + 20.32803917, + 24.37140465, + 27.24079132, + 28.40307808, + 28.67787933, + 28.70550728, + 28.50283432, + 27.68538666, + 26.36607552, + 25.73583984, + 26.78374672, + 28.8236084, + 30.36226463, + 30.91939545, + 31.22146797, + 31.97431755, + 32.95656204, + 33.4611969, + 33.23248672, + 32.3250885, + 30.64473915, + 28.72983551, + 28.05199242, + 29.29024887, + 31.3501091, + 32.7331543, + 32.87995529, + 32.28799438, + 31.99738503, + ], + dtype=np.float32, + ), + )