1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
|
%global _empty_manifest_terminate_build 0
Name: python-pyuvdata
Version: 2.3.2
Release: 1
Summary: an interface for astronomical interferometeric datasets in python
License: BSD
URL: https://github.com/RadioAstronomySoftwareGroup/pyuvdata
Source0: https://mirrors.nju.edu.cn/pypi/web/packages/d2/b1/284acc6752b9b2f1eadddd469f384a0385610cb78379097b78567215558f/pyuvdata-2.3.2.tar.gz
Requires: python3-astropy
Requires: python3-h5py
Requires: python3-numpy
Requires: python3-pyerfa
Requires: python3-scipy
Requires: python3-setuptools-scm
Requires: python3-astroquery
Requires: python3-casacore
Requires: python3-pyyaml
Requires: python3-hdf5plugin
Requires: python3-astropy-healpix
Requires: python3-lunarsky
Requires: python3-novas
Requires: python3-novas-de405
Requires: python3-astroquery
Requires: python3-casacore
Requires: python3-pyyaml
Requires: python3-astroquery
Requires: python3-casacore
Requires: python3-pyyaml
Requires: python3-hdf5plugin
Requires: python3-astropy-healpix
Requires: python3-lunarsky
Requires: python3-novas
Requires: python3-novas-de405
Requires: python3-pytest
Requires: python3-pytest-xdist
Requires: python3-pytest-cases
Requires: python3-pytest-cov
Requires: python3-cython
Requires: python3-coverage
Requires: python3-pre-commit
Requires: python3-sphinx
Requires: python3-pypandoc
Requires: python3-sphinx
Requires: python3-pypandoc
Requires: python3-hdf5plugin
Requires: python3-astropy-healpix
Requires: python3-lunarsky
Requires: python3-novas
Requires: python3-novas-de405
Requires: python3-astroquery
Requires: python3-casacore
Requires: python3-pyyaml
Requires: python3-hdf5plugin
Requires: python3-astropy-healpix
Requires: python3-lunarsky
Requires: python3-novas
Requires: python3-novas-de405
Requires: python3-pytest
Requires: python3-pytest-xdist
Requires: python3-pytest-cases
Requires: python3-pytest-cov
Requires: python3-cython
Requires: python3-coverage
Requires: python3-pre-commit
%description
# pyuvdata
[](https://circleci.com/gh/RadioAstronomySoftwareGroup/pyuvdata?branch=main)
[](https://github.com/RadioAstronomySoftwareGroup/pyuvdata/actions)
[](https://codecov.io/gh/RadioAstronomySoftwareGroup/pyuvdata)
[](https://doi.org/10.21105/joss.00140)
pyuvdata defines a pythonic interface to interferometric data sets.
Currently pyuvdata supports reading and writing of miriad, uvfits, CASA measurement sets
and uvh5 files and reading of FHD
([Fast Holographic Deconvolution](https://github.com/EoRImaging/FHD)) visibility save files, SMA Mir files and MWA correlator FITS files.
API documentation and a tutorial is hosted on [ReadTheDocs](https://pyuvdata.readthedocs.io).
# Motivation
The main goals are:
1. To provide a high quality, well documented path to convert between data formats
2. Support the direct use of datasets from python with minimal software
3. Provide precise data definition via both human readable code and high quality online documentation
# Package Details
pyuvdata has four major user classes:
* UVData: supports interferometric data (visibilities) and associated metadata
* UVCal: supports interferometric calibration solutions (antenna-based) and
associated metadata (Note that this is a fairly new object, consider it to be a beta version)
* UVBeam: supports primary beams (E-field or power) and associated metadata
(Note that this is a fairly new object, consider it to be a beta version)
* UVFlag: A class to handle the manipulation and combination of flags for data sets.
Also can convert raw data quality metrics into flags using thresholding.
(This object is very new and experimental. Consider it to be a beta version)
## UVData File standard notes
* miriad has been thoroughly tested with aipy-style miriad files and minimally
tested with ATCA files
* uvfits conforms to AIPS memo 117 (as of March 2020). It is tested against
FHD, CASA, and AIPS. However AIPS is limited to <80 antennas and CASA uvfits
import does not seem to support >255 antennas. Users with data sets containing > 255
antennas should use the measurement set writer instead.
* CASA measurement sets, primarily conforming to [CASA Memo 229](https://casa.nrao.edu/Memos/229.html), with some elements taken from the proposed v3.0 format documented in [CASA Memo 264](https://casacore.github.io/casacore-notes/264.html). Measurement sets are tested against
VLA and MWA data sets, (the latter filled via cotter), with some manual verification
haven been performed against ALMA and SMA data sets, the latter filled using the `importuvfits` task of CASA.
tested against ALMA-filled datasets and with SMA datasets
* uvh5 is an HDF5-based file format defined by the HERA collaboration,
details in the [uvh5 memo](docs/references/uvh5_memo.pdf). Note that this is a
somewhat new format, so it may evolve a bit
but we will strive to make future versions backwards compatible with the current format.
It is probably not compatible with other interferometric HDF5 files defined by other groups.
* FHD (read-only support, tested against MWA and PAPER data)
* MIR (read-only support, though experimental write functions are available, tested against SMA data)
* MWA correlator FITS files (read-only support, tested against Cotter outputs and FHD)
## UVCal file formats
* calfits: a new format defined in pyuvdata, details in the [calfits_memo](docs/references/calfits_memo.pdf).
Note that this format was recently defined and may change in coming versions,
based on user needs. Consider it to be a beta version, but we will strive to
make future versions backwards compatible with the current format.
* FHD calibration files (read-only support)
## UVBeam file formats
* regularly gridded fits for both E-field and power beams
* non-standard HEALPix fits for both E-field and power beams (in an ImageHDU
rather than a binary table to support frequency, polarization and E-field vector axes)
* read support for CST beam text files, with a defined yaml file format for
metadata, details here: [cst settings file](docs/cst_settings_yaml.rst)
## Under Development
* UVCal: object and calfits file format (beta version)
* UVBeam: object and beamfits file format (beta version)
* UVFlag: object and HDF5 file format. (beta version)
* Mir: object (part of UVData class) (beta version)
* MirParser: object and python interface for MIR file format (beta version)
## Known Issues and Planned Improvements
* UVBeam: support phased-array antenna beams (e.g. MWA beams).
* UVFlag: Adding requires a high level knowledge of individual objects. (see [issue #653](https://github.com/RadioAstronomySoftwareGroup/pyuvdata/issues/653))
For details see the [issue log](https://github.com/RadioAstronomySoftwareGroup/pyuvdata/issues).
## Community Guidelines
Contributions to this package to add new file formats or address any of the
issues in the [issue log](https://github.com/RadioAstronomySoftwareGroup/pyuvdata/issues)
are very welcome, as are bug reports and feature requests.
Please see our [guide on contributing](.github/CONTRIBUTING.md)
# Telescopes
pyuvdata has been used with data from the following telescopes. If you use it on
data from a telescope we don't have listed here please let us know how it went
via an issue! We would like to make pyuvdata generally useful to the community for
as many telescopes as possible.
* MWA
* HERA
* PAPER
* LWA
* ALMA
* VLA
* ATCA
* SMA
# Versioning
We use a `generation.major.minor` version number format. We use the `generation`
number for very significant improvements or major rewrites, the `major` number
to indicate substantial package changes (intended to be released every 3-4 months)
and the `minor` number to release smaller incremental updates (intended to be
released approximately monthly and which usually do not include breaking API
changes). We do our best to provide a significant period (usually 2 major
generations) of deprecation warnings for all breaking changes to the API.
We track all changes in our [changelog](https://github.com/RadioAstronomySoftwareGroup/pyuvdata/blob/main/CHANGELOG.md).
# History
pyuvdata was originally developed in the low frequency 21cm community to support
the development of and interchange of data between calibration and foreground
subtraction pipelines. Particular focus has been paid to supporting drift and
phased array modes.
# Citation
Please cite pyuvdata by citing our JOSS paper:
Hazelton et al, (2017), pyuvdata: an interface for astronomical interferometeric
datasets in python, Journal of Open Source Software, 2(10), 140, doi:10.21105/joss.00140
[ADS Link](https://ui.adsabs.harvard.edu/abs/2017JOSS....2..140H);
[Bibtex entry](http://adsabs.harvard.edu/cgi-bin/nph-bib_query?bibcode=2017JOSS....2..140H&data_type=BIBTEX&db_key=GEN&nocookieset=1)
# Installation
Simple installation via conda is available for users, developers should follow
the directions under [Developer Installation](#developer-installation) below.
For simple installation, the latest stable version is available via conda
(preferred: ```conda install -c conda-forge pyuvdata```) or pip (```pip install pyuvdata```).
There are some optional dependencies that are required for specific functionality,
which will not be installed automatically by conda or pip.
See [Dependencies](#dependencies) for details on installing optional dependencies.
Note that as of v2.2, `pyuvdata` is only supported on python 3.7+.
## Dependencies
Required:
* astropy >= 5.0.4
* h5py >= 3.1
* numpy >= 1.20
* pyerfa >= 2.0
* scipy >= 1.5
* setuptools_scm <7.0|>=7.0.3
Optional:
* astropy-healpix >= 0.6 (for working with beams in HEALPix formats)
* astroquery >= 0.4.4 (for enabling phasing to ephemeris objects using JPL-Horizons)
* hdf5plugin >= 3.1.0 (for enabling bitshuffle and other hdf5 compression filters in uvh5 files)
* lunarsky >=0.2.1 (for working with simulated datasets for lunar telescopes)
* novas and novas_de405 (for using the NOVAS library for astrometry)
* python-casacore >= 3.3.1 (for working with CASA measurement sets)
* pyyaml >= 5.1 (for working with settings files for CST beam files)
The numpy and astropy versions are important, so make sure these are up to date.
We suggest using conda to install all the dependencies. If you want to install
python-casacore and astropy-healpix, you'll need to add conda-forge as a channel
(```conda config --add channels conda-forge```).
If you do not want to use conda, the packages are also available on PyPI
(python-casacore may require more effort, see details for that package below).
You can install the optional dependencies via pip by specifying an option
when you install pyuvdata, as in ```pip install pyuvdata[healpix]```
which will install all the required packages for using the HEALPix functionality
in pyuvdata. The options that can be passed in this way are:
[`astroquery`, `casa`, `cst`, `hdf5_compression`, `healpix`, `lunar`, `novas`, `all`, `test`,
`doc`, `dev`]. The first set (`astroquery`, `casa`, `cst`, `hdf5_compression`,
`healpix`, `lunar`, `novas`) enable various specific functionality while `all` will install all
optional dependencies. The last three (`test`, `doc`, `dev`) may be useful for developers
of pyuvdata.
### Installing python-casacore
python-casacore requires the casacore c++ libraries. It can be installed easily
using conda (```python-casacore``` on conda-forge).
If you do not want to use conda, the casacore c++ libraries are available for
ubuntu through the [kern suite](http://kernsuite.info/). On OSX, casacore is
available through the [ska-sa brew tap](https://github.com/ska-sa/homebrew-tap).
The python-casacore library (with manual install instructions) is available at
https://github.com/casacore/python-casacore
## Developer Installation
Clone the repository using
```git clone https://github.com/RadioAstronomySoftwareGroup/pyuvdata.git```
Navigate into the pyuvdata directory and run `pip install .`
(note that `python setup.py install` does not work).
Note that this will attempt to automatically install any missing dependencies.
If you use anaconda or another package manager you might prefer to first install
the dependencies as described in [Dependencies](#dependencies).
To install without dependencies, run `pip install --no-deps .`
To compile the binary extension modules such that you can successfully run
`import pyuvdata` from the top-level directory of your Git checkout, run:
```python setup.py build_ext --inplace```
If you want to do development on pyuvdata, in addition to the other dependencies
you will need the following packages:
* pytest >= 6.2
* pytest-cases >= 3.6.9
* pytest-cov
* cython >=0.23 (This is necessary for coverage reporting of cython extensions)
* coverage
* pre-commit
* sphinx
* pypandoc
One other package, pytest-xdist, is not required, but can be used to speed up running
the test suite by running tests in parallel. To use it call pytest with the
```-n auto``` option.
One way to ensure you have all the needed packages is to use the included
`environment.yaml` file to create a new environment that will
contain all the optional dependencies along with dependencies required for
testing and development (```conda env create -f environment.yaml```).
Alternatively, you can specify `test`, `doc`, or `dev` when installing pyuvdata
(as in `pip install pyuvdata[dev]`) to install the packages needed for testing
(including coverage and linting) and documentation development;
`dev` includes everything in `test` and `doc`.
To use pre-commit to prevent committing code that does not follow our style, you'll
need to run `pre-commit install` in the top level `pyuvdata` directory.
## Tests
Uses the `pytest` package to execute test suite.
From the source pyuvdata directory run ```pytest``` or ```python -m pytest```.
Testing of `UVFlag` module requires the `pytest-cases` plug-in.
# API
The primary interface to data from python is via the UVData object. It provides
import functionality from all supported file formats (UVFITS, Miriad, UVH5, FHD,
CASA measurement sets, SMA Mir, MWA correlator FITS) and export to UVFITS, Miriad,
CASA measurement sets and UVH5 formats and can
be interacted with directly. Similarly, the primary calibration, beam, and flag
interfaces are via the UVCal, UVBeam, and UVflag objects. The attributes of the UVData,
UVCal, UVBeam, and UVFlag objects are described in the UVData Parameters, UVCal Parameters,
UVBeam Parameters and UVFlag Parameters pages on [ReadTheDocs](https://pyuvdata.readthedocs.io).
# Maintainers
pyuvdata is maintained by the RASG Managers, which currently include:
- Adam Beardsley (Winona State University)
- Bryna Hazelton (University of Washington)
- Daniel Jacobs (Arizona State University)
- Matt Kolopanis (Arizona State University)
- Paul La Plante (University of Nevada, Las Vegas)
- Jonathan Pober (Brown University)
Please use the channels discussed in the [guide on contributing](.github/CONTRIBUTING.md)
for code-related discussions. You can contact us privately if needed at
[rasgmanagers@gmail.com](mailto:rasgmanagers@gmail.com).
# Acknowledgments
Support for pyuvdata was provided by NSF awards #1835421 and #1835120.
%package -n python3-pyuvdata
Summary: an interface for astronomical interferometeric datasets in python
Provides: python-pyuvdata
BuildRequires: python3-devel
BuildRequires: python3-setuptools
BuildRequires: python3-pip
BuildRequires: python3-cffi
BuildRequires: gcc
BuildRequires: gdb
%description -n python3-pyuvdata
# pyuvdata
[](https://circleci.com/gh/RadioAstronomySoftwareGroup/pyuvdata?branch=main)
[](https://github.com/RadioAstronomySoftwareGroup/pyuvdata/actions)
[](https://codecov.io/gh/RadioAstronomySoftwareGroup/pyuvdata)
[](https://doi.org/10.21105/joss.00140)
pyuvdata defines a pythonic interface to interferometric data sets.
Currently pyuvdata supports reading and writing of miriad, uvfits, CASA measurement sets
and uvh5 files and reading of FHD
([Fast Holographic Deconvolution](https://github.com/EoRImaging/FHD)) visibility save files, SMA Mir files and MWA correlator FITS files.
API documentation and a tutorial is hosted on [ReadTheDocs](https://pyuvdata.readthedocs.io).
# Motivation
The main goals are:
1. To provide a high quality, well documented path to convert between data formats
2. Support the direct use of datasets from python with minimal software
3. Provide precise data definition via both human readable code and high quality online documentation
# Package Details
pyuvdata has four major user classes:
* UVData: supports interferometric data (visibilities) and associated metadata
* UVCal: supports interferometric calibration solutions (antenna-based) and
associated metadata (Note that this is a fairly new object, consider it to be a beta version)
* UVBeam: supports primary beams (E-field or power) and associated metadata
(Note that this is a fairly new object, consider it to be a beta version)
* UVFlag: A class to handle the manipulation and combination of flags for data sets.
Also can convert raw data quality metrics into flags using thresholding.
(This object is very new and experimental. Consider it to be a beta version)
## UVData File standard notes
* miriad has been thoroughly tested with aipy-style miriad files and minimally
tested with ATCA files
* uvfits conforms to AIPS memo 117 (as of March 2020). It is tested against
FHD, CASA, and AIPS. However AIPS is limited to <80 antennas and CASA uvfits
import does not seem to support >255 antennas. Users with data sets containing > 255
antennas should use the measurement set writer instead.
* CASA measurement sets, primarily conforming to [CASA Memo 229](https://casa.nrao.edu/Memos/229.html), with some elements taken from the proposed v3.0 format documented in [CASA Memo 264](https://casacore.github.io/casacore-notes/264.html). Measurement sets are tested against
VLA and MWA data sets, (the latter filled via cotter), with some manual verification
haven been performed against ALMA and SMA data sets, the latter filled using the `importuvfits` task of CASA.
tested against ALMA-filled datasets and with SMA datasets
* uvh5 is an HDF5-based file format defined by the HERA collaboration,
details in the [uvh5 memo](docs/references/uvh5_memo.pdf). Note that this is a
somewhat new format, so it may evolve a bit
but we will strive to make future versions backwards compatible with the current format.
It is probably not compatible with other interferometric HDF5 files defined by other groups.
* FHD (read-only support, tested against MWA and PAPER data)
* MIR (read-only support, though experimental write functions are available, tested against SMA data)
* MWA correlator FITS files (read-only support, tested against Cotter outputs and FHD)
## UVCal file formats
* calfits: a new format defined in pyuvdata, details in the [calfits_memo](docs/references/calfits_memo.pdf).
Note that this format was recently defined and may change in coming versions,
based on user needs. Consider it to be a beta version, but we will strive to
make future versions backwards compatible with the current format.
* FHD calibration files (read-only support)
## UVBeam file formats
* regularly gridded fits for both E-field and power beams
* non-standard HEALPix fits for both E-field and power beams (in an ImageHDU
rather than a binary table to support frequency, polarization and E-field vector axes)
* read support for CST beam text files, with a defined yaml file format for
metadata, details here: [cst settings file](docs/cst_settings_yaml.rst)
## Under Development
* UVCal: object and calfits file format (beta version)
* UVBeam: object and beamfits file format (beta version)
* UVFlag: object and HDF5 file format. (beta version)
* Mir: object (part of UVData class) (beta version)
* MirParser: object and python interface for MIR file format (beta version)
## Known Issues and Planned Improvements
* UVBeam: support phased-array antenna beams (e.g. MWA beams).
* UVFlag: Adding requires a high level knowledge of individual objects. (see [issue #653](https://github.com/RadioAstronomySoftwareGroup/pyuvdata/issues/653))
For details see the [issue log](https://github.com/RadioAstronomySoftwareGroup/pyuvdata/issues).
## Community Guidelines
Contributions to this package to add new file formats or address any of the
issues in the [issue log](https://github.com/RadioAstronomySoftwareGroup/pyuvdata/issues)
are very welcome, as are bug reports and feature requests.
Please see our [guide on contributing](.github/CONTRIBUTING.md)
# Telescopes
pyuvdata has been used with data from the following telescopes. If you use it on
data from a telescope we don't have listed here please let us know how it went
via an issue! We would like to make pyuvdata generally useful to the community for
as many telescopes as possible.
* MWA
* HERA
* PAPER
* LWA
* ALMA
* VLA
* ATCA
* SMA
# Versioning
We use a `generation.major.minor` version number format. We use the `generation`
number for very significant improvements or major rewrites, the `major` number
to indicate substantial package changes (intended to be released every 3-4 months)
and the `minor` number to release smaller incremental updates (intended to be
released approximately monthly and which usually do not include breaking API
changes). We do our best to provide a significant period (usually 2 major
generations) of deprecation warnings for all breaking changes to the API.
We track all changes in our [changelog](https://github.com/RadioAstronomySoftwareGroup/pyuvdata/blob/main/CHANGELOG.md).
# History
pyuvdata was originally developed in the low frequency 21cm community to support
the development of and interchange of data between calibration and foreground
subtraction pipelines. Particular focus has been paid to supporting drift and
phased array modes.
# Citation
Please cite pyuvdata by citing our JOSS paper:
Hazelton et al, (2017), pyuvdata: an interface for astronomical interferometeric
datasets in python, Journal of Open Source Software, 2(10), 140, doi:10.21105/joss.00140
[ADS Link](https://ui.adsabs.harvard.edu/abs/2017JOSS....2..140H);
[Bibtex entry](http://adsabs.harvard.edu/cgi-bin/nph-bib_query?bibcode=2017JOSS....2..140H&data_type=BIBTEX&db_key=GEN&nocookieset=1)
# Installation
Simple installation via conda is available for users, developers should follow
the directions under [Developer Installation](#developer-installation) below.
For simple installation, the latest stable version is available via conda
(preferred: ```conda install -c conda-forge pyuvdata```) or pip (```pip install pyuvdata```).
There are some optional dependencies that are required for specific functionality,
which will not be installed automatically by conda or pip.
See [Dependencies](#dependencies) for details on installing optional dependencies.
Note that as of v2.2, `pyuvdata` is only supported on python 3.7+.
## Dependencies
Required:
* astropy >= 5.0.4
* h5py >= 3.1
* numpy >= 1.20
* pyerfa >= 2.0
* scipy >= 1.5
* setuptools_scm <7.0|>=7.0.3
Optional:
* astropy-healpix >= 0.6 (for working with beams in HEALPix formats)
* astroquery >= 0.4.4 (for enabling phasing to ephemeris objects using JPL-Horizons)
* hdf5plugin >= 3.1.0 (for enabling bitshuffle and other hdf5 compression filters in uvh5 files)
* lunarsky >=0.2.1 (for working with simulated datasets for lunar telescopes)
* novas and novas_de405 (for using the NOVAS library for astrometry)
* python-casacore >= 3.3.1 (for working with CASA measurement sets)
* pyyaml >= 5.1 (for working with settings files for CST beam files)
The numpy and astropy versions are important, so make sure these are up to date.
We suggest using conda to install all the dependencies. If you want to install
python-casacore and astropy-healpix, you'll need to add conda-forge as a channel
(```conda config --add channels conda-forge```).
If you do not want to use conda, the packages are also available on PyPI
(python-casacore may require more effort, see details for that package below).
You can install the optional dependencies via pip by specifying an option
when you install pyuvdata, as in ```pip install pyuvdata[healpix]```
which will install all the required packages for using the HEALPix functionality
in pyuvdata. The options that can be passed in this way are:
[`astroquery`, `casa`, `cst`, `hdf5_compression`, `healpix`, `lunar`, `novas`, `all`, `test`,
`doc`, `dev`]. The first set (`astroquery`, `casa`, `cst`, `hdf5_compression`,
`healpix`, `lunar`, `novas`) enable various specific functionality while `all` will install all
optional dependencies. The last three (`test`, `doc`, `dev`) may be useful for developers
of pyuvdata.
### Installing python-casacore
python-casacore requires the casacore c++ libraries. It can be installed easily
using conda (```python-casacore``` on conda-forge).
If you do not want to use conda, the casacore c++ libraries are available for
ubuntu through the [kern suite](http://kernsuite.info/). On OSX, casacore is
available through the [ska-sa brew tap](https://github.com/ska-sa/homebrew-tap).
The python-casacore library (with manual install instructions) is available at
https://github.com/casacore/python-casacore
## Developer Installation
Clone the repository using
```git clone https://github.com/RadioAstronomySoftwareGroup/pyuvdata.git```
Navigate into the pyuvdata directory and run `pip install .`
(note that `python setup.py install` does not work).
Note that this will attempt to automatically install any missing dependencies.
If you use anaconda or another package manager you might prefer to first install
the dependencies as described in [Dependencies](#dependencies).
To install without dependencies, run `pip install --no-deps .`
To compile the binary extension modules such that you can successfully run
`import pyuvdata` from the top-level directory of your Git checkout, run:
```python setup.py build_ext --inplace```
If you want to do development on pyuvdata, in addition to the other dependencies
you will need the following packages:
* pytest >= 6.2
* pytest-cases >= 3.6.9
* pytest-cov
* cython >=0.23 (This is necessary for coverage reporting of cython extensions)
* coverage
* pre-commit
* sphinx
* pypandoc
One other package, pytest-xdist, is not required, but can be used to speed up running
the test suite by running tests in parallel. To use it call pytest with the
```-n auto``` option.
One way to ensure you have all the needed packages is to use the included
`environment.yaml` file to create a new environment that will
contain all the optional dependencies along with dependencies required for
testing and development (```conda env create -f environment.yaml```).
Alternatively, you can specify `test`, `doc`, or `dev` when installing pyuvdata
(as in `pip install pyuvdata[dev]`) to install the packages needed for testing
(including coverage and linting) and documentation development;
`dev` includes everything in `test` and `doc`.
To use pre-commit to prevent committing code that does not follow our style, you'll
need to run `pre-commit install` in the top level `pyuvdata` directory.
## Tests
Uses the `pytest` package to execute test suite.
From the source pyuvdata directory run ```pytest``` or ```python -m pytest```.
Testing of `UVFlag` module requires the `pytest-cases` plug-in.
# API
The primary interface to data from python is via the UVData object. It provides
import functionality from all supported file formats (UVFITS, Miriad, UVH5, FHD,
CASA measurement sets, SMA Mir, MWA correlator FITS) and export to UVFITS, Miriad,
CASA measurement sets and UVH5 formats and can
be interacted with directly. Similarly, the primary calibration, beam, and flag
interfaces are via the UVCal, UVBeam, and UVflag objects. The attributes of the UVData,
UVCal, UVBeam, and UVFlag objects are described in the UVData Parameters, UVCal Parameters,
UVBeam Parameters and UVFlag Parameters pages on [ReadTheDocs](https://pyuvdata.readthedocs.io).
# Maintainers
pyuvdata is maintained by the RASG Managers, which currently include:
- Adam Beardsley (Winona State University)
- Bryna Hazelton (University of Washington)
- Daniel Jacobs (Arizona State University)
- Matt Kolopanis (Arizona State University)
- Paul La Plante (University of Nevada, Las Vegas)
- Jonathan Pober (Brown University)
Please use the channels discussed in the [guide on contributing](.github/CONTRIBUTING.md)
for code-related discussions. You can contact us privately if needed at
[rasgmanagers@gmail.com](mailto:rasgmanagers@gmail.com).
# Acknowledgments
Support for pyuvdata was provided by NSF awards #1835421 and #1835120.
%package help
Summary: Development documents and examples for pyuvdata
Provides: python3-pyuvdata-doc
%description help
# pyuvdata
[](https://circleci.com/gh/RadioAstronomySoftwareGroup/pyuvdata?branch=main)
[](https://github.com/RadioAstronomySoftwareGroup/pyuvdata/actions)
[](https://codecov.io/gh/RadioAstronomySoftwareGroup/pyuvdata)
[](https://doi.org/10.21105/joss.00140)
pyuvdata defines a pythonic interface to interferometric data sets.
Currently pyuvdata supports reading and writing of miriad, uvfits, CASA measurement sets
and uvh5 files and reading of FHD
([Fast Holographic Deconvolution](https://github.com/EoRImaging/FHD)) visibility save files, SMA Mir files and MWA correlator FITS files.
API documentation and a tutorial is hosted on [ReadTheDocs](https://pyuvdata.readthedocs.io).
# Motivation
The main goals are:
1. To provide a high quality, well documented path to convert between data formats
2. Support the direct use of datasets from python with minimal software
3. Provide precise data definition via both human readable code and high quality online documentation
# Package Details
pyuvdata has four major user classes:
* UVData: supports interferometric data (visibilities) and associated metadata
* UVCal: supports interferometric calibration solutions (antenna-based) and
associated metadata (Note that this is a fairly new object, consider it to be a beta version)
* UVBeam: supports primary beams (E-field or power) and associated metadata
(Note that this is a fairly new object, consider it to be a beta version)
* UVFlag: A class to handle the manipulation and combination of flags for data sets.
Also can convert raw data quality metrics into flags using thresholding.
(This object is very new and experimental. Consider it to be a beta version)
## UVData File standard notes
* miriad has been thoroughly tested with aipy-style miriad files and minimally
tested with ATCA files
* uvfits conforms to AIPS memo 117 (as of March 2020). It is tested against
FHD, CASA, and AIPS. However AIPS is limited to <80 antennas and CASA uvfits
import does not seem to support >255 antennas. Users with data sets containing > 255
antennas should use the measurement set writer instead.
* CASA measurement sets, primarily conforming to [CASA Memo 229](https://casa.nrao.edu/Memos/229.html), with some elements taken from the proposed v3.0 format documented in [CASA Memo 264](https://casacore.github.io/casacore-notes/264.html). Measurement sets are tested against
VLA and MWA data sets, (the latter filled via cotter), with some manual verification
haven been performed against ALMA and SMA data sets, the latter filled using the `importuvfits` task of CASA.
tested against ALMA-filled datasets and with SMA datasets
* uvh5 is an HDF5-based file format defined by the HERA collaboration,
details in the [uvh5 memo](docs/references/uvh5_memo.pdf). Note that this is a
somewhat new format, so it may evolve a bit
but we will strive to make future versions backwards compatible with the current format.
It is probably not compatible with other interferometric HDF5 files defined by other groups.
* FHD (read-only support, tested against MWA and PAPER data)
* MIR (read-only support, though experimental write functions are available, tested against SMA data)
* MWA correlator FITS files (read-only support, tested against Cotter outputs and FHD)
## UVCal file formats
* calfits: a new format defined in pyuvdata, details in the [calfits_memo](docs/references/calfits_memo.pdf).
Note that this format was recently defined and may change in coming versions,
based on user needs. Consider it to be a beta version, but we will strive to
make future versions backwards compatible with the current format.
* FHD calibration files (read-only support)
## UVBeam file formats
* regularly gridded fits for both E-field and power beams
* non-standard HEALPix fits for both E-field and power beams (in an ImageHDU
rather than a binary table to support frequency, polarization and E-field vector axes)
* read support for CST beam text files, with a defined yaml file format for
metadata, details here: [cst settings file](docs/cst_settings_yaml.rst)
## Under Development
* UVCal: object and calfits file format (beta version)
* UVBeam: object and beamfits file format (beta version)
* UVFlag: object and HDF5 file format. (beta version)
* Mir: object (part of UVData class) (beta version)
* MirParser: object and python interface for MIR file format (beta version)
## Known Issues and Planned Improvements
* UVBeam: support phased-array antenna beams (e.g. MWA beams).
* UVFlag: Adding requires a high level knowledge of individual objects. (see [issue #653](https://github.com/RadioAstronomySoftwareGroup/pyuvdata/issues/653))
For details see the [issue log](https://github.com/RadioAstronomySoftwareGroup/pyuvdata/issues).
## Community Guidelines
Contributions to this package to add new file formats or address any of the
issues in the [issue log](https://github.com/RadioAstronomySoftwareGroup/pyuvdata/issues)
are very welcome, as are bug reports and feature requests.
Please see our [guide on contributing](.github/CONTRIBUTING.md)
# Telescopes
pyuvdata has been used with data from the following telescopes. If you use it on
data from a telescope we don't have listed here please let us know how it went
via an issue! We would like to make pyuvdata generally useful to the community for
as many telescopes as possible.
* MWA
* HERA
* PAPER
* LWA
* ALMA
* VLA
* ATCA
* SMA
# Versioning
We use a `generation.major.minor` version number format. We use the `generation`
number for very significant improvements or major rewrites, the `major` number
to indicate substantial package changes (intended to be released every 3-4 months)
and the `minor` number to release smaller incremental updates (intended to be
released approximately monthly and which usually do not include breaking API
changes). We do our best to provide a significant period (usually 2 major
generations) of deprecation warnings for all breaking changes to the API.
We track all changes in our [changelog](https://github.com/RadioAstronomySoftwareGroup/pyuvdata/blob/main/CHANGELOG.md).
# History
pyuvdata was originally developed in the low frequency 21cm community to support
the development of and interchange of data between calibration and foreground
subtraction pipelines. Particular focus has been paid to supporting drift and
phased array modes.
# Citation
Please cite pyuvdata by citing our JOSS paper:
Hazelton et al, (2017), pyuvdata: an interface for astronomical interferometeric
datasets in python, Journal of Open Source Software, 2(10), 140, doi:10.21105/joss.00140
[ADS Link](https://ui.adsabs.harvard.edu/abs/2017JOSS....2..140H);
[Bibtex entry](http://adsabs.harvard.edu/cgi-bin/nph-bib_query?bibcode=2017JOSS....2..140H&data_type=BIBTEX&db_key=GEN&nocookieset=1)
# Installation
Simple installation via conda is available for users, developers should follow
the directions under [Developer Installation](#developer-installation) below.
For simple installation, the latest stable version is available via conda
(preferred: ```conda install -c conda-forge pyuvdata```) or pip (```pip install pyuvdata```).
There are some optional dependencies that are required for specific functionality,
which will not be installed automatically by conda or pip.
See [Dependencies](#dependencies) for details on installing optional dependencies.
Note that as of v2.2, `pyuvdata` is only supported on python 3.7+.
## Dependencies
Required:
* astropy >= 5.0.4
* h5py >= 3.1
* numpy >= 1.20
* pyerfa >= 2.0
* scipy >= 1.5
* setuptools_scm <7.0|>=7.0.3
Optional:
* astropy-healpix >= 0.6 (for working with beams in HEALPix formats)
* astroquery >= 0.4.4 (for enabling phasing to ephemeris objects using JPL-Horizons)
* hdf5plugin >= 3.1.0 (for enabling bitshuffle and other hdf5 compression filters in uvh5 files)
* lunarsky >=0.2.1 (for working with simulated datasets for lunar telescopes)
* novas and novas_de405 (for using the NOVAS library for astrometry)
* python-casacore >= 3.3.1 (for working with CASA measurement sets)
* pyyaml >= 5.1 (for working with settings files for CST beam files)
The numpy and astropy versions are important, so make sure these are up to date.
We suggest using conda to install all the dependencies. If you want to install
python-casacore and astropy-healpix, you'll need to add conda-forge as a channel
(```conda config --add channels conda-forge```).
If you do not want to use conda, the packages are also available on PyPI
(python-casacore may require more effort, see details for that package below).
You can install the optional dependencies via pip by specifying an option
when you install pyuvdata, as in ```pip install pyuvdata[healpix]```
which will install all the required packages for using the HEALPix functionality
in pyuvdata. The options that can be passed in this way are:
[`astroquery`, `casa`, `cst`, `hdf5_compression`, `healpix`, `lunar`, `novas`, `all`, `test`,
`doc`, `dev`]. The first set (`astroquery`, `casa`, `cst`, `hdf5_compression`,
`healpix`, `lunar`, `novas`) enable various specific functionality while `all` will install all
optional dependencies. The last three (`test`, `doc`, `dev`) may be useful for developers
of pyuvdata.
### Installing python-casacore
python-casacore requires the casacore c++ libraries. It can be installed easily
using conda (```python-casacore``` on conda-forge).
If you do not want to use conda, the casacore c++ libraries are available for
ubuntu through the [kern suite](http://kernsuite.info/). On OSX, casacore is
available through the [ska-sa brew tap](https://github.com/ska-sa/homebrew-tap).
The python-casacore library (with manual install instructions) is available at
https://github.com/casacore/python-casacore
## Developer Installation
Clone the repository using
```git clone https://github.com/RadioAstronomySoftwareGroup/pyuvdata.git```
Navigate into the pyuvdata directory and run `pip install .`
(note that `python setup.py install` does not work).
Note that this will attempt to automatically install any missing dependencies.
If you use anaconda or another package manager you might prefer to first install
the dependencies as described in [Dependencies](#dependencies).
To install without dependencies, run `pip install --no-deps .`
To compile the binary extension modules such that you can successfully run
`import pyuvdata` from the top-level directory of your Git checkout, run:
```python setup.py build_ext --inplace```
If you want to do development on pyuvdata, in addition to the other dependencies
you will need the following packages:
* pytest >= 6.2
* pytest-cases >= 3.6.9
* pytest-cov
* cython >=0.23 (This is necessary for coverage reporting of cython extensions)
* coverage
* pre-commit
* sphinx
* pypandoc
One other package, pytest-xdist, is not required, but can be used to speed up running
the test suite by running tests in parallel. To use it call pytest with the
```-n auto``` option.
One way to ensure you have all the needed packages is to use the included
`environment.yaml` file to create a new environment that will
contain all the optional dependencies along with dependencies required for
testing and development (```conda env create -f environment.yaml```).
Alternatively, you can specify `test`, `doc`, or `dev` when installing pyuvdata
(as in `pip install pyuvdata[dev]`) to install the packages needed for testing
(including coverage and linting) and documentation development;
`dev` includes everything in `test` and `doc`.
To use pre-commit to prevent committing code that does not follow our style, you'll
need to run `pre-commit install` in the top level `pyuvdata` directory.
## Tests
Uses the `pytest` package to execute test suite.
From the source pyuvdata directory run ```pytest``` or ```python -m pytest```.
Testing of `UVFlag` module requires the `pytest-cases` plug-in.
# API
The primary interface to data from python is via the UVData object. It provides
import functionality from all supported file formats (UVFITS, Miriad, UVH5, FHD,
CASA measurement sets, SMA Mir, MWA correlator FITS) and export to UVFITS, Miriad,
CASA measurement sets and UVH5 formats and can
be interacted with directly. Similarly, the primary calibration, beam, and flag
interfaces are via the UVCal, UVBeam, and UVflag objects. The attributes of the UVData,
UVCal, UVBeam, and UVFlag objects are described in the UVData Parameters, UVCal Parameters,
UVBeam Parameters and UVFlag Parameters pages on [ReadTheDocs](https://pyuvdata.readthedocs.io).
# Maintainers
pyuvdata is maintained by the RASG Managers, which currently include:
- Adam Beardsley (Winona State University)
- Bryna Hazelton (University of Washington)
- Daniel Jacobs (Arizona State University)
- Matt Kolopanis (Arizona State University)
- Paul La Plante (University of Nevada, Las Vegas)
- Jonathan Pober (Brown University)
Please use the channels discussed in the [guide on contributing](.github/CONTRIBUTING.md)
for code-related discussions. You can contact us privately if needed at
[rasgmanagers@gmail.com](mailto:rasgmanagers@gmail.com).
# Acknowledgments
Support for pyuvdata was provided by NSF awards #1835421 and #1835120.
%prep
%autosetup -n pyuvdata-2.3.2
%build
%py3_build
%install
%py3_install
install -d -m755 %{buildroot}/%{_pkgdocdir}
if [ -d doc ]; then cp -arf doc %{buildroot}/%{_pkgdocdir}; fi
if [ -d docs ]; then cp -arf docs %{buildroot}/%{_pkgdocdir}; fi
if [ -d example ]; then cp -arf example %{buildroot}/%{_pkgdocdir}; fi
if [ -d examples ]; then cp -arf examples %{buildroot}/%{_pkgdocdir}; fi
pushd %{buildroot}
if [ -d usr/lib ]; then
find usr/lib -type f -printf "/%h/%f\n" >> filelist.lst
fi
if [ -d usr/lib64 ]; then
find usr/lib64 -type f -printf "/%h/%f\n" >> filelist.lst
fi
if [ -d usr/bin ]; then
find usr/bin -type f -printf "/%h/%f\n" >> filelist.lst
fi
if [ -d usr/sbin ]; then
find usr/sbin -type f -printf "/%h/%f\n" >> filelist.lst
fi
touch doclist.lst
if [ -d usr/share/man ]; then
find usr/share/man -type f -printf "/%h/%f.gz\n" >> doclist.lst
fi
popd
mv %{buildroot}/filelist.lst .
mv %{buildroot}/doclist.lst .
%files -n python3-pyuvdata -f filelist.lst
%dir %{python3_sitearch}/*
%files help -f doclist.lst
%{_docdir}/*
%changelog
* Thu May 18 2023 Python_Bot <Python_Bot@openeuler.org> - 2.3.2-1
- Package Spec generated
|