View Issue Details

IDProjectCategoryView StatusLast Update
0000217Siril[All Projects] Sirilpublic2018-05-17 10:41
ReportervinvinAssigned Tovinvin 
PrioritynormalSeverityminorReproducibilityN/A
Status resolvedResolutionfixed 
Product Version 
Target Version0.9.9Fixed in Version0.9.9 
Summary0000217: Parallelize all registrations
Descriptionor make the registrations faster, in particular the global star.
For the global star registration, and other star detection algorithms, the blocking point is currently that wavelets are used and computing wavelets relies on the creation of a file and this code is not thread-safe.
TagsNo tags attached.

Activities

vinvin

2017-12-26 22:59

administrator   ~0000445

Maybe we can change the star detection algorithm as well, some are documented in the literature, here's an example: https://arxiv.org/abs/astro-ph/0002435v1

vinvin

2018-04-17 23:50

administrator   ~0000529

Profiling the global registration with gperftools [1], of siril compiled in -O0 -g, in a run in which a sequence of 12 images was loaded and registered, spent time is:
- 35.7% for get_wavelet_layers (23.6% of which are pave_2d_bspline_smooth and 7.1% test_ind)
- 10% for fits_img_stats_ushort (of which 8.4% are FnNoise1_ushort and 1.6% are FnMeanSigma_ushort)
- 5.4% for memcpy (called by wavelet_transform_data, get_wavelet_layers, peaker, register_star_alignment)
- 3.4% for siril_stats_ushort_median
- 0.95% for reassign_to_non_null_data in stats

peaker() itself doesn't consume CPU noticably outside the functions listed above.

The other half (47%) of the CPU consumption is distributed in a way I am not sure to follow:
- 2.4% of psf_Gaussian_df
- 1.6% of psf_global_minimization, in which 1.4% is removeHotPixels called in psf_init_data
- 1.6% of round_to_WORD
- the rest seems to be thread management and various opencv and gsl calls from no particular origin.

I will retry in -O2 and without OpenMP to have a more accurate representation of the thing.

[1] https://github.com/gperftools/gperftools

pprof9574.1.ps (55,685 bytes)

vinvin

2018-04-18 00:35

administrator   ~0000530

With -O2 and no OpenMP:
     914 14.7% 14.7% 1078 17.3% pave_2d_bspline_smooth
     650 10.4% 25.1% 650 10.4% __memcpy_sse2_unaligned_erms
     543 8.7% 33.8% 543 8.7% pixman_rasterize_edges
     518 8.3% 42.2% 518 8.3% cblas_dnrm2
     291 4.7% 46.8% 291 4.7% cv::HoughCircles
     211 3.4% 50.2% 211 3.4% siril_stats_ushort_median
     205 3.3% 53.5% 345 5.5% FnNoise1_ushort (inline)
     201 3.2% 56.8% 201 3.2% psf_Gaussian_df
     186 3.0% 59.7% 186 3.0% __cosh_finite
     151 2.4% 62.2% 157 2.5% cv::remap
     145 2.3% 64.5% 145 2.3% gsl_linalg_householder_hm
     140 2.2% 66.7% 140 2.2% FnMeanSigma_int (inline)
     132 2.1% 68.9% 132 2.1% test_ind (inline)

pprof16063.1.ps (25,714 bytes)

Related Changesets

Siril: 0.9 r2219

2018-04-17 00:05:36

vinvin

Details
Adding a function to access wavelets directly, without relying on a file to pass data. Ref issue 0000217.
Affected Issues
0000217
mod - /branches/0.9/src/algos/Def_Wavelet.h
mod - /branches/0.9/src/algos/quantize.c
mod - /branches/0.9/src/algos/star_finder.c
mod - /branches/0.9/src/algos/transform.c
mod - /branches/0.9/src/core/siril.c

Siril: 0.9 r2318

2018-05-16 01:40:36

vinvin

Details
Parallelizing the global registration, issue 0000217.
The global registration had to be ported to the generic processing function to do so, which wasn't straightforward:
- image hook has the same arguments as save hook now, with the two indices passed
- improved the output files of the generic sequence processing with the actual file number instead of the index of the input image
- added a thread-safe redraw in case we want to draw stars from the registration
- adding a crazy SER compacting function that allows SER to be created with holes during a parallel processing that can partially fail
- the loading of the new sequence after registration is now managed by the generic registration idle and not by the global registration
- moved all that to a new registration/global.c file

Registration now sets the best image as reference image of the sequence if none was defined.

Automatic exclusion of partial failures in generic sequence processing has been
disabled as it caused too many issues and was not functionally great imho.

Identifying the error condition in stats too, but not fixing it. Changing the way star_finder_params is used.
Saving stats at the end of a generic sequence processing because some may have been generated and they were not cached.
Affected Issues
0000217
mod - /branches/0.9/src/Makefile.am
mod - /branches/0.9/src/algos/cosmetic_correction.c
mod - /branches/0.9/src/algos/star_finder.c
mod - /branches/0.9/src/algos/star_finder.h
mod - /branches/0.9/src/algos/statistics.c
mod - /branches/0.9/src/core/command.c
mod - /branches/0.9/src/core/processing.c
mod - /branches/0.9/src/core/processing.h
mod - /branches/0.9/src/core/siril.c
mod - /branches/0.9/src/core/siril.h
mod - /branches/0.9/src/gui/callbacks.c
mod - /branches/0.9/src/gui/callbacks.h
mod - /branches/0.9/src/io/sequence.c
mod - /branches/0.9/src/io/ser.c
mod - /branches/0.9/src/io/ser.h
add - /branches/0.9/src/registration/global.c
mod - /branches/0.9/src/registration/matching/atpmatch.h
mod - /branches/0.9/src/registration/registration.c
mod - /branches/0.9/src/registration/registration.h
mod - /branches/0.9/src/stacking/stacking.c
mod - /branches/0.9/src/stacking/sum.c

Issue History

Date Modified Username Field Change
2017-12-07 20:57 vinvin New Issue
2017-12-26 22:26 vinvin Status new => acknowledged
2017-12-26 22:26 vinvin Description Updated View Revisions
2017-12-26 22:59 vinvin Note Added: 0000445
2018-04-12 00:43 vinvin Status acknowledged => confirmed
2018-04-17 00:05 vinvin Changeset attached => Siril 0.9 r2219
2018-04-17 23:50 vinvin File Added: pprof9574.1.ps
2018-04-17 23:50 vinvin Note Added: 0000529
2018-04-18 00:35 vinvin File Added: pprof16063.1.ps
2018-04-18 00:35 vinvin Note Added: 0000530
2018-05-10 10:51 lock42 Assigned To => vinvin
2018-05-10 10:51 lock42 Status confirmed => assigned
2018-05-10 10:51 lock42 Target Version 1.0.0 => 0.9.9
2018-05-16 01:40 vinvin Changeset attached => Siril 0.9 r2318
2018-05-17 10:41 lock42 Status assigned => resolved
2018-05-17 10:41 lock42 Resolution open => fixed
2018-05-17 10:41 lock42 Fixed in Version => 0.9.9