Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix clock normalisation and convert sp3 position nodata values to NaNs #30

Merged
merged 5 commits into from
Jul 18, 2024
Merged

Fix clock normalisation and convert sp3 position nodata values to NaNs #30

merged 5 commits into from
Jul 18, 2024

Conversation

EugeneDu-GA
Copy link
Collaborator

  • Fixed 'daily' and 'epoch' clock normalisation with compare_clk() when one of the files have missing data;
  • Added option to convert sp3 positional nodata values to NaNs.

@ronaldmaj
Copy link
Collaborator

Awesome, thanks for the PR @EugeneDu-GA !
Do you have any example files that you have used in order to test the two new features?

@EugeneDu-GA
Copy link
Collaborator Author

Awesome, thanks for the PR @EugeneDu-GA ! Do you have any example files that you have used in order to test the two new features?

Yes I have, where to upload them?

@ronaldmaj
Copy link
Collaborator

If they're from CDDIS or elsewhere, perhaps just the link to the files, otherwise it seems like you can attached files to comments in GitHub so you should be able to put them here

@EugeneDu-GA
Copy link
Collaborator Author

Test examples & results:

Test 1. $ orbq -i SSRA00BKG0_20241620000_ORB.sp3 IGS0OPSRAP_20241620000_01D_15M_ORB.SP3
Before fixing:

PRN	R_RMS	A_RMS	C_RMS	dX_RMS	dY_RMS	dZ_RMS	MEAN	AVG	3D_RMS
G01				15662439.64081	14073738.7273	16243211.65011	15326463.3394	-363614.73842	26593684.6258
G02	0.01994	0.03846	0.03459	0.02851	0.03484	0.03223	0.03186	0.01207	0.05537
G03	0.02628	0.10491	0.02295	0.06793	0.06287	0.06065	0.06382	0.00345	0.11066
G04	0.01292	0.05682	0.01264	0.03929	0.03401	0.02922	0.03417	0.00907	0.05961
G05	0.00987	0.05237	0.02391	0.03965	0.03119	0.02947	0.03344	-0.00333	0.05843
G06	0.00714	0.03137	0.02281	0.02041	0.01703	0.02913	0.02219	0.00316	0.03943
G07	0.00658	0.05336	0.0288	0.0381	0.02866	0.03808	0.03495	0.00844	0.06102
G08	0.01787	0.05567	0.02095	0.04473	0.03384	0.02649	0.03502	-0.01771	0.06203
G09	0.00593	0.05549	0.02627	0.04049	0.03574	0.02981	0.03535	0.00985	0.06169
G10	0.01669	0.07011	0.03186	0.05504	0.04631	0.03218	0.04451	-0.00187	0.0788
G11	0.01586	0.0279	0.02989	0.02889	0.02461	0.02197	0.02515	-0.00357	0.04385
G12	0.02301	0.02427	0.03085	0.02842	0.02727	0.02283	0.02617	-0.00232	0.04552
G13	0.01512	0.04839	0.02235	0.03519	0.02901	0.03147	0.03189	-0.00698	0.05541
G14	0.01648	0.03348	0.02993	0.0338	0.02691	0.0205	0.02707	0.00486	0.04782
G15	0.01768	0.04849	0.0272	0.03344	0.03621	0.03114	0.0336	-0.00664	0.0583
G16	0.01286	0.04692	0.01331	0.02688	0.03533	0.02398	0.02873	-0.00496	0.05046
G17	0.01663	0.04716	0.03016	0.03983	0.02805	0.03215	0.03334	0.00895	0.05837
G18	0.01724	0.01786	0.02916	0.02559	0.01509	0.02416	0.02161	-0.00546	0.03829
G19	0.01409	0.04047	0.02396	0.03401	0.02525	0.02488	0.02805	0.00158	0.04913
G20	0.0087	0.04591	0.03066	0.03715	0.03766	0.01807	0.03096	-0.0023	0.0559
G21	0.01706	0.02388	0.03083	0.02765	0.02327	0.02238	0.02443	0.00465	0.04251
G22	0.01119	0.04867	0.0221	0.03774	0.03507	0.01805	0.03029	-0.00415	0.05459
G23	0.02109	7978914.98065	0.02738	6972213.13688	8881201.44903	6327034.79143	7393483.12578	5636.17277	12942907.88255
G24	0.01462	0.03201	0.01921	0.0309	0.01881	0.01722	0.02231	0.00713	0.04007
G25	0.00657	0.0385	0.03299	0.02445	0.03665	0.026	0.02904	0.00588	0.05116
G26	0.00859	0.04494	0.02228	0.02663	0.04013	0.01645	0.02774	0.00167	0.05089
G27	0.00597	0.0188	0.01911	0.01361	0.01579	0.01787	0.01576	-0.00094	0.02746
G28	0.01364	0.06789	0.03001	0.05047	0.02814	0.04855	0.04239	0.01772	0.07548
G29	0.0191	0.0395	0.03886	0.04548	0.03072	0.02058	0.03226	-0.00587	0.05861
G30	0.00705	0.0629	0.02249	0.03864	0.03881	0.03885	0.03877	0.00418	0.06715
G31	0.00597	0.03652	0.0244	0.02754	0.02858	0.0197	0.02527	-0.00475	0.04431
G32	0.00833	0.05223	0.01858	0.04213	0.02874	0.02333	0.0314	0.00615	0.05608

	R_RMS	A_RMS	C_RMS	dX_RMS	dY_RMS	dZ_RMS	MEAN	AVG	3D_RMS
AVG	0.01355	257384.39826	0.02582	707332.93251	717341.90972	705320.22715	709998.35646	-11186.82899	1235518.56771
STD	0.01255	256784.75673	0.02381	684454.9859	708018.09546	696643.68455			361497.75435
RMS	0.01461	1433055.43495	0.02648	3030695.93662	2941863.04142	3081564.60849	3008135.17412	64286.33322	5228356.9552

After Fixing:

PRN	R_RMS	A_RMS	C_RMS	dX_RMS	dY_RMS	dZ_RMS	MEAN	AVG	3D_RMS
G02	0.01994	0.03846	0.03459	0.02851	0.03484	0.03223	0.03186	0.01207	0.05537
G03	0.02628	0.10491	0.02295	0.06793	0.06287	0.06065	0.06382	0.00345	0.11066
G04	0.01292	0.05682	0.01264	0.03929	0.03401	0.02922	0.03417	0.00907	0.05961
G05	0.00987	0.05237	0.02391	0.03965	0.03119	0.02947	0.03344	-0.00333	0.05843
G06	0.00714	0.03137	0.02281	0.02041	0.01703	0.02913	0.02219	0.00316	0.03943
G07	0.00658	0.05336	0.0288	0.0381	0.02866	0.03808	0.03495	0.00844	0.06102
G08	0.01787	0.05567	0.02095	0.04473	0.03384	0.02649	0.03502	-0.01771	0.06203
G09	0.00593	0.05549	0.02627	0.04049	0.03574	0.02981	0.03535	0.00985	0.06169
G10	0.01669	0.07011	0.03186	0.05504	0.04631	0.03218	0.04451	-0.00187	0.0788
G11	0.01586	0.0279	0.02989	0.02889	0.02461	0.02197	0.02515	-0.00357	0.04385
G12	0.02301	0.02427	0.03085	0.02842	0.02727	0.02283	0.02617	-0.00232	0.04552
G13	0.01512	0.04839	0.02235	0.03519	0.02901	0.03147	0.03189	-0.00698	0.05541
G14	0.01648	0.03348	0.02993	0.0338	0.02691	0.0205	0.02707	0.00486	0.04782
G15	0.01768	0.04849	0.0272	0.03344	0.03621	0.03114	0.0336	-0.00664	0.0583
G16	0.01286	0.04692	0.01331	0.02688	0.03533	0.02398	0.02873	-0.00496	0.05046
G17	0.01663	0.04716	0.03016	0.03983	0.02805	0.03215	0.03334	0.00895	0.05837
G18	0.01724	0.01786	0.02916	0.02559	0.01509	0.02416	0.02161	-0.00546	0.03829
G19	0.01409	0.04047	0.02396	0.03401	0.02525	0.02488	0.02805	0.00158	0.04913
G20	0.0087	0.04591	0.03066	0.03715	0.03766	0.01807	0.03096	-0.0023	0.0559
G21	0.01706	0.02388	0.03083	0.02765	0.02327	0.02238	0.02443	0.00465	0.04251
G22	0.01119	0.04867	0.0221	0.03774	0.03507	0.01805	0.03029	-0.00415	0.05459
G24	0.01462	0.03201	0.01921	0.0309	0.01881	0.01722	0.02231	0.00713	0.04007
G25	0.00657	0.0385	0.03299	0.02445	0.03665	0.026	0.02904	0.00588	0.05116
G26	0.00859	0.04494	0.02228	0.02663	0.04013	0.01645	0.02774	0.00167	0.05089
G27	0.00597	0.0188	0.01911	0.01361	0.01579	0.01787	0.01576	-0.00094	0.02746
G28	0.01364	0.06789	0.03001	0.05047	0.02814	0.04855	0.04239	0.01772	0.07548
G29	0.0191	0.0395	0.03886	0.04548	0.03072	0.02058	0.03226	-0.00587	0.05861
G30	0.00705	0.0629	0.02249	0.03864	0.03881	0.03885	0.03877	0.00418	0.06715
G31	0.00597	0.03652	0.0244	0.02754	0.02858	0.0197	0.02527	-0.00475	0.04431
G32	0.00833	0.05223	0.01858	0.04213	0.02874	0.02333	0.0314	0.00615	0.05608

	R_RMS	A_RMS	C_RMS	dX_RMS	dY_RMS	dZ_RMS	MEAN	AVG	3D_RMS
AVG	0.0133	0.04551	0.02577	0.03542	0.03115	0.02758	0.03138	0.00127	0.05528
STD	0.01245	0.03627	0.02383	0.03278	0.02912	0.02387			0.02471
RMS	0.01435	0.04867	0.02645	0.03697	0.03252	0.02915	0.03252	0.00725	0.05722

Note that orbq automatically removes satellites containing NaN values.

2. $ clkq -i SSRA00BKG0_20241620000_CLK.clk IGS0OPSRAP_20241620000_01D_05M_CLK.CLK -o clkq_test/2024162_daily_epoch.csv -p clkq_test/2024162_daily_epoch.png --norm daily --norm epoch
Before fixing:
image
After fixing:
image

Example files:
IGS0OPSRAP_20241620000_01D_15M_ORB.SP3.gz
IGS0OPSRAP_20241620000_01D_05M_CLK.CLK.gz
SSRA00BKG0_20241620000_ORB.sp3.gz
SSRA00BKG0_20241620000_CLK.clk.gz

@EugeneDu-GA
Copy link
Collaborator Author

Backlog: option to keep valid orbital data/epoch of a satellite containing NaN values.

@ronaldmaj
Copy link
Collaborator

Backlog: option to keep valid orbital data/epoch of a satellite containing NaN values.

This is a good idea. You can probably write this up as a ticket so we have it for future reference

@ronaldmaj
Copy link
Collaborator

Test examples & results:

Test 1. $ orbq -i SSRA00BKG0_20241620000_ORB.sp3 IGS0OPSRAP_20241620000_01D_15M_ORB.SP3 Before fixing:

PRN	R_RMS	A_RMS	C_RMS	dX_RMS	dY_RMS	dZ_RMS	MEAN	AVG	3D_RMS
G01				15662439.64081	14073738.7273	16243211.65011	15326463.3394	-363614.73842	26593684.6258
G02	0.01994	0.03846	0.03459	0.02851	0.03484	0.03223	0.03186	0.01207	0.05537
G03	0.02628	0.10491	0.02295	0.06793	0.06287	0.06065	0.06382	0.00345	0.11066
G04	0.01292	0.05682	0.01264	0.03929	0.03401	0.02922	0.03417	0.00907	0.05961
G05	0.00987	0.05237	0.02391	0.03965	0.03119	0.02947	0.03344	-0.00333	0.05843
G06	0.00714	0.03137	0.02281	0.02041	0.01703	0.02913	0.02219	0.00316	0.03943
G07	0.00658	0.05336	0.0288	0.0381	0.02866	0.03808	0.03495	0.00844	0.06102
G08	0.01787	0.05567	0.02095	0.04473	0.03384	0.02649	0.03502	-0.01771	0.06203
G09	0.00593	0.05549	0.02627	0.04049	0.03574	0.02981	0.03535	0.00985	0.06169
G10	0.01669	0.07011	0.03186	0.05504	0.04631	0.03218	0.04451	-0.00187	0.0788
G11	0.01586	0.0279	0.02989	0.02889	0.02461	0.02197	0.02515	-0.00357	0.04385
G12	0.02301	0.02427	0.03085	0.02842	0.02727	0.02283	0.02617	-0.00232	0.04552
G13	0.01512	0.04839	0.02235	0.03519	0.02901	0.03147	0.03189	-0.00698	0.05541
G14	0.01648	0.03348	0.02993	0.0338	0.02691	0.0205	0.02707	0.00486	0.04782
G15	0.01768	0.04849	0.0272	0.03344	0.03621	0.03114	0.0336	-0.00664	0.0583
G16	0.01286	0.04692	0.01331	0.02688	0.03533	0.02398	0.02873	-0.00496	0.05046
G17	0.01663	0.04716	0.03016	0.03983	0.02805	0.03215	0.03334	0.00895	0.05837
G18	0.01724	0.01786	0.02916	0.02559	0.01509	0.02416	0.02161	-0.00546	0.03829
G19	0.01409	0.04047	0.02396	0.03401	0.02525	0.02488	0.02805	0.00158	0.04913
G20	0.0087	0.04591	0.03066	0.03715	0.03766	0.01807	0.03096	-0.0023	0.0559
G21	0.01706	0.02388	0.03083	0.02765	0.02327	0.02238	0.02443	0.00465	0.04251
G22	0.01119	0.04867	0.0221	0.03774	0.03507	0.01805	0.03029	-0.00415	0.05459
G23	0.02109	7978914.98065	0.02738	6972213.13688	8881201.44903	6327034.79143	7393483.12578	5636.17277	12942907.88255
G24	0.01462	0.03201	0.01921	0.0309	0.01881	0.01722	0.02231	0.00713	0.04007
G25	0.00657	0.0385	0.03299	0.02445	0.03665	0.026	0.02904	0.00588	0.05116
G26	0.00859	0.04494	0.02228	0.02663	0.04013	0.01645	0.02774	0.00167	0.05089
G27	0.00597	0.0188	0.01911	0.01361	0.01579	0.01787	0.01576	-0.00094	0.02746
G28	0.01364	0.06789	0.03001	0.05047	0.02814	0.04855	0.04239	0.01772	0.07548
G29	0.0191	0.0395	0.03886	0.04548	0.03072	0.02058	0.03226	-0.00587	0.05861
G30	0.00705	0.0629	0.02249	0.03864	0.03881	0.03885	0.03877	0.00418	0.06715
G31	0.00597	0.03652	0.0244	0.02754	0.02858	0.0197	0.02527	-0.00475	0.04431
G32	0.00833	0.05223	0.01858	0.04213	0.02874	0.02333	0.0314	0.00615	0.05608

	R_RMS	A_RMS	C_RMS	dX_RMS	dY_RMS	dZ_RMS	MEAN	AVG	3D_RMS
AVG	0.01355	257384.39826	0.02582	707332.93251	717341.90972	705320.22715	709998.35646	-11186.82899	1235518.56771
STD	0.01255	256784.75673	0.02381	684454.9859	708018.09546	696643.68455			361497.75435
RMS	0.01461	1433055.43495	0.02648	3030695.93662	2941863.04142	3081564.60849	3008135.17412	64286.33322	5228356.9552

After Fixing:

PRN	R_RMS	A_RMS	C_RMS	dX_RMS	dY_RMS	dZ_RMS	MEAN	AVG	3D_RMS
G02	0.01994	0.03846	0.03459	0.02851	0.03484	0.03223	0.03186	0.01207	0.05537
G03	0.02628	0.10491	0.02295	0.06793	0.06287	0.06065	0.06382	0.00345	0.11066
G04	0.01292	0.05682	0.01264	0.03929	0.03401	0.02922	0.03417	0.00907	0.05961
G05	0.00987	0.05237	0.02391	0.03965	0.03119	0.02947	0.03344	-0.00333	0.05843
G06	0.00714	0.03137	0.02281	0.02041	0.01703	0.02913	0.02219	0.00316	0.03943
G07	0.00658	0.05336	0.0288	0.0381	0.02866	0.03808	0.03495	0.00844	0.06102
G08	0.01787	0.05567	0.02095	0.04473	0.03384	0.02649	0.03502	-0.01771	0.06203
G09	0.00593	0.05549	0.02627	0.04049	0.03574	0.02981	0.03535	0.00985	0.06169
G10	0.01669	0.07011	0.03186	0.05504	0.04631	0.03218	0.04451	-0.00187	0.0788
G11	0.01586	0.0279	0.02989	0.02889	0.02461	0.02197	0.02515	-0.00357	0.04385
G12	0.02301	0.02427	0.03085	0.02842	0.02727	0.02283	0.02617	-0.00232	0.04552
G13	0.01512	0.04839	0.02235	0.03519	0.02901	0.03147	0.03189	-0.00698	0.05541
G14	0.01648	0.03348	0.02993	0.0338	0.02691	0.0205	0.02707	0.00486	0.04782
G15	0.01768	0.04849	0.0272	0.03344	0.03621	0.03114	0.0336	-0.00664	0.0583
G16	0.01286	0.04692	0.01331	0.02688	0.03533	0.02398	0.02873	-0.00496	0.05046
G17	0.01663	0.04716	0.03016	0.03983	0.02805	0.03215	0.03334	0.00895	0.05837
G18	0.01724	0.01786	0.02916	0.02559	0.01509	0.02416	0.02161	-0.00546	0.03829
G19	0.01409	0.04047	0.02396	0.03401	0.02525	0.02488	0.02805	0.00158	0.04913
G20	0.0087	0.04591	0.03066	0.03715	0.03766	0.01807	0.03096	-0.0023	0.0559
G21	0.01706	0.02388	0.03083	0.02765	0.02327	0.02238	0.02443	0.00465	0.04251
G22	0.01119	0.04867	0.0221	0.03774	0.03507	0.01805	0.03029	-0.00415	0.05459
G24	0.01462	0.03201	0.01921	0.0309	0.01881	0.01722	0.02231	0.00713	0.04007
G25	0.00657	0.0385	0.03299	0.02445	0.03665	0.026	0.02904	0.00588	0.05116
G26	0.00859	0.04494	0.02228	0.02663	0.04013	0.01645	0.02774	0.00167	0.05089
G27	0.00597	0.0188	0.01911	0.01361	0.01579	0.01787	0.01576	-0.00094	0.02746
G28	0.01364	0.06789	0.03001	0.05047	0.02814	0.04855	0.04239	0.01772	0.07548
G29	0.0191	0.0395	0.03886	0.04548	0.03072	0.02058	0.03226	-0.00587	0.05861
G30	0.00705	0.0629	0.02249	0.03864	0.03881	0.03885	0.03877	0.00418	0.06715
G31	0.00597	0.03652	0.0244	0.02754	0.02858	0.0197	0.02527	-0.00475	0.04431
G32	0.00833	0.05223	0.01858	0.04213	0.02874	0.02333	0.0314	0.00615	0.05608

	R_RMS	A_RMS	C_RMS	dX_RMS	dY_RMS	dZ_RMS	MEAN	AVG	3D_RMS
AVG	0.0133	0.04551	0.02577	0.03542	0.03115	0.02758	0.03138	0.00127	0.05528
STD	0.01245	0.03627	0.02383	0.03278	0.02912	0.02387			0.02471
RMS	0.01435	0.04867	0.02645	0.03697	0.03252	0.02915	0.03252	0.00725	0.05722

Note that orbq automatically removes satellites containing NaN values.

2. $ clkq -i SSRA00BKG0_20241620000_CLK.clk IGS0OPSRAP_20241620000_01D_05M_CLK.CLK -o clkq_test/2024162_daily_epoch.csv -p clkq_test/2024162_daily_epoch.png --norm daily --norm epoch Before fixing: image After fixing: image

Example files: IGS0OPSRAP_20241620000_01D_15M_ORB.SP3.gz IGS0OPSRAP_20241620000_01D_05M_CLK.CLK.gz SSRA00BKG0_20241620000_ORB.sp3.gz SSRA00BKG0_20241620000_CLK.clk.gz

Awesome, thanks so much for putting this together and showing the improvement with before / after examples.

I've run this locally and I'm getting the same results as you, so seems to be working well.

I've added a couple more comments in individual lines of code, but I think this is mostly all good to merge in

ronaldmaj
ronaldmaj requested changes Jul 15, 2024
View reviewed changes
Copy link
Collaborator

@ronaldmaj ronaldmaj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great, I think this has come out really nicely and will make comparisons via the CLI easier.
The one change I've requested is just to use the one-line-sphinx documentation style for the new function you've introduced. Everything else looks good!

gnssanalysis/gn_io/sp3.py Outdated
SP3_CLOCK_STD_NODATA = -1000
SP3_POS_STD_NODATA = -100


def sp3_pos_nodata_to_nan(sp3_df):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a long term clean-up project, but we're hoping to slowly move all python code to follow a documentation standard. In our case, we've chosen one-line sphinx style: https://github.com/NilsJPWerner/autoDocstring/blob/HEAD/docs/one-line-sphinx.md

There's an example of this in the compare_clk function:

gnssanalysis/gnssanalysis/gn_diffaux.py

Line 307 in 2297ba7

def compare_clk(

Ideally all new functions follow this format and in future we'll work towards going through the backlog and changing all the old functions to the new doc standard as well

I think the main thing for your function sp3_pos_nodata_to_nan() is to include type hints and an explanation of the input / outputs in the Docstring

ronaldmaj
ronaldmaj approved these changes Jul 18, 2024
View reviewed changes
Copy link
Collaborator

@ronaldmaj ronaldmaj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great! Thanks for getting those docstrings put through. Ready to go in now 👍

@ronaldmaj ronaldmaj merged commit 82087f9 into main Jul 18, 2024
@EugeneDu-GA EugeneDu-GA deleted the NPI-3304-SSR-stream-quality branch September 19, 2024 00:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ronaldmaj ronaldmaj ronaldmaj approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@EugeneDu-GA @ronaldmaj