Why does the cloud cover percent displayed in the MTL .txt file of the force-level1-landsat program differ from the cloud cover % displayed in the log of force-level2

[paulvpop]

I ran the following lines of codes (1 and 2) in Powershell to search and download all the LANDSAT 8 OLI sensor images in 2024 with cloud cover between 0 and 10%, clipped to the a .shp vector (I was able to make FORCE work through Windows Powershell, since it did not work in the Windows Subsystem for Linux, and I am stuck with a Windows workstation at work (I prefer Linux)):

(1) force-level1-landsat search -s OLI -c 0,10 -d 20240101,20241231 --secret /home/docker/force/force/text.txt /home/docker/force/force/Siang_SA_Prj/Siang_SA_Prj.shp /home/docker/force/force/level1

OUTPUTS:

Sensor(s): OLI
Tile(s): 134040,134041,135040,135041,136040
Date range: 2024-01-01 to 2024-12-31
Included months: 1,2,3,4,5,6,7,8,9,10,11,12
Cloud cover: 0% to 10%

44 Landsat Level 1 scenes matching criteria found
51.61 GB data volume found
2 product bundles found in output directory, 43 not downloaded yet.
Remaining download size: 50.4 GB
Writing download links to /home/docker/force/force/level1/urls_landsat_OLI_20250319T074910.txt

(2) force-level1-landsat download /home/docker/force/force/level1/urls_landsat_OLI_20250319T074910.txt /home/docker/force/force/level1

OUTPUT:

Loading urls from /home/docker/force/force/level1/urls_landsat_OLI_20250319T074910.txt

Found 43 product bundle URLs.
Downloading: 100%|=====================================================================================================================================| 43/43 [28:55<00:00, 40.36s/product bundle]
Download complete

Since the imagery is all between 0 and 10% cloud cover, I thought that keeping the cloud over percent as 10 (%) for the parameter MAX_CLOUD_COVER_FRAME under CLOUD DETECTION OPTIONS for the force-level2 program would not have any effect, since the paramater has been described as

# This parameter cancels the processing of images that exceed the given
# threshold. The processing will be canceled after cloud detection.
# Type: Integer. Valid range: ]0,100]

However, after I run force-level2 /home/docker/force/force/parameter_level2.txt and I check the .log files under the "dir_log_level2" directory, I see that many files have been skipped because their cloud cover exceeds 10%. I will provide an example,

The file "LC09_L1TP_136040_20240310_20240310_02_T1_MTL.txt" in the LC09_L1TP_136040_20240310_20240310_02_T1.tar states:

CLOUD_COVER = 6.15
CLOUD_COVER_LAND = 6.15

However, when I check the corresponding force-level2 log files (after running force-level2 /home/docker/force/force/parameter_level2.txt), I see this:

dc: 68.20%. wc: 0.13%. sc: 37.73%. cc: 100.00%. Skip. Processing time: 00 mins 36 secs

Why is the cloud cover 100% in this and skipping the processing? Is this a bug?

I am attaching both the metadata and log files mentioned in this question
LC09_L1TP_136040_20240310_20240310_02_T1.tar.log
LC09_L1TP_136040_20240310_20240310_02_T1_MTL.txt

With kind regards,
Paul

[davidfrantz]

Hi @paulvpop,

FORCE uses a different version of Fmask for internal cloud detection. Also, USGS metadata of cloud cover do not include cloud shadow, and also no buffer distances. FORCE cloud coverages are usually somewhat larger.

Re the specific image: I cannot reproduce this, thus there still might be some issue on your end. I still end up with too much cloud cover, but this is all confusion between snow/ice and clouds, and hence less relevant for most applications. I am not sure what could cause this on your end, however. Probably the DEM (I used the Copernicus DEM)? FORCE version (I used 3.8.01)?

FORCE Level 2 Processing System
-----------------------------------------------------------

Start of processing: 2025-03-28 07:42:02
Image: /home/ahsoka/frantz/temp/paulvpop/LC09_L1TP_136040_20240310_20240310_02_T1.tar
Unpacking tar container successful

Start core processing
-----------------------------------------------------------

dc:  68.20%. wc:   0.15%. sc:  37.73%. cc:  41.98%.  1 product(s) written. Success! Processing time: 01 mins 22 secs

-----------------------------------------------------------
Core processing signaled DONE

End of processing: 2025-03-28 07:43:26
May the FORCE be with you!

Best,
David

[paulvpop]

Hello David,

Thanks for the clarification. Yeah, I also think that the algorithm may be confusing snow cover for clouds. I actually would need the imagery with snow cover (but less cloud cover) since I will be using it for land cover classification, including snow as a class. I checked the specific imagery in question. It indeed does have a lot of snow. I am attaching the imagery (showing the outline of the study area map touching the imagery).

Based on literature survey, I found that a spline-interpolated void-filled 90 m CGIAR-updated SRTM DEM was best (in comparison to using a 30 m resolution SRTM available from any source or ASTER). So, I prepared the interpolated 30 m DEM using Google Earth Engine myself for my specific study area. I had not considered a Copernicus DEM or an ALOS World 3D which I am reading is better. So, I will try with them as well. Thanks for bringing into my attention the availability of the Copernicus DEM.

I am also using v 3.8.01.

With kind regards,
Paul