Header Table Corruption

[mikhail-dcl]

Hello, first, let me thank you, @Benedicht, for the amazing work.

Currently, at Decentraland I am working on the integration of your package as a full replacement for Unity WebRequests, and I encountered an issue I can't overcome so far on my own.

We are doing massive concurrent partial requests (Range header) to our CDN (CloudFront) and sporadically receive the following exception:

{"tid":125,"div":"HTTP2Stream","msg":"[155] ProcessIncomingFrames - Header Frames: [CommonFrameView [HTTP2FrameHeaderAndPayload Type: HEADERS, Flags: 00000100, StreamId: 155, DontUseMemPool: True, Payload: [BufferSegment Offset: 0 Count: 259 Data: [8A, 61, 96, DD, 6D, 5F, 4A, 08, 0A, 43, 5D, 8A, 08, 02, 6D, 40, B9, 70, 2E, DC, 03, EA, 62, D1, BF, EF, 5C, 85, 75, C6, 9F, 13, DF, EF, 6C, 96, DF, 3D, BF, 4A, 00, 4A, 68, 1F, A5, 04, 01, 34, A0, 1B, B8, D0, 6E, 36, FA, 98, B4, 6F, 62, 9A, FE, 5E, 6C, 4D, 38, EC, 6D, BA, 51, 48, 56, D9, 69, F7, 59, 94, 63, 90, 61, 71, 97, 0A, CB, E0, FE, 7F, C8, 7C, AF, 0A, E1, 52, 51, 49, 23, 31, B8, DB, 2F, 06, 51, C8, D8, C7, 23, ED, 0C, 8F, B4, 39, 1D, 03, 4C, CA, F2, E4, A0, F6, C9, 2D, 87, A9, 2B, D4, 54, A9, FD, 5E, ...]]] Payload: [BufferSegment Offset: 0 Count: 259 Data: [8A, 61, 96, DD, 6D, 5F, 4A, 08, 0A, 43, 5D, 8A, 08, 02, 6D, 40, B9, 70, 2E, DC, 03, EA, 62, D1, BF, EF, 5C, 85, 75, C6, 9F, 13, DF, EF, 6C, 96, DF, 3D, BF, 4A, 00, 4A, 68, 1F, A5, 04, 01, 34, A0, 1B, B8, D0, 6E, 36, FA, 98, B4, 6F, 62, 9A, FE, 5E, 6C, 4D, 38, EC, 6D, BA, 51, 48, 56, D9, 69, F7, 59, 94, 63, 90, 61, 71, 97, 0A, CB, E0, FE, 7F, C8, 7C, AF, 0A, E1, 52, 51, 49, 23, 31, B8, DB, 2F, 06, 51, C8, D8, C7, 23, ED, 0C, 8F, B4, 39, 1D, 03, 4C, CA, F2, E4, A0, F6, C9, 2D, 87, A9, 2B, D4, 54, A9, FD, 5E, ...]]\n], Encoder: [HeaderTable DynamicTable count: 4, DynamicTableSize: 304, MaxDynamicTableSize: 4096, \"range\": \"bytes=0-14680064\", \"cookie\": \"DCL-Session=06d42c4b-7b1e-474c-8fb1-ece6114d9393\", \"user-agent\": \"com.Tivadar.Best.HTTP v3.0.14.0\/Unity 6000.0.44f1\", \"accept-encoding\": \"br, gzip, deflate, identity\", ][HeaderTable DynamicTable count: 58, DynamicTableSize: 4051, MaxDynamicTableSize: 4096, \"via\": \"1.1 f2dd3b65381fad5aad94ac94ad7043f8.cloudfront.net (CloudFront)\", \"etag\": \"\"852467b57e2de534973faada1636e390\"\", \"last-modified\": \"Thu, 02 May 2024 05:41:59 GMT\", \"content-length\": \"764928\", \"date\": \"Sun, 20 Apr 2025 16:17:09 GMT\", \"via\": \"9335f334ea9fb8ca-AMS\", \"content-range\": \"bytes 0-893839\/893840\", \"age\": \"10347\", \"last-modified\": \"8OyW2xg3mO3hipFl0ASsK5UEyzMUJhPc0h5YS0Y2QPHiobijkJ9LAQ==\", \"last-modified\": \"AMS58-P5\", \"via\": \"1.1 3f24561b20ab2825cb11ac40fc1c2434.cloudfront.net (CloudFront)\", \"etag\": \"Miss from cloudfront\", \"etag\": \"\"26b0b49aab735bd77e7a83fbb3dfd045\"\", \"last-modified\": \"Fri, 03 May 2024 02:45:31 GMT\", \"content-length\": \"893840\", \"strict-transport-security\": \"9335f33419fbb8ca-AMS\", \"age\": \"10491\", \"strict-transport-security\": \"9335f33419ffb8ca-AMS\", \"server\": \"cloudflare\", \"x-content-type-options\": \"nosniff\", \"strict-transport-security\": \"max-age=15552000; includeSubDomains; preload\", \"vary\": \"Accept-Encoding\", \"content-range\": \"bytes 0-754255\/754256\", \"expires\": \"Mon, 20 Apr 2026 16:17:07 GMT\", \"age\": \"1654\", \"content-range\": \"OrpQmlv2-dVNfdbW9797M2D7W-rp6oEa_zjSQc_Op6KIveJt-Sp9Kg==\", \"last-modified\": \"CDG52-P3\", \"via\": \"1.1 0b61c24c96e423ae09ca12c7ee3a196c.cloudfront.net (CloudFront)\", \"etag\": \"\"93b1c78e9d4f1786515a6891a6a03abb\"\", \"last-modified\": \"Mon, 29 Apr 2024 14:19:28 GMT\", \"content-length\": \"754256\", \"date\": \"Sun, 20 Apr 2025 16:17:07 GMT\", \"cf-ray\": \"9335f32acaa6b8ca-AMS\", \"x-amz-cf-id\": \"Pmgk412R450bd4_wS4FX4zz7TA4LophNAM1XLzSdzz6DlXqtbcabRA==\", \"etag\": \"W\/\"25899db80af1908764f58e7d5f2aaa35\"\", \"last-modified\": \"Fri, 03 May 2024 02:17:06 GMT\", \"cf-ray\": \"9335f32a4a04b8ca-AMS\", \"x-amz-cf-id\": \"deL2JvlxcbJDIBUgWf5g9PldjFoCA9qZXSVrOJDx4bmHR1KBUiQ_4Q==\", \"via\": \"1.1 261dbbb56805d787e4e94407d9511a46.cloudfront.net (CloudFront)\", \"etag\": \"W\/\"18787bbcd27ab1635585379fe4023b02\"\", \"last-modified\": \"Fri, 31 Jan 2025 20:11:08 GMT\", \"cache-control\": \"public, max-age=31536000\", \"content-type\": \"application\/wasm\", \"cf-ray\": \"9335f32a4a05b8ca-AMS\", \"content-range\": \"bytes 0-46782\/46783\", \"cf-cache-status\": \"HIT\", \"x-amz-cf-id\": \"4Ux8N23MjkG1ATJUvx32vmDWoCpv4D6PXgYfPfLAIORL4BeHQLbFBA==\", \"x-amz-cf-pop\": \"MAN50-C2\", \"via\": \"1.1 bcc4ad8f771bded524c65fdce1b7af50.cloudfront.net (CloudFront)\", \"etag\": \"\"6a5fbd7ad49373b4af3528cb87b303a2\"\", \"last-modified\": \"Tue, 28 Jan 2025 05:59:16 GMT\", \"content-length\": \"46783\", \"cf-ray\": \"9335f32a4a03b8ca-AMS\", \"content-range\": \"bytes 0-774719\/774720\", \"age\": \"10488\", \"x-amz-cf-id\": \"867eofm7qXfZWBEpvRfxmsXTGRIYsGJz9WmCmzY3kgtCvHUrWjafyw==\", \"via\": \"1.1 2358c92c016b37a066a4efcb933f0da0.cloudfront.net (CloudFront)\", \"etag\": \"\"027d504685cbfaa78272cb1babf0a7e2\"\", ]","ex": [{"msg": "Index was out of range. Must be non-negative and less than the size of the collection.\r\nParameter name: index", "stack": " at System.Collections.Generic.List`1[T].get_Item (System.Int32 index) [0x00009] in <314938d17f3848e8ac683e11b27f62ee>:0 \r\n at Best.HTTP.Hosts.Connections.HTTP2.HeaderTable.GetKey (System.UInt32 index) [0x0001b] in D:\\Decentraland\\unity-explorer-packages\\BestHTTP\\Runtime\\HTTP\\Hosts\\Connections\\HTTP2\\HeaderTable.cs:135 \r\n at Best.HTTP.Hosts.Connections.HTTP2.HPACKEncoder.ReadLiteralHeaderFieldWithIncrementalIndexing_IndexedName (System.Byte firstByte, System.IO.Stream stream) [0x0000a] in D:\\Decentraland\\unity-explorer-packages\\BestHTTP\\Runtime\\HTTP\\Hosts\\Connections\\HTTP2\\HPACKEncoder.cs:243 \r\n at Best.HTTP.Hosts.Connections.HTTP2.HPACKEncoder.Decode (Best.HTTP.Hosts.Connections.HTTP2.HTTP2Stream context, System.IO.Stream stream, System.Collections.Generic.List`1[T] to) [0x0011f] in D:\\Decentraland\\unity-explorer-packages\\BestHTTP\\Runtime\\HTTP\\Hosts\\Connections\\HTTP2\\HPACKEncoder.cs:149 \r\n at Best.HTTP.Hosts.Connections.HTTP2.HTTP2Stream.ProcessIncomingFrames (System.Collections.Generic.List`1[T] outgoingFrames) [0x00194] in D:\\Decentraland\\unity-explorer-packages\\BestHTTP\\Runtime\\HTTP\\Hosts\\Connections\\HTTP2\\HTTP2Stream.cs:307 "}],"stack":" at Hosts.Connections.HTTP2.HTTP2Stream.ProcessIncomingFrames (System.Collections.Generic.List`1[T] outgoingFrames) [0x001af] in D:\\Decentraland\\unity-explorer-packages\\BestHTTP\\Runtime\\HTTP\\Hosts\\Connections\\HTTP2\\HTTP2Stream.cs:311 \r at Hosts.Connections.HTTP2.HTTP2Stream.AddFrame (Hosts.Connections.HTTP2.HTTP2FrameHeaderAndPayload frame, System.Collections.Generic.List`1[T] outgoingFrames) [0x0000e] in D:\\Decentraland\\unity-explorer-packages\\BestHTTP\\Runtime\\HTTP\\Hosts\\Connections\\HTTP2\\HTTP2Stream.cs:211 \r at Hosts.Connections.HTTP2.HTTP2ContentConsumer.RunHandler () [0x0044c] in D:\\Decentraland\\unity-explorer-packages\\BestHTTP\\Runtime\\HTTP\\Hosts\\Connections\\HTTP2\\HTTP2ContentConsumer.cs:262 \r at Hosts.Connections.HTTPOverTCPConnection.ThreadFunc () [0x00001] in D:\\Decentraland\\unity-explorer-packages\\BestHTTP\\Runtime\\HTTP\\Hosts\\Connections\\HTTPOverTCPConnection.cs:366 \r at Shared.PlatformSupport.Threading.ThreadedRunner+<>c__DisplayClass12_0.<RunLongLiving>b__0 (System.Object param) [0x00009] in D:\\Decentraland\\unity-explorer-packages\\BestHTTP\\Runtime\\Shared\\PlatformSupport\\Threading\\ThreadedRunner.cs:78 \r at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) [0x00000] in <314938d17f3848e8ac683e11b27f62ee>:0 \r at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <314938d17f3848e8ac683e11b27f62ee>:0 \r at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <314938d17f3848e8ac683e11b27f62ee>:0 \r at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) [0x00000] in <314938d17f3848e8ac683e11b27f62ee>:0 \r at System.Threading.ThreadHelper.ThreadStart (System.Object obj) [0x00000] in <314938d17f3848e8ac683e11b27f62ee>:0 ","ctx": [{"TypeName": "HTTP2Stream", "Hash": "78a441b0aad40400cb5f0f32669512b9", "id": 155}, {"TypeName": "HTTP2ContentConsumer", "Hash": "77c9cdc537ac83ceac4ace1e0934dd00", "Connection": "1f78938c7d97737be801448626e46ca1"}, {"TypeName": "HTTPOverTCPConnection", "Hash": "1f78938c7d97737be801448626e46ca1", "HostKey": "{\"Uri\":\"https:\/\/ab-cdn.decentraland.org\", \"Proxy\": , \"Key\": 97d76f145d9bbbdce0bc62ac9986cc3f}", "RequestHandler": "77c9cdc537ac83ceac4ace1e0934dd00"}, {"TypeName": "HTTPRequest", "Hash": "0648244211426e251866a0e529124c1f", "Connection": "1f78938c7d97737be801448626e46ca1"}, {"TypeName": "HostVariant", "Hash": "7bf54631bbca2c4fd4309662f35e7c25", "Host": "ab-cdn.decentraland.org"}],"t":638807626295736921,"ll":"Exception","bh":1}

A bit of prettified output:

The error occurs in the HTTP/2 stream processing:

Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index
  at System.Collections.Generic.List`1[T].get_Item (System.Int32 index)
  at Best.HTTP.Hosts.Connections.HTTP2.HeaderTable.GetKey (System.UInt32 index)
  at Best.HTTP.Hosts.Connections.HTTP2.HPACKEncoder.ReadLiteralHeaderFieldWithIncrementalIndexing_IndexedName

{
  "tid": 125,
  "div": "HTTP2Stream",
  "msg": "[155] ProcessIncomingFrames - Header Frames: [CommonFrameView [HTTP2FrameHeaderAndPayload 
          Type: HEADERS, 
          Flags: 00000100, 
          StreamId: 155, 
          DontUseMemPool: True, 
          Payload: [BufferSegment Offset: 0 Count: 259 Data: [8A, 61, 96, DD, 6D, 5F, 4A, 08, 0A, 43, 5D, 8A, 08, 02, 6D, 40, B9, 70, 2E, DC, 03, EA, 62, D1, BF, EF, 5C, 85, 75, C6, 9F, 13, DF, EF, 6C, 96, DF, 3D, BF, 4A, 00, 4A, 68, 1F, A5, 04, 01, 34, A0, 1B, B8, D0, 6E, 36, FA, 98, B4, 6F, 62, 9A, FE, 5E, 6C, 4D, 38, EC, 6D, BA, 51, 48, 56, D9, 69, F7, 59, 94, 63, 90, 61, 71, 97, 0A, CB, E0, FE, 7F, C8, 7C, AF, 0A, E1, 52, 51, 49, 23, 31, B8, DB, 2F, 06, 51, C8, D8, C7, 23, ED, 0C, 8F, B4, 39, 1D, 03, 4C, CA, F2, E4, A0, F6, C9, 2D, 87, A9, 2B, D4, 54, A9, FD, 5E, ...]]] 
          Payload: [BufferSegment Offset: 0 Count: 259 Data: [8A, 61, 96, DD, 6D, 5F, 4A, 08, 0A, 43, 5D, 8A, 08, 02, 6D, 40, B9, 70, 2E, DC, 03, EA, 62, D1, BF, EF, 5C, 85, 75, C6, 9F, 13, DF, EF, 6C, 96, DF, 3D, BF, 4A, 00, 4A, 68, 1F, A5, 04, 01, 34, A0, 1B, B8, D0, 6E, 36, FA, 98, B4, 6F, 62, 9A, FE, 5E, 6C, 4D, 38, EC, 6D, BA, 51, 48, 56, D9, 69, F7, 59, 94, 63, 90, 61, 71, 97, 0A, CB, E0, FE, 7F, C8, 7C, AF, 0A, E1, 52, 51, 49, 23, 31, B8, DB, 2F, 06, 51, C8, D8, C7, 23, ED, 0C, 8F, B4, 39, 1D, 03, 4C, CA, F2, E4, A0, F6, C9, 2D, 87, A9, 2B, D4, 54, A9, FD, 5E, ...]]"
}

HPACK Encoder State

Encoder: [
  HeaderTable DynamicTable count: 4, DynamicTableSize: 304, MaxDynamicTableSize: 4096, 
    "range": "bytes=0-14680064", 
    "cookie": "DCL-Session=06d42c4b-7b1e-474c-8fb1-ece6114d9393", 
    "user-agent": "com.Tivadar.Best.HTTP v3.0.14.0/Unity 6000.0.44f1", 
    "accept-encoding": "br, gzip, deflate, identity"
][
  HeaderTable DynamicTable count: 58, DynamicTableSize: 4051, MaxDynamicTableSize: 4096, 
    "via": "1.1 f2dd3b65381fad5aad94ac94ad7043f8.cloudfront.net (CloudFront)", 
    "etag": ""852467b57e2de534973faada1636e390"", 
    "last-modified": "Thu, 02 May 2024 05:41:59 GMT", 
    "content-length": "764928", 
    "date": "Sun, 20 Apr 2025 16:17:09 GMT", 
    "via": "9335f334ea9fb8ca-AMS", 
    "content-range": "bytes 0-893839/893840", 
    "age": "10347", 
    "last-modified": "8OyW2xg3mO3hipFl0ASsK5UEyzMUJhPc0h5YS0Y2QPHiobijkJ9LAQ==", 
    "last-modified": "AMS58-P5", 
    "via": "1.1 3f24561b20ab2825cb11ac40fc1c2434.cloudfront.net (CloudFront)", 
    "etag": "Miss from cloudfront", 
    "etag": ""26b0b49aab735bd77e7a83fbb3dfd045"", 
    "last-modified": "Fri, 03 May 2024 02:45:31 GMT", 
    "content-length": "893840", 
    "strict-transport-security": "9335f33419fbb8ca-AMS", 
    "age": "10491", 
    "strict-transport-security": "9335f33419ffb8ca-AMS", 
    "server": "cloudflare", 
    "x-content-type-options": "nosniff", 
    "strict-transport-security": "max-age=15552000; includeSubDomains; preload", 
    "vary": "Accept-Encoding", 
    "content-range": "bytes 0-754255/754256", 
    "expires": "Mon, 20 Apr 2026 16:17:07 GMT", 
    "age": "1654", 
    "content-range": "OrpQmlv2-dVNfdbW9797M2D7W-rp6oEa_zjSQc_Op6KIveJt-Sp9Kg==", 
    "last-modified": "CDG52-P3", 
    "via": "1.1 0b61c24c96e423ae09ca12c7ee3a196c.cloudfront.net (CloudFront)", 
    "etag": ""93b1c78e9d4f1786515a6891a6a03abb"", 
    "last-modified": "Mon, 29 Apr 2024 14:19:28 GMT", 
    "content-length": "754256", 
    "date": "Sun, 20 Apr 2025 16:17:07 GMT", 
    "cf-ray": "9335f32acaa6b8ca-AMS", 
    "x-amz-cf-id": "Pmgk412R450bd4_wS4FX4zz7TA4LophNAM1XLzSdzz6DlXqtbcabRA==", 
    "etag": "W/"25899db80af1908764f58e7d5f2aaa35"", 
    "last-modified": "Fri, 03 May 2024 02:17:06 GMT", 
    "cf-ray": "9335f32a4a04b8ca-AMS", 
    "x-amz-cf-id": "deL2JvlxcbJDIBUgWf5g9PldjFoCA9qZXSVrOJDx4bmHR1KBUiQ_4Q==", 
    "via": "1.1 261dbbb56805d787e4e94407d9511a46.cloudfront.net (CloudFront)", 
    "etag": "W/"18787bbcd27ab1635585379fe4023b02"", 
    "last-modified": "Fri, 31 Jan 2025 20:11:08 GMT", 
    "cache-control": "public, max-age=31536000", 
    "content-type": "application/wasm", 
    "cf-ray": "9335f32a4a05b8ca-AMS", 
    "content-range": "bytes 0-46782/46783", 
    "cf-cache-status": "HIT", 
    "x-amz-cf-id": "4Ux8N23MjkG1ATJUvx32vmDWoCpv4D6PXgYfPfLAIORL4BeHQLbFBA==", 
    "x-amz-cf-pop": "MAN50-C2", 
    "via": "1.1 bcc4ad8f771bded524c65fdce1b7af50.cloudfront.net (CloudFront)", 
    "etag": ""6a5fbd7ad49373b4af3528cb87b303a2"", 
    "last-modified": "Tue, 28 Jan 2025 05:59:16 GMT", 
    "content-length": "46783", 
    "cf-ray": "9335f32a4a03b8ca-AMS", 
    "content-range": "bytes 0-774719/774720", 
    "age": "10488", 
    "x-amz-cf-id": "867eofm7qXfZWBEpvRfxmsXTGRIYsGJz9WmCmzY3kgtCvHUrWjafyw==", 
    "via": "1.1 2358c92c016b37a066a4efcb933f0da0.cloudfront.net (CloudFront)", 
    "etag": ""027d504685cbfaa78272cb1babf0a7e2""
]

Context Information

{
  "ctx": [
    {
      "TypeName": "HTTP2Stream",
      "Hash": "78a441b0aad40400cb5f0f32669512b9",
      "id": 155
    },
    {
      "TypeName": "HTTP2ContentConsumer",
      "Hash": "77c9cdc537ac83ceac4ace1e0934dd00",
      "Connection": "1f78938c7d97737be801448626e46ca1"
    },
    {
      "TypeName": "HTTPOverTCPConnection", 
      "Hash": "1f78938c7d97737be801448626e46ca1",
      "HostKey": {
        "Uri": "https://ab-cdn.decentraland.org",
        "Proxy": "",
        "Key": "97d76f145d9bbbdce0bc62ac9986cc3f"
      },
      "RequestHandler": "77c9cdc537ac83ceac4ace1e0934dd00"
    },
    {
      "TypeName": "HTTPRequest",
      "Hash": "0648244211426e251866a0e529124c1f",
      "Connection": "1f78938c7d97737be801448626e46ca1"
    },
    {
      "TypeName": "HostVariant",
      "Hash": "7bf54631bbca2c4fd4309662f35e7c25",
      "Host": "ab-cdn.decentraland.org"
    }
  ],
  "t": 638807626295736921,
  "ll": "Exception",
  "bh": 1
}

The consequence

The state of the header table seems to be entirely broken after the exception, and it affects other requests.
Sporadic mix-up of headers values results unpredictably in improper content-length, encoding or other chaotic failures. Other requests can receive chunks of data of improper size (probably because of how BufferSegments are allocated).

E.g.:

{"tid":122,"div":"DecompressorFactory","msg":"GetDecompressor - unsupported encoding '9335f34669659726-AMS'!","stack":" at Response.Decompression.DecompressorFactory.GetDecompressor (System.String encoding, Shared.Logger.LoggingContext context) [0x0009f] in D:\\Decentraland\\unity-explorer-packages\\BestHTTP\\Runtime\\HTTP\\Response\\Decompression\\DecompressorFactory.cs:65 \r at Hosts.Connections.HTTP2.HTTP2Response.ProcessData (Shared.PlatformSupport.Memory.BufferSegment payload) [0x0001f] in D:\\Decentraland\\unity-explorer-packages\\BestHTTP\\Runtime\\HTTP\\Hosts\\Connections\\HTTP2\\HTTP2Response.cs:95 \r at Hosts.Connections.HTTP2.HTTP2Stream.ProcessIncomingDATAFrame (Hosts.Connections.HTTP2.HTTP2FrameHeaderAndPayload& frame) [0x00054] in D:\\Decentraland\\unity-explorer-packages\\BestHTTP\\Runtime\\HTTP\\Hosts\\Connections\\HTTP2\\HTTP2Stream.cs:404 \r at Hosts.Connections.HTTP2.HTTP2Stream.ProcessIncomingFrames (System.Collections.Generic.List`1[T] outgoingFrames) [0x002c0] in D:\\Decentraland\\unity-explorer-packages\\BestHTTP\\Runtime\\HTTP\\Hosts\\Connections\\HTTP2\\HTTP2Stream.cs:346 \r at Hosts.Connections.HTTP2.HTTP2Stream.AddFrame (Hosts.Connections.HTTP2.HTTP2FrameHeaderAndPayload frame, System.Collections.Generic.List`1[T] outgoingFrames) [0x0000e] in D:\\Decentraland\\unity-explorer-packages\\BestHTTP\\Runtime\\HTTP\\Hosts\\Connections\\HTTP2\\HTTP2Stream.cs:211 \r at Hosts.Connections.HTTP2.HTTP2ContentConsumer.RunHandler () [0x0044c] in D:\\Decentraland\\unity-explorer-packages\\BestHTTP\\Runtime\\HTTP\\Hosts\\Connections\\HTTP2\\HTTP2ContentConsumer.cs:262 \r at Hosts.Connections.HTTPOverTCPConnection.ThreadFunc () [0x00001] in D:\\Decentraland\\unity-explorer-packages\\BestHTTP\\Runtime\\HTTP\\Hosts\\Connections\\HTTPOverTCPConnection.cs:366 \r at Shared.PlatformSupport.Threading.ThreadedRunner+<>c__DisplayClass12_0.<RunLongLiving>b__0 (System.Object param) [0x00009] in D:\\Decentraland\\unity-explorer-packages\\BestHTTP\\Runtime\\Shared\\PlatformSupport\\Threading\\ThreadedRunner.cs:78 \r at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) [0x00000] in <314938d17f3848e8ac683e11b27f62ee>:0 \r at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <314938d17f3848e8ac683e11b27f62ee>:0 \r at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <314938d17f3848e8ac683e11b27f62ee>:0 \r at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) [0x00000] in <314938d17f3848e8ac683e11b27f62ee>:0 \r at System.Threading.ThreadHelper.ThreadStart (System.Object obj) [0x00000] in <314938d17f3848e8ac683e11b27f62ee>:0 ","ctx": [{"TypeName": "HTTP2Response", "Hash": "aa16655a4ee403c1ec9fe8a7009d6648"}, {"TypeName": "HTTPRequest", "Hash": "579e644ebb545044fb3b2021e0fa95b4", "Connection": "76ec5921b0810120f17b13e52e4e5a41"}, {"TypeName": "HostVariant", "Hash": "7bf54631bbca2c4fd4309662f35e7c25", "Host": "ab-cdn.decentraland.org"}],"t":638807626301845184,"ll":"Warning","bh":1}

which obviously is not a correct value for "content-encoding"

What I've tried to fix it:

Align indices calculation with HPACK specification (though I am not an expert and haven't delved there deep enough)

public string GetKey(UInt32 index)
{
    // Index 0 is reserved and should never be used
    if (index == 0)
        throw new ArgumentException("Index 0 is reserved in HPACK and must not be used");

    if (index < StaticTable.Length)
        return StaticTable[index];

    // The -1 is critical here - it accounts for the 1-based to 0-based conversion
    int dynamicIndex = (int)(index - StaticTable.Length);
    if (dynamicIndex < 0 || dynamicIndex >= DynamicTable.Count)
        throw new ArgumentOutOfRangeException(nameof(index), $"Invalid dynamic table index: {dynamicIndex} (original index: {index}, static table length: {StaticTable.Length}, dynamic table count: {DynamicTable.Count})");

    return this.DynamicTable[dynamicIndex].Key;
}

And the similar changes in AddKey

Eliminate possible concurrency issues

I wrapped access to the list with a lock

Both of these seem to decrease the entropy (and the frequency) of exceptions, but didn't eliminate them entirely.

Other observations

• It seems to fail only when the dynamic table is almost full
• I am worried about the number of "content-range" headers cached (not sure if they should be somehow handled exclusively according to the RFC)

Waiting for your help, let me know if you need me to conduct more tests, if you have any stable scenario in mind to narrow down the issue.

[Benedicht]

Thanks for the detailed report! Right now i'm out of office for the week, but will try to replicate the error and fix it asap.

I wrapped access to the list with a lock

lock didn't work because execution order of HPACK encode/decode must be done in the same order the server sent them, so handling of incoming http/2 frames are done on a single thread.

Also, because it's depending on previous header inserts, the problem can be elsewhere but failing later in the GetKey call.

I am worried about the number of "content-range" headers cached (not sure if they should be somehow handled exclusively according to the RFC)

The static table is the same for every client, defined by the HTTP/2 RFC, but the dynamic table is there for the server&client to insert and reference in consecutive requests/responses to reduce sent bytes.

[m3taphysics]

Hi @Benedicht do you have any further information on this issue? It is blocking us to the important HTTP/2 upgrade.

[Benedicht]

Sorry, had to line up an other travel, wasn't in my office since.

However, i still went through the http/2 spec and my implementation, and checked my code because it was years ago i had to touch that part.
My implementation seems correct, also tried to make a quick repro code, but it worked correctly.

What would be very helful is if you could make a repro project and send it over to me.