[{"data":1,"prerenderedAt":22084},["ShallowReactive",2],{"blog-list":3},[4,2596,4157,4215,6673,6983,8729,9882,11354,13147,13785,18530,18891,20642,21137,22037],{"id":5,"title":6,"body":7,"book":2585,"date":2586,"description":2587,"extension":2588,"meta":2589,"navigation":587,"path":2590,"seo":2591,"stem":2592,"tags":2593,"__hash__":2595},"blog\u002Fblog\u002Fwhy-i-built-cloudhttp.md","Spreading HttpClient connections across Kubernetes pods",{"type":8,"value":9,"toc":2566},"minimark",[10,15,25,28,40,46,53,69,73,76,156,163,166,169,183,186,190,197,410,430,627,637,648,715,719,724,735,804,808,815,920,923,927,934,1020,1023,1034,1037,1041,1044,1047,1050,1054,1063,1117,1120,1274,1281,1326,1343,1356,1360,1371,1378,1631,1638,1661,1664,1678,1768,1778,1782,1788,1802,1822,1829,2042,2045,2049,2052,2058,2128,2131,2136,2240,2245,2378,2381,2385,2388,2424,2427,2431,2434,2451,2454,2468,2472,2475,2521,2524,2532,2538,2546,2550,2553,2556,2562],[11,12,14],"h2",{"id":13},"background","Background",[16,17,18,19,24],"p",{},"Back in 2024 I wrote ",[20,21,23],"a",{"href":22},"\u002Fblog\u002Fload-balancing-long-lived-connections-in-kubernetes","Load Balancing Long Lived Connections in Kubernetes",". The short version: long-lived HTTP\u002F2 connections from a .NET service to a Kubernetes Service do not spread across the upstream pods the way most people assume. DNS is consulted once. kube-proxy picks a backend. After that, every request rides the same TCP connection to the same pod until the handler decides to recycle.",[16,26,27],{},"That post ended with four workarounds: a client-side load balancer written by hand, a service mesh, scaling down so it stops mattering, or shortening the pooled connection lifetime. Useful, but the first is a lot of code to maintain, and the others involve tradeoffs that not every team wants.",[16,29,30,31,35,36,39],{},"The lightweight option I always reached for in real services was the one I never properly packaged: register N named ",[32,33,34],"code",{},"HttpClient"," instances for the same logical upstream, give each its own ",[32,37,38],{},"SocketsHttpHandler"," and its own connection pool, then round-robin (or weight, or health-check) across them. Each pool independently does its own DNS lookup, opens its own TCP connection, and lands on whichever backend kube-proxy hands it. Not perfect, but each pool gets a fresh chance at the load balancer.",[16,41,42,43,45],{},"That hand-written pattern is what CloudHttp is now. Plus the ",[32,44,38],{}," defaults I keep typing in every new service, plus a few small ergonomic helpers around it.",[16,47,48],{},[20,49,50],{"href":50,"rel":51},"https:\u002F\u002Fgithub.com\u002Fhaiilong\u002FCloudHttp",[52],"nofollow",[54,55,60],"pre",{"className":56,"code":57,"language":58,"meta":59,"style":59},"language-sh shiki shiki-themes one-light one-dark-pro","dotnet add package haiilong.http.extensions\n","sh","",[32,61,62],{"__ignoreMap":59},[63,64,67],"span",{"class":65,"line":66},"line",1,[63,68,57],{},[11,70,72],{"id":71},"the-problem-more-concretely","The problem more concretely",[16,74,75],{},"A typical .NET service in Kubernetes calls another service through a cluster DNS name:",[54,77,81],{"className":78,"code":79,"language":80,"meta":59,"style":59},"language-csharp shiki shiki-themes one-light one-dark-pro","services.AddHttpClient\u003CInventoryClient>(c =>\n{\n    c.BaseAddress = new Uri(\"https:\u002F\u002Finventory.svc.cluster.local\");\n});\n","csharp",[32,82,83,113,119,150],{"__ignoreMap":59},[63,84,85,89,93,97,100,104,107,110],{"class":65,"line":66},[63,86,88],{"class":87},"s7GmK","services",[63,90,92],{"class":91},"s5ixo",".",[63,94,96],{"class":95},"sAdtL","AddHttpClient",[63,98,99],{"class":91},"\u003C",[63,101,103],{"class":102},"sC09Y","InventoryClient",[63,105,106],{"class":91},">(",[63,108,109],{"class":87},"c",[63,111,112],{"class":91}," =>\n",[63,114,116],{"class":65,"line":115},2,[63,117,118],{"class":91},"{\n",[63,120,122,125,127,130,134,137,140,143,147],{"class":65,"line":121},3,[63,123,124],{"class":87},"    c",[63,126,92],{"class":91},[63,128,129],{"class":87},"BaseAddress",[63,131,133],{"class":132},"sknuh"," =",[63,135,136],{"class":91}," new ",[63,138,139],{"class":102},"Uri",[63,141,142],{"class":91},"(",[63,144,146],{"class":145},"sDhpE","\"https:\u002F\u002Finventory.svc.cluster.local\"",[63,148,149],{"class":91},");\n",[63,151,153],{"class":65,"line":152},4,[63,154,155],{"class":91},"});\n",[16,157,158,159,162],{},"DNS resolves the Service to the kube-proxy ClusterIP. kube-proxy probabilistically picks one backend pod. The TCP connection is set up. ",[32,160,161],{},"IHttpClientFactory"," keeps the handler alive for two minutes by default. With HTTP\u002F2 multiplexing, every request from this caller pod rides the same TCP connection for those two minutes, and every one of those requests lands on the same backend pod.",[16,164,165],{},"Even if the upstream Service has ten replicas, this caller-pod-to-upstream-replica edge is sticky.",[16,167,168],{},"Two reasonable workarounds at the connection layer:",[170,171,172,180],"ol",{},[173,174,175,176,179],"li",{},"Shorten the pool lifetime so the connection rotates more often, which gives kube-proxy a fresh chance to pick a different backend. This is what ",[32,177,178],{},"PooledConnectionLifetime = 2 minutes"," does.",[173,181,182],{},"Run several connection pools in parallel so different requests from the same caller pod can ride different connections. Three pools means three chances at the load balancer instead of one.",[16,184,185],{},"CloudHttp does the second, and pulls in the first for free via the cloud-tuned defaults.",[11,187,189],{"id":188},"wiring-it-up","Wiring it up",[16,191,192,193,196],{},"The entry point is ",[32,194,195],{},"DistributedHttpClient",". Register it like a normal HTTP client, but with a count:",[54,198,200],{"className":78,"code":199,"language":80,"meta":59,"style":59},"builder.Services.AddDistributedHttpClient(\n    name: \"inventory\",\n    configureOptions: opts =>\n    {\n        opts.Mode = DistributionMode.RoundRobin;\n        opts.ClientCount = 4;\n    },\n    configureClient: c =>\n    {\n        c.BaseAddress = new Uri(\"https:\u002F\u002Finventory.svc.cluster.local\");\n        c.DefaultRequestHeaders.Accept.Add(new(\"application\u002Fjson\"));\n    },\n    configureBuilder: clientBuilder =>\n    {\n        clientBuilder.AddStandardResilienceHandler();\n    });\n",[32,201,202,220,234,246,251,275,293,299,311,316,338,367,372,385,390,404],{"__ignoreMap":59},[63,203,204,207,209,212,214,217],{"class":65,"line":66},[63,205,206],{"class":87},"builder",[63,208,92],{"class":91},[63,210,211],{"class":87},"Services",[63,213,92],{"class":91},[63,215,216],{"class":95},"AddDistributedHttpClient",[63,218,219],{"class":91},"(\n",[63,221,222,225,228,231],{"class":65,"line":115},[63,223,224],{"class":87},"    name",[63,226,227],{"class":91},": ",[63,229,230],{"class":145},"\"inventory\"",[63,232,233],{"class":91},",\n",[63,235,236,239,241,244],{"class":65,"line":121},[63,237,238],{"class":87},"    configureOptions",[63,240,227],{"class":91},[63,242,243],{"class":87},"opts",[63,245,112],{"class":91},[63,247,248],{"class":65,"line":152},[63,249,250],{"class":91},"    {\n",[63,252,254,257,259,262,264,267,269,272],{"class":65,"line":253},5,[63,255,256],{"class":87},"        opts",[63,258,92],{"class":91},[63,260,261],{"class":87},"Mode",[63,263,133],{"class":132},[63,265,266],{"class":87}," DistributionMode",[63,268,92],{"class":91},[63,270,271],{"class":87},"RoundRobin",[63,273,274],{"class":91},";\n",[63,276,278,280,282,285,287,291],{"class":65,"line":277},6,[63,279,256],{"class":87},[63,281,92],{"class":91},[63,283,284],{"class":87},"ClientCount",[63,286,133],{"class":132},[63,288,290],{"class":289},"sAGMh"," 4",[63,292,274],{"class":91},[63,294,296],{"class":65,"line":295},7,[63,297,298],{"class":91},"    },\n",[63,300,302,305,307,309],{"class":65,"line":301},8,[63,303,304],{"class":87},"    configureClient",[63,306,227],{"class":91},[63,308,109],{"class":87},[63,310,112],{"class":91},[63,312,314],{"class":65,"line":313},9,[63,315,250],{"class":91},[63,317,319,322,324,326,328,330,332,334,336],{"class":65,"line":318},10,[63,320,321],{"class":87},"        c",[63,323,92],{"class":91},[63,325,129],{"class":87},[63,327,133],{"class":132},[63,329,136],{"class":91},[63,331,139],{"class":102},[63,333,142],{"class":91},[63,335,146],{"class":145},[63,337,149],{"class":91},[63,339,341,343,345,348,350,353,355,358,361,364],{"class":65,"line":340},11,[63,342,321],{"class":87},[63,344,92],{"class":91},[63,346,347],{"class":87},"DefaultRequestHeaders",[63,349,92],{"class":91},[63,351,352],{"class":87},"Accept",[63,354,92],{"class":91},[63,356,357],{"class":95},"Add",[63,359,360],{"class":91},"(new(",[63,362,363],{"class":145},"\"application\u002Fjson\"",[63,365,366],{"class":91},"));\n",[63,368,370],{"class":65,"line":369},12,[63,371,298],{"class":91},[63,373,375,378,380,383],{"class":65,"line":374},13,[63,376,377],{"class":87},"    configureBuilder",[63,379,227],{"class":91},[63,381,382],{"class":87},"clientBuilder",[63,384,112],{"class":91},[63,386,388],{"class":65,"line":387},14,[63,389,250],{"class":91},[63,391,393,396,398,401],{"class":65,"line":392},15,[63,394,395],{"class":87},"        clientBuilder",[63,397,92],{"class":91},[63,399,400],{"class":95},"AddStandardResilienceHandler",[63,402,403],{"class":91},"();\n",[63,405,407],{"class":65,"line":406},16,[63,408,409],{"class":91},"    });\n",[16,411,412,413,416,417,420,421,423,424,426,427,429],{},"Under the hood this creates four named clients (",[32,414,415],{},"inventory#0"," through ",[32,418,419],{},"inventory#3","), each with its own ",[32,422,38],{}," and its own connection pool. The ",[32,425,195],{}," itself is registered as a keyed singleton, and you inject it where you would normally inject ",[32,428,34],{},":",[54,431,433],{"className":78,"code":432,"language":80,"meta":59,"style":59},"public sealed class InventoryService(\n    [FromKeyedServices(\"inventory\")] DistributedHttpClient http)\n{\n    public Task\u003CStockLevel?> GetStockAsync(string sku, CancellationToken ct)\n    {\n        var path = HttpRouteBuilder.BuildPath(\n            \"\u002Fstock\u002F{sku}\",\n            new Dictionary\u003Cstring, object?> { [\"sku\"] = sku });\n\n        return http.GetAsync\u003CStockLevel>(path, ct);\n    }\n}\n",[32,434,435,452,475,479,517,521,542,549,583,589,617,622],{"__ignoreMap":59},[63,436,437,441,444,447,450],{"class":65,"line":66},[63,438,440],{"class":439},"sLKXg","public",[63,442,443],{"class":439}," sealed",[63,445,446],{"class":439}," class",[63,448,449],{"class":102}," InventoryService",[63,451,219],{"class":91},[63,453,454,457,460,462,464,467,469,472],{"class":65,"line":115},[63,455,456],{"class":91},"    [",[63,458,459],{"class":102},"FromKeyedServices",[63,461,142],{"class":91},[63,463,230],{"class":145},[63,465,466],{"class":91},")] ",[63,468,195],{"class":102},[63,470,471],{"class":87}," http",[63,473,474],{"class":91},")\n",[63,476,477],{"class":65,"line":121},[63,478,118],{"class":91},[63,480,481,484,487,489,492,495,498,500,503,506,509,512,515],{"class":65,"line":152},[63,482,483],{"class":439},"    public",[63,485,486],{"class":102}," Task",[63,488,99],{"class":91},[63,490,491],{"class":102},"StockLevel",[63,493,494],{"class":91},"?> ",[63,496,497],{"class":95},"GetStockAsync",[63,499,142],{"class":91},[63,501,502],{"class":439},"string",[63,504,505],{"class":87}," sku",[63,507,508],{"class":91},", ",[63,510,511],{"class":102},"CancellationToken",[63,513,514],{"class":87}," ct",[63,516,474],{"class":91},[63,518,519],{"class":65,"line":253},[63,520,250],{"class":91},[63,522,523,526,530,532,535,537,540],{"class":65,"line":277},[63,524,525],{"class":439},"        var",[63,527,529],{"class":528},"sz0mV"," path",[63,531,133],{"class":132},[63,533,534],{"class":87}," HttpRouteBuilder",[63,536,92],{"class":91},[63,538,539],{"class":95},"BuildPath",[63,541,219],{"class":91},[63,543,544,547],{"class":65,"line":295},[63,545,546],{"class":145},"            \"\u002Fstock\u002F{sku}\"",[63,548,233],{"class":91},[63,550,551,554,557,559,561,563,566,569,572,575,578,580],{"class":65,"line":301},[63,552,553],{"class":91},"            new ",[63,555,556],{"class":102},"Dictionary",[63,558,99],{"class":91},[63,560,502],{"class":439},[63,562,508],{"class":91},[63,564,565],{"class":439},"object",[63,567,568],{"class":91},"?> { [",[63,570,571],{"class":145},"\"sku\"",[63,573,574],{"class":91},"] ",[63,576,577],{"class":132},"=",[63,579,505],{"class":528},[63,581,582],{"class":91}," });\n",[63,584,585],{"class":65,"line":313},[63,586,588],{"emptyLinePlaceholder":587},true,"\n",[63,590,591,594,596,598,601,603,605,607,610,612,615],{"class":65,"line":318},[63,592,593],{"class":439},"        return",[63,595,471],{"class":87},[63,597,92],{"class":91},[63,599,600],{"class":95},"GetAsync",[63,602,99],{"class":91},[63,604,491],{"class":102},[63,606,106],{"class":91},[63,608,609],{"class":528},"path",[63,611,508],{"class":91},[63,613,614],{"class":528},"ct",[63,616,149],{"class":91},[63,618,619],{"class":65,"line":340},[63,620,621],{"class":91},"    }\n",[63,623,624],{"class":65,"line":369},[63,625,626],{"class":91},"}\n",[16,628,629,630,633,634,636],{},"Every call to ",[32,631,632],{},"http.GetAsync\u003CT>(...)"," picks one of the four underlying named clients based on the distribution mode, sends the request, and returns. From the caller's perspective, this looks identical to a normal ",[32,635,34],{}," call.",[16,638,639,640,643,644,647],{},"The four underlying handlers each get the cloud-tuned defaults from ",[32,641,642],{},"ConfigureForCloud()",". If you want to tweak them per pool, there is a ",[32,645,646],{},"configurePrimaryHandler"," callback that runs once for each handler, after the defaults:",[54,649,651],{"className":78,"code":650,"language":80,"meta":59,"style":59},"configurePrimaryHandler: handler =>\n{\n    handler.ConnectTimeout = TimeSpan.FromSeconds(3);\n    handler.MaxConnectionsPerServer = 200;\n}\n",[32,652,653,664,668,695,711],{"__ignoreMap":59},[63,654,655,657,659,662],{"class":65,"line":66},[63,656,646],{"class":528},[63,658,227],{"class":91},[63,660,661],{"class":87},"handler",[63,663,112],{"class":91},[63,665,666],{"class":65,"line":115},[63,667,118],{"class":91},[63,669,670,673,675,678,680,683,685,688,690,693],{"class":65,"line":121},[63,671,672],{"class":87},"    handler",[63,674,92],{"class":91},[63,676,677],{"class":87},"ConnectTimeout",[63,679,133],{"class":132},[63,681,682],{"class":87}," TimeSpan",[63,684,92],{"class":91},[63,686,687],{"class":95},"FromSeconds",[63,689,142],{"class":91},[63,691,692],{"class":289},"3",[63,694,149],{"class":91},[63,696,697,699,701,704,706,709],{"class":65,"line":152},[63,698,672],{"class":87},[63,700,92],{"class":91},[63,702,703],{"class":87},"MaxConnectionsPerServer",[63,705,133],{"class":132},[63,707,708],{"class":289}," 200",[63,710,274],{"class":91},[63,712,713],{"class":65,"line":253},[63,714,626],{"class":91},[11,716,718],{"id":717},"the-three-distribution-modes","The three distribution modes",[720,721,723],"h3",{"id":722},"round-robin","Round-robin",[16,725,726,727,730,731,734],{},"This is the default mode. An atomic counter increments on every call and the index is ",[32,728,729],{},"counter % ClientCount",". The increment is lock-free via ",[32,732,733],{},"Interlocked.Increment",", with no allocation and no tuning required. Round-robin is the right choice when the upstream pods are interchangeable and roughly equal in capacity.",[54,736,738],{"className":78,"code":737,"language":80,"meta":59,"style":59},"services.AddRoundRobinDistribution(\n    name: \"payments\",\n    clientCount: 4,\n    configureClient: c => c.BaseAddress = new Uri(\"https:\u002F\u002Fpayments.svc.cluster.local\"));\n",[32,739,740,751,762,774],{"__ignoreMap":59},[63,741,742,744,746,749],{"class":65,"line":66},[63,743,88],{"class":87},[63,745,92],{"class":91},[63,747,748],{"class":95},"AddRoundRobinDistribution",[63,750,219],{"class":91},[63,752,753,755,757,760],{"class":65,"line":115},[63,754,224],{"class":87},[63,756,227],{"class":91},[63,758,759],{"class":145},"\"payments\"",[63,761,233],{"class":91},[63,763,764,767,769,772],{"class":65,"line":121},[63,765,766],{"class":87},"    clientCount",[63,768,227],{"class":91},[63,770,771],{"class":289},"4",[63,773,233],{"class":91},[63,775,776,778,780,782,785,787,789,791,793,795,797,799,802],{"class":65,"line":152},[63,777,304],{"class":87},[63,779,227],{"class":91},[63,781,109],{"class":87},[63,783,784],{"class":91}," => ",[63,786,109],{"class":87},[63,788,92],{"class":91},[63,790,129],{"class":87},[63,792,133],{"class":132},[63,794,136],{"class":91},[63,796,139],{"class":102},[63,798,142],{"class":91},[63,800,801],{"class":145},"\"https:\u002F\u002Fpayments.svc.cluster.local\"",[63,803,366],{"class":91},[720,805,807],{"id":806},"weighted","Weighted",[16,809,810,811,814],{},"Weighted distribution is useful for canary deployments or mixed-capacity pools. Each client index gets a relative weight, and selection is ",[32,812,813],{},"Random.Shared.NextDouble() * totalWeight"," plus a binary search into a sorted cumulative ladder.",[54,816,818],{"className":78,"code":817,"language":80,"meta":59,"style":59},"services.AddWeightedDistribution(\n    name: \"search\",\n    weights: new Dictionary\u003Cint, double> { [0] = 9, [1] = 1 },\n    configureClient: c => c.BaseAddress = new Uri(\"https:\u002F\u002Fsearch.svc.cluster.local\"));\n",[32,819,820,831,842,891],{"__ignoreMap":59},[63,821,822,824,826,829],{"class":65,"line":66},[63,823,88],{"class":87},[63,825,92],{"class":91},[63,827,828],{"class":95},"AddWeightedDistribution",[63,830,219],{"class":91},[63,832,833,835,837,840],{"class":65,"line":115},[63,834,224],{"class":87},[63,836,227],{"class":91},[63,838,839],{"class":145},"\"search\"",[63,841,233],{"class":91},[63,843,844,847,850,852,854,857,859,862,865,868,870,872,875,878,881,883,885,888],{"class":65,"line":121},[63,845,846],{"class":87},"    weights",[63,848,849],{"class":91},": new ",[63,851,556],{"class":102},[63,853,99],{"class":91},[63,855,856],{"class":439},"int",[63,858,508],{"class":91},[63,860,861],{"class":439},"double",[63,863,864],{"class":91},"> { [",[63,866,867],{"class":289},"0",[63,869,574],{"class":91},[63,871,577],{"class":132},[63,873,874],{"class":289}," 9",[63,876,877],{"class":91},", [",[63,879,880],{"class":289},"1",[63,882,574],{"class":91},[63,884,577],{"class":132},[63,886,887],{"class":289}," 1",[63,889,890],{"class":91}," },\n",[63,892,893,895,897,899,901,903,905,907,909,911,913,915,918],{"class":65,"line":152},[63,894,304],{"class":87},[63,896,227],{"class":91},[63,898,109],{"class":87},[63,900,784],{"class":91},[63,902,109],{"class":87},[63,904,92],{"class":91},[63,906,129],{"class":87},[63,908,133],{"class":132},[63,910,136],{"class":91},[63,912,139],{"class":102},[63,914,142],{"class":91},[63,916,917],{"class":145},"\"https:\u002F\u002Fsearch.svc.cluster.local\"",[63,919,366],{"class":91},[16,921,922],{},"That example sends roughly 10% of traffic to the second pool. Useful when you want to dip into a different upstream gradually: a different cluster, a different version of a service, a different region.",[720,924,926],{"id":925},"health-aware","Health-aware",[16,928,929,930,933],{},"Health-aware mode is round-robin with a temporary degraded list. When a pool returns a transient status code or throws a transient exception, CloudHttp records ",[32,931,932],{},"now + HealthDegradedTimeout"," for that pool and skips it on subsequent picks until the timestamp expires. The default timeout is 30 seconds.",[54,935,937],{"className":78,"code":936,"language":80,"meta":59,"style":59},"services.AddHealthAwareDistribution(\n    name: \"inventory\",\n    clientCount: 4,\n    degradedTimeout: TimeSpan.FromSeconds(30),\n    configureClient: c => c.BaseAddress = new Uri(\"https:\u002F\u002Finventory.svc.cluster.local\"));\n",[32,938,939,950,960,970,992],{"__ignoreMap":59},[63,940,941,943,945,948],{"class":65,"line":66},[63,942,88],{"class":87},[63,944,92],{"class":91},[63,946,947],{"class":95},"AddHealthAwareDistribution",[63,949,219],{"class":91},[63,951,952,954,956,958],{"class":65,"line":115},[63,953,224],{"class":87},[63,955,227],{"class":91},[63,957,230],{"class":145},[63,959,233],{"class":91},[63,961,962,964,966,968],{"class":65,"line":121},[63,963,766],{"class":87},[63,965,227],{"class":91},[63,967,771],{"class":289},[63,969,233],{"class":91},[63,971,972,975,977,980,982,984,986,989],{"class":65,"line":152},[63,973,974],{"class":87},"    degradedTimeout",[63,976,227],{"class":91},[63,978,979],{"class":87},"TimeSpan",[63,981,92],{"class":91},[63,983,687],{"class":95},[63,985,142],{"class":91},[63,987,988],{"class":289},"30",[63,990,991],{"class":91},"),\n",[63,993,994,996,998,1000,1002,1004,1006,1008,1010,1012,1014,1016,1018],{"class":65,"line":253},[63,995,304],{"class":87},[63,997,227],{"class":91},[63,999,109],{"class":87},[63,1001,784],{"class":91},[63,1003,109],{"class":87},[63,1005,92],{"class":91},[63,1007,129],{"class":87},[63,1009,133],{"class":132},[63,1011,136],{"class":91},[63,1013,139],{"class":102},[63,1015,142],{"class":91},[63,1017,146],{"class":145},[63,1019,366],{"class":91},[16,1021,1022],{},"Recovery is purely time-based. There is no positive \"this pool is healthy again\" signal that immediately reinstates it. Concurrent in-flight requests complete out of order, and a stale healthy marker should not overwrite a newer degraded one.",[16,1024,1025,1026,1029,1030,1033],{},"The clock is ",[32,1027,1028],{},"Environment.TickCount64"," rather than ",[32,1031,1032],{},"DateTime.UtcNow",", so NTP nudges to the wall clock do not corrupt the degradation state.",[16,1035,1036],{},"If every pool ends up degraded at the same time, the selector falls back to round-robin instead of failing outright. Calling a possibly-degraded pool is better than refusing to call anything.",[11,1038,1040],{"id":1039},"a-reality-check-on-what-this-can-do","A reality check on what this can do",[16,1042,1043],{},"Several sentences in this post say \"more chances at the load balancer\" rather than \"always different backends\". That hedging is real and worth being honest about.",[16,1045,1046],{},"The N-pool setup gives kube-proxy N opportunities to pick different backends. If the upstream Service has only two replicas and you have four pools, by the pigeonhole principle at least two pools share a backend. If kube-proxy's hash function happens to map two pools to the same backend, you live with that for the next connection lifetime.",[16,1048,1049],{},"A proper service mesh sidecar (Envoy under Istio, the Linkerd2-proxy) does L7 client-side load balancing and will distribute each request reliably. If you have a mesh, prefer the mesh. CloudHttp is the workaround for when you do not have one, cannot write a custom client-side load balancer, and still want better odds than a single TCP connection's worth of luck.",[11,1051,1053],{"id":1052},"cloud-tuned-handler-defaults","Cloud-tuned handler defaults",[16,1055,1056,1057,1059,1060,1062],{},"The other thing the library does is bundle up the ",[32,1058,38],{}," defaults I have ended up typing into every cloud service for the last few years. They are applied automatically inside ",[32,1061,216],{},", but you can also use them on their own with a normal named client:",[54,1064,1066],{"className":78,"code":1065,"language":80,"meta":59,"style":59},"services.AddHttpClient(\"orders\", c => c.BaseAddress = new Uri(\"https:\u002F\u002Forders.svc\"))\n    .ConfigureForCloud();\n",[32,1067,1068,1107],{"__ignoreMap":59},[63,1069,1070,1072,1074,1076,1078,1081,1083,1085,1087,1089,1091,1093,1095,1097,1099,1101,1104],{"class":65,"line":66},[63,1071,88],{"class":87},[63,1073,92],{"class":91},[63,1075,96],{"class":95},[63,1077,142],{"class":91},[63,1079,1080],{"class":145},"\"orders\"",[63,1082,508],{"class":91},[63,1084,109],{"class":87},[63,1086,784],{"class":91},[63,1088,109],{"class":87},[63,1090,92],{"class":91},[63,1092,129],{"class":87},[63,1094,133],{"class":132},[63,1096,136],{"class":91},[63,1098,139],{"class":102},[63,1100,142],{"class":91},[63,1102,1103],{"class":145},"\"https:\u002F\u002Forders.svc\"",[63,1105,1106],{"class":91},"))\n",[63,1108,1109,1112,1115],{"class":65,"line":115},[63,1110,1111],{"class":91},"    .",[63,1113,1114],{"class":95},"ConfigureForCloud",[63,1116,403],{"class":91},[16,1118,1119],{},"The values it picks, and why:",[1121,1122,1123,1139],"table",{},[1124,1125,1126],"thead",{},[1127,1128,1129,1133,1136],"tr",{},[1130,1131,1132],"th",{},"Property",[1130,1134,1135],{},"Value",[1130,1137,1138],{},"What it gets you",[1140,1141,1142,1156,1168,1180,1197,1210,1223,1236,1249,1262],"tbody",{},[1127,1143,1144,1150,1153],{},[1145,1146,1147],"td",{},[32,1148,1149],{},"PooledConnectionLifetime",[1145,1151,1152],{},"2 minutes",[1145,1154,1155],{},"DNS refresh on the cadence of rolling deploys; the .NET default is infinite, which means a connection opened on day 1 still hits the same pod on day 30.",[1127,1157,1158,1162,1165],{},[1145,1159,1160],{},[32,1161,677],{},[1145,1163,1164],{},"5 seconds",[1145,1166,1167],{},"Fail fast on broken routes. The .NET default is also infinite, which is the wrong shape for cluster traffic.",[1127,1169,1170,1174,1177],{},[1145,1171,1172],{},[32,1173,703],{},[1145,1175,1176],{},"100",[1145,1178,1179],{},"Bounded concurrency per origin, with enough headroom for bursty service traffic.",[1127,1181,1182,1187,1190],{},[1145,1183,1184],{},[32,1185,1186],{},"AutomaticDecompression",[1145,1188,1189],{},"All",[1145,1191,1192,1193,1196],{},"gzip, deflate, brotli (and zstd on .NET 10). Adds ",[32,1194,1195],{},"Accept-Encoding"," automatically.",[1127,1198,1199,1204,1207],{},[1145,1200,1201],{},[32,1202,1203],{},"EnableMultipleHttp2Connections",[1145,1205,1206],{},"true",[1145,1208,1209],{},"Lets the handler open another HTTP\u002F2 connection when stream limits saturate.",[1127,1211,1212,1217,1220],{},[1145,1213,1214],{},[32,1215,1216],{},"InitialHttp2StreamWindowSize",[1145,1218,1219],{},"128 KiB",[1145,1221,1222],{},"Larger per-stream flow-control window. Fewer round trips on non-trivial response bodies.",[1127,1224,1225,1230,1233],{},[1145,1226,1227],{},[32,1228,1229],{},"KeepAlivePingDelay",[1145,1231,1232],{},"30 seconds",[1145,1234,1235],{},"Detect dead connections proactively. The default is infinite, meaning no pings at all.",[1127,1237,1238,1243,1246],{},[1145,1239,1240],{},[32,1241,1242],{},"KeepAlivePingTimeout",[1145,1244,1245],{},"10 seconds",[1145,1247,1248],{},"How long to wait for a pong before declaring the connection dead.",[1127,1250,1251,1256,1259],{},[1145,1252,1253],{},[32,1254,1255],{},"KeepAlivePingPolicy",[1145,1257,1258],{},"WithActiveRequests",[1145,1260,1261],{},"Only ping while requests are in flight. The default pings idle connections too, which is wasted work.",[1127,1263,1264,1269,1271],{},[1145,1265,1266],{},[32,1267,1268],{},"ResponseDrainTimeout",[1145,1270,1164],{},[1145,1272,1273],{},"Bound the time spent draining an unread response body when a request is disposed. The .NET default is 2 seconds; 5 is more pool-friendly.",[16,1275,1276,1277,1280],{},"The builder version (",[32,1278,1279],{},"IHttpClientBuilder.ConfigureForCloud()",") adds two more knobs that only make sense at the factory layer:",[1121,1282,1283,1293],{},[1124,1284,1285],{},[1127,1286,1287,1289,1291],{},[1130,1288,1132],{},[1130,1290,1135],{},[1130,1292,1138],{},[1140,1294,1295,1307],{},[1127,1296,1297,1302,1304],{},[1145,1298,1299],{},[32,1300,1301],{},"HttpClient.Timeout",[1145,1303,1232],{},[1145,1305,1306],{},"Bounded total per-request time. The .NET default is 100 seconds, which is far too long for cluster-internal calls.",[1127,1308,1309,1312,1317],{},[1145,1310,1311],{},"Factory handler lifetime",[1145,1313,1314],{},[32,1315,1316],{},"Timeout.InfiniteTimeSpan",[1145,1318,1319,1320,1322,1323,1325],{},"Stops ",[32,1321,161],{}," from rotating handlers on its own schedule. ",[32,1324,1149],{}," handles connection recycling instead.",[16,1327,1328,1329,1331,1332,1334,1335,1339,1340,1342],{},"The last one matters more than it looks. By default, ",[32,1330,161],{}," rotates the whole handler every two minutes for DNS-refresh reasons. If you also set ",[32,1333,178],{}," on the handler, you are paying twice: the factory throws the whole pool away every two minutes, AND each connection has its own two-minute clock that never gets to run because the handler dies first. I wrote about this in more detail in ",[20,1336,1338],{"href":1337},"\u002Fblog\u002Fhttpclient-connection-lifetime-observed","HttpClient connection lifetime, observed","; the short version is \"pin the factory lifetime to infinite, let ",[32,1341,1149],{}," do the rotation\".",[16,1344,1345,1346,1351,1352,1355],{},"Each setting has a longer explanation in ",[20,1347,1350],{"href":1348,"rel":1349},"https:\u002F\u002Fgithub.com\u002Fhaiilong\u002FCloudHttp\u002Fblob\u002Fmain\u002Fdocs\u002Fcloud-defaults.md",[52],"docs\u002Fcloud-defaults.md"," inside the repo. Every default is overridable through the ",[32,1353,1354],{},"customize"," callback if your case is different.",[11,1357,1359],{"id":1358},"composition-with-microsoftextensionshttpresilience","Composition with Microsoft.Extensions.Http.Resilience",[16,1361,1362,1363,1370],{},"CloudHttp does pool selection. It deliberately does not do retries, backoff, jitter, or circuit breaking. Microsoft's ",[20,1364,1367],{"href":1365,"rel":1366},"https:\u002F\u002Flearn.microsoft.com\u002Fdotnet\u002Fcore\u002Fresilience\u002Fhttp-resilience",[52],[32,1368,1369],{},"Microsoft.Extensions.Http.Resilience"," package does all of that on top of Polly v8, and it does it well.",[16,1372,1373,1374,1377],{},"The two libraries compose cleanly. The ",[32,1375,1376],{},"configureBuilder"," callback runs per underlying named client, so the resilience handler attaches inside each pool:",[54,1379,1381],{"className":78,"code":1380,"language":80,"meta":59,"style":59},"services.AddDistributedHttpClient(\"payments\",\n    configureOptions: opts => opts.ClientCount = 4,\n    configureClient: c => c.BaseAddress = new Uri(\"https:\u002F\u002Fpayments.svc\"),\n    configureBuilder: cb => cb.AddStandardResilienceHandler(o =>\n    {\n        o.Retry.MaxRetryAttempts = 3;\n        o.Retry.UseJitter = true;\n        o.CircuitBreaker.FailureRatio = 0.2;\n        o.CircuitBreaker.BreakDuration = TimeSpan.FromSeconds(30);\n\n        o.AttemptTimeout.Timeout = TimeSpan.FromSeconds(5);\n        o.TotalRequestTimeout.Timeout = TimeSpan.FromSeconds(30);\n    }));\n",[32,1382,1383,1397,1419,1448,1472,1476,1498,1518,1539,1566,1570,1599,1626],{"__ignoreMap":59},[63,1384,1385,1387,1389,1391,1393,1395],{"class":65,"line":66},[63,1386,88],{"class":87},[63,1388,92],{"class":91},[63,1390,216],{"class":95},[63,1392,142],{"class":91},[63,1394,759],{"class":145},[63,1396,233],{"class":91},[63,1398,1399,1401,1403,1405,1407,1409,1411,1413,1415,1417],{"class":65,"line":115},[63,1400,238],{"class":87},[63,1402,227],{"class":91},[63,1404,243],{"class":87},[63,1406,784],{"class":91},[63,1408,243],{"class":87},[63,1410,92],{"class":91},[63,1412,284],{"class":87},[63,1414,133],{"class":132},[63,1416,290],{"class":289},[63,1418,233],{"class":91},[63,1420,1421,1423,1425,1427,1429,1431,1433,1435,1437,1439,1441,1443,1446],{"class":65,"line":121},[63,1422,304],{"class":87},[63,1424,227],{"class":91},[63,1426,109],{"class":87},[63,1428,784],{"class":91},[63,1430,109],{"class":87},[63,1432,92],{"class":91},[63,1434,129],{"class":87},[63,1436,133],{"class":132},[63,1438,136],{"class":91},[63,1440,139],{"class":102},[63,1442,142],{"class":91},[63,1444,1445],{"class":145},"\"https:\u002F\u002Fpayments.svc\"",[63,1447,991],{"class":91},[63,1449,1450,1452,1454,1457,1459,1461,1463,1465,1467,1470],{"class":65,"line":152},[63,1451,377],{"class":87},[63,1453,227],{"class":91},[63,1455,1456],{"class":87},"cb",[63,1458,784],{"class":91},[63,1460,1456],{"class":87},[63,1462,92],{"class":91},[63,1464,400],{"class":95},[63,1466,142],{"class":91},[63,1468,1469],{"class":87},"o",[63,1471,112],{"class":91},[63,1473,1474],{"class":65,"line":253},[63,1475,250],{"class":91},[63,1477,1478,1481,1483,1486,1488,1491,1493,1496],{"class":65,"line":277},[63,1479,1480],{"class":87},"        o",[63,1482,92],{"class":91},[63,1484,1485],{"class":87},"Retry",[63,1487,92],{"class":91},[63,1489,1490],{"class":87},"MaxRetryAttempts",[63,1492,133],{"class":132},[63,1494,1495],{"class":289}," 3",[63,1497,274],{"class":91},[63,1499,1500,1502,1504,1506,1508,1511,1513,1516],{"class":65,"line":295},[63,1501,1480],{"class":87},[63,1503,92],{"class":91},[63,1505,1485],{"class":87},[63,1507,92],{"class":91},[63,1509,1510],{"class":87},"UseJitter",[63,1512,133],{"class":132},[63,1514,1515],{"class":289}," true",[63,1517,274],{"class":91},[63,1519,1520,1522,1524,1527,1529,1532,1534,1537],{"class":65,"line":301},[63,1521,1480],{"class":87},[63,1523,92],{"class":91},[63,1525,1526],{"class":87},"CircuitBreaker",[63,1528,92],{"class":91},[63,1530,1531],{"class":87},"FailureRatio",[63,1533,133],{"class":132},[63,1535,1536],{"class":289}," 0.2",[63,1538,274],{"class":91},[63,1540,1541,1543,1545,1547,1549,1552,1554,1556,1558,1560,1562,1564],{"class":65,"line":313},[63,1542,1480],{"class":87},[63,1544,92],{"class":91},[63,1546,1526],{"class":87},[63,1548,92],{"class":91},[63,1550,1551],{"class":87},"BreakDuration",[63,1553,133],{"class":132},[63,1555,682],{"class":87},[63,1557,92],{"class":91},[63,1559,687],{"class":95},[63,1561,142],{"class":91},[63,1563,988],{"class":289},[63,1565,149],{"class":91},[63,1567,1568],{"class":65,"line":318},[63,1569,588],{"emptyLinePlaceholder":587},[63,1571,1572,1574,1576,1579,1581,1584,1586,1588,1590,1592,1594,1597],{"class":65,"line":340},[63,1573,1480],{"class":87},[63,1575,92],{"class":91},[63,1577,1578],{"class":87},"AttemptTimeout",[63,1580,92],{"class":91},[63,1582,1583],{"class":87},"Timeout",[63,1585,133],{"class":132},[63,1587,682],{"class":87},[63,1589,92],{"class":91},[63,1591,687],{"class":95},[63,1593,142],{"class":91},[63,1595,1596],{"class":289},"5",[63,1598,149],{"class":91},[63,1600,1601,1603,1605,1608,1610,1612,1614,1616,1618,1620,1622,1624],{"class":65,"line":369},[63,1602,1480],{"class":87},[63,1604,92],{"class":91},[63,1606,1607],{"class":87},"TotalRequestTimeout",[63,1609,92],{"class":91},[63,1611,1583],{"class":87},[63,1613,133],{"class":132},[63,1615,682],{"class":87},[63,1617,92],{"class":91},[63,1619,687],{"class":95},[63,1621,142],{"class":91},[63,1623,988],{"class":289},[63,1625,149],{"class":91},[63,1627,1628],{"class":65,"line":374},[63,1629,1630],{"class":91},"    }));\n",[16,1632,1633,1634,1637],{},"The execution order for a ",[32,1635,1636],{},"GetAsync\u003CT>"," call:",[170,1639,1640,1646,1652,1655,1658],{},[173,1641,1642,1643,92],{},"The distributor picks client ",[32,1644,1645],{},"#0",[173,1647,1648,1649,1651],{},"The resilience pipeline on ",[32,1650,1645],{}," runs, including jittered retries. Each retry stays on the same pool.",[173,1653,1654],{},"If the response or exception is still transient after that pipeline, CloudHttp rotates once to a different client.",[173,1656,1657],{},"The resilience pipeline on the rotated client runs.",[173,1659,1660],{},"Whatever the second client returns is what the caller gets. No further rotation.",[16,1662,1663],{},"Each named client has its own circuit breaker, so a stuck pool does not drag the others down with it.",[16,1665,1666,1667,1670,1671,1674,1675,429],{},"The part of the math people miss: with ",[32,1668,1669],{},"MaxRetryAttempts = 3",", the caller can see up to eight attempts total (four on the first pool, four on the rotated pool). With ",[32,1672,1673],{},"c.Timeout = 30s",", the worst-case wall clock for two attempts is 60 seconds. If you want to bound the total wall clock, set it at the caller with ",[32,1676,1677],{},"CancellationTokenSource.CancelAfter",[54,1679,1681],{"className":78,"code":1680,"language":80,"meta":59,"style":59},"using var cts = CancellationTokenSource.CreateLinkedTokenSource(callerCt);\ncts.CancelAfter(TimeSpan.FromSeconds(30));\nawait http.GetAsync\u003CFoo>(\"\u002Fx\", cts.Token);\n",[32,1682,1683,1711,1735],{"__ignoreMap":59},[63,1684,1685,1688,1691,1694,1696,1699,1701,1704,1706,1709],{"class":65,"line":66},[63,1686,1687],{"class":439},"using",[63,1689,1690],{"class":439}," var",[63,1692,1693],{"class":528}," cts",[63,1695,133],{"class":132},[63,1697,1698],{"class":87}," CancellationTokenSource",[63,1700,92],{"class":91},[63,1702,1703],{"class":95},"CreateLinkedTokenSource",[63,1705,142],{"class":91},[63,1707,1708],{"class":528},"callerCt",[63,1710,149],{"class":91},[63,1712,1713,1716,1718,1721,1723,1725,1727,1729,1731,1733],{"class":65,"line":115},[63,1714,1715],{"class":87},"cts",[63,1717,92],{"class":91},[63,1719,1720],{"class":95},"CancelAfter",[63,1722,142],{"class":91},[63,1724,979],{"class":87},[63,1726,92],{"class":91},[63,1728,687],{"class":95},[63,1730,142],{"class":91},[63,1732,988],{"class":289},[63,1734,366],{"class":91},[63,1736,1737,1740,1743,1745,1747,1749,1752,1754,1757,1759,1761,1763,1766],{"class":65,"line":121},[63,1738,1739],{"class":91},"await ",[63,1741,1742],{"class":87},"http",[63,1744,92],{"class":91},[63,1746,600],{"class":95},[63,1748,99],{"class":91},[63,1750,1751],{"class":102},"Foo",[63,1753,106],{"class":91},[63,1755,1756],{"class":145},"\"\u002Fx\"",[63,1758,508],{"class":91},[63,1760,1715],{"class":87},[63,1762,92],{"class":91},[63,1764,1765],{"class":87},"Token",[63,1767,149],{"class":91},[16,1769,1770,1771,1773,1774,1777],{},"Caller cancellation always wins. CloudHttp does not rotate when the caller's ",[32,1772,511],{}," fires; it rethrows ",[32,1775,1776],{},"OperationCanceledException"," immediately.",[11,1779,1781],{"id":1780},"reads-can-rotate-writes-cannot","Reads can rotate, writes cannot",[16,1783,1784,1785,1787],{},"Two operations on ",[32,1786,195],{}," can rotate after a transient failure:",[1789,1790,1791,1796],"ul",{},[173,1792,1793],{},[32,1794,1795],{},"GetAsync\u003CT>(path, ct)",[173,1797,1798,1801],{},[32,1799,1800],{},"SendAsync(factory, ct)"," (the explicit \"build your own request\" version)",[16,1803,1804,1805,508,1808,508,1811,508,1814,1817,1818,1821],{},"The mutating JSON helpers (",[32,1806,1807],{},"PostAsync",[32,1809,1810],{},"PutAsync",[32,1812,1813],{},"PatchAsync",[32,1815,1816],{},"DeleteAsync",") do not auto-rotate. That is deliberate. A ",[32,1819,1820],{},"POST"," that times out may have already created the row, charged the card, or sent the email. Replaying it can duplicate side effects in ways CloudHttp has no way to detect from the outside.",[16,1823,1824,1825,1828],{},"If a write is genuinely safe to replay, make that explicit at the call site with an idempotency key and use ",[32,1826,1827],{},"SendAsync"," directly:",[54,1830,1832],{"className":78,"code":1831,"language":80,"meta":59,"style":59},"public Task\u003CHttpResponseMessage> ChargeAsync(\n    ChargeRequest body,\n    string idempotencyKey,\n    CancellationToken ct)\n{\n    return http.SendAsync((client, token) =>\n    {\n        using var request = new HttpRequestMessage(HttpMethod.Post, \"\u002Fcharges\")\n        {\n            Content = JsonContent.Create(body)\n        };\n        request.Headers.Add(\"Idempotency-Key\", idempotencyKey);\n        return client.SendAsync(request, token);\n    }, ct);\n}\n",[32,1833,1834,1853,1863,1873,1882,1886,1911,1915,1949,1954,1976,1981,2007,2029,2038],{"__ignoreMap":59},[63,1835,1836,1838,1840,1842,1845,1848,1851],{"class":65,"line":66},[63,1837,440],{"class":439},[63,1839,486],{"class":102},[63,1841,99],{"class":91},[63,1843,1844],{"class":102},"HttpResponseMessage",[63,1846,1847],{"class":91},"> ",[63,1849,1850],{"class":95},"ChargeAsync",[63,1852,219],{"class":91},[63,1854,1855,1858,1861],{"class":65,"line":115},[63,1856,1857],{"class":102},"    ChargeRequest",[63,1859,1860],{"class":87}," body",[63,1862,233],{"class":91},[63,1864,1865,1868,1871],{"class":65,"line":121},[63,1866,1867],{"class":439},"    string",[63,1869,1870],{"class":87}," idempotencyKey",[63,1872,233],{"class":91},[63,1874,1875,1878,1880],{"class":65,"line":152},[63,1876,1877],{"class":102},"    CancellationToken",[63,1879,514],{"class":87},[63,1881,474],{"class":91},[63,1883,1884],{"class":65,"line":253},[63,1885,118],{"class":91},[63,1887,1888,1891,1893,1895,1897,1900,1903,1905,1908],{"class":65,"line":277},[63,1889,1890],{"class":439},"    return",[63,1892,471],{"class":87},[63,1894,92],{"class":91},[63,1896,1827],{"class":95},[63,1898,1899],{"class":91},"((",[63,1901,1902],{"class":87},"client",[63,1904,508],{"class":91},[63,1906,1907],{"class":87},"token",[63,1909,1910],{"class":91},") =>\n",[63,1912,1913],{"class":65,"line":295},[63,1914,250],{"class":91},[63,1916,1917,1920,1922,1925,1927,1929,1932,1934,1937,1939,1942,1944,1947],{"class":65,"line":301},[63,1918,1919],{"class":439},"        using",[63,1921,1690],{"class":439},[63,1923,1924],{"class":528}," request",[63,1926,133],{"class":132},[63,1928,136],{"class":91},[63,1930,1931],{"class":102},"HttpRequestMessage",[63,1933,142],{"class":91},[63,1935,1936],{"class":87},"HttpMethod",[63,1938,92],{"class":91},[63,1940,1941],{"class":87},"Post",[63,1943,508],{"class":91},[63,1945,1946],{"class":145},"\"\u002Fcharges\"",[63,1948,474],{"class":91},[63,1950,1951],{"class":65,"line":313},[63,1952,1953],{"class":91},"        {\n",[63,1955,1956,1959,1961,1964,1966,1969,1971,1974],{"class":65,"line":318},[63,1957,1958],{"class":528},"            Content",[63,1960,133],{"class":132},[63,1962,1963],{"class":87}," JsonContent",[63,1965,92],{"class":91},[63,1967,1968],{"class":95},"Create",[63,1970,142],{"class":91},[63,1972,1973],{"class":528},"body",[63,1975,474],{"class":91},[63,1977,1978],{"class":65,"line":340},[63,1979,1980],{"class":91},"        };\n",[63,1982,1983,1986,1988,1991,1993,1995,1997,2000,2002,2005],{"class":65,"line":369},[63,1984,1985],{"class":87},"        request",[63,1987,92],{"class":91},[63,1989,1990],{"class":87},"Headers",[63,1992,92],{"class":91},[63,1994,357],{"class":95},[63,1996,142],{"class":91},[63,1998,1999],{"class":145},"\"Idempotency-Key\"",[63,2001,508],{"class":91},[63,2003,2004],{"class":528},"idempotencyKey",[63,2006,149],{"class":91},[63,2008,2009,2011,2014,2016,2018,2020,2023,2025,2027],{"class":65,"line":374},[63,2010,593],{"class":439},[63,2012,2013],{"class":87}," client",[63,2015,92],{"class":91},[63,2017,1827],{"class":95},[63,2019,142],{"class":91},[63,2021,2022],{"class":528},"request",[63,2024,508],{"class":91},[63,2026,1907],{"class":528},[63,2028,149],{"class":91},[63,2030,2031,2034,2036],{"class":65,"line":387},[63,2032,2033],{"class":91},"    }, ",[63,2035,614],{"class":528},[63,2037,149],{"class":91},[63,2039,2040],{"class":65,"line":392},[63,2041,626],{"class":91},[16,2043,2044],{},"The idempotency key belongs to the API contract between your caller and the upstream. CloudHttp cannot invent it.",[11,2046,2048],{"id":2047},"a-few-smaller-helpers","A few smaller helpers",[16,2050,2051],{},"Three small utilities that ended up in the library because they kept coming up in real service code:",[16,2053,2054,429],{},[2055,2056,2057],"strong",{},"Route templates",[54,2059,2061],{"className":78,"code":2060,"language":80,"meta":59,"style":59},"var path = HttpRouteBuilder.BuildPath(\n    \"\u002Fapi\u002Fv{ver}\u002Fusers\u002F{id}\",\n    new Dictionary\u003Cstring, object?> { [\"ver\"] = 2, [\"id\"] = userId });\n",[32,2062,2063,2080,2087],{"__ignoreMap":59},[63,2064,2065,2068,2070,2072,2074,2076,2078],{"class":65,"line":66},[63,2066,2067],{"class":439},"var",[63,2069,529],{"class":528},[63,2071,133],{"class":132},[63,2073,534],{"class":87},[63,2075,92],{"class":91},[63,2077,539],{"class":95},[63,2079,219],{"class":91},[63,2081,2082,2085],{"class":65,"line":115},[63,2083,2084],{"class":145},"    \"\u002Fapi\u002Fv{ver}\u002Fusers\u002F{id}\"",[63,2086,233],{"class":91},[63,2088,2089,2092,2094,2096,2098,2100,2102,2104,2107,2109,2111,2114,2116,2119,2121,2123,2126],{"class":65,"line":121},[63,2090,2091],{"class":91},"    new ",[63,2093,556],{"class":102},[63,2095,99],{"class":91},[63,2097,502],{"class":439},[63,2099,508],{"class":91},[63,2101,565],{"class":439},[63,2103,568],{"class":91},[63,2105,2106],{"class":145},"\"ver\"",[63,2108,574],{"class":91},[63,2110,577],{"class":132},[63,2112,2113],{"class":289}," 2",[63,2115,877],{"class":91},[63,2117,2118],{"class":145},"\"id\"",[63,2120,574],{"class":91},[63,2122,577],{"class":132},[63,2124,2125],{"class":528}," userId",[63,2127,582],{"class":91},[16,2129,2130],{},"It URL-encodes each value, uses no template engine, and allocates nothing per call beyond the final string.",[16,2132,2133,429],{},[2055,2134,2135],{},"Query merging",[54,2137,2139],{"className":78,"code":2138,"language":80,"meta":59,"style":59},"var uri = new Uri(\"https:\u002F\u002Fapi.example.com\u002Fsearch\")\n    .AddQuery(new[]\n    {\n        new KeyValuePair\u003Cstring, string?>(\"q\", searchTerm),\n        new KeyValuePair\u003Cstring, string?>(\"page\", page.ToString()),\n    });\n",[32,2140,2141,2161,2171,2175,2204,2236],{"__ignoreMap":59},[63,2142,2143,2145,2148,2150,2152,2154,2156,2159],{"class":65,"line":66},[63,2144,2067],{"class":439},[63,2146,2147],{"class":528}," uri",[63,2149,133],{"class":132},[63,2151,136],{"class":91},[63,2153,139],{"class":102},[63,2155,142],{"class":91},[63,2157,2158],{"class":145},"\"https:\u002F\u002Fapi.example.com\u002Fsearch\"",[63,2160,474],{"class":91},[63,2162,2163,2165,2168],{"class":65,"line":115},[63,2164,1111],{"class":91},[63,2166,2167],{"class":95},"AddQuery",[63,2169,2170],{"class":91},"(new[]\n",[63,2172,2173],{"class":65,"line":121},[63,2174,250],{"class":91},[63,2176,2177,2180,2183,2185,2187,2189,2191,2194,2197,2199,2202],{"class":65,"line":152},[63,2178,2179],{"class":91},"        new ",[63,2181,2182],{"class":102},"KeyValuePair",[63,2184,99],{"class":91},[63,2186,502],{"class":439},[63,2188,508],{"class":91},[63,2190,502],{"class":439},[63,2192,2193],{"class":91},"?>(",[63,2195,2196],{"class":145},"\"q\"",[63,2198,508],{"class":91},[63,2200,2201],{"class":528},"searchTerm",[63,2203,991],{"class":91},[63,2205,2206,2208,2210,2212,2214,2216,2218,2220,2223,2225,2228,2230,2233],{"class":65,"line":253},[63,2207,2179],{"class":91},[63,2209,2182],{"class":102},[63,2211,99],{"class":91},[63,2213,502],{"class":439},[63,2215,508],{"class":91},[63,2217,502],{"class":439},[63,2219,2193],{"class":91},[63,2221,2222],{"class":145},"\"page\"",[63,2224,508],{"class":91},[63,2226,2227],{"class":87},"page",[63,2229,92],{"class":91},[63,2231,2232],{"class":95},"ToString",[63,2234,2235],{"class":91},"()),\n",[63,2237,2238],{"class":65,"line":277},[63,2239,409],{"class":91},[16,2241,2242,429],{},[2055,2243,2244],{},"Best-effort error fallback",[54,2246,2248],{"className":78,"code":2247,"language":80,"meta":59,"style":59},"public async Task\u003CFeatureFlags> GetFlagsAsync(HttpClient client, ILogger logger, CancellationToken ct)\n{\n    return await client.GetWithErrorHandlingAsync(\n        \"\u002Fflags\",\n        defaultResponse: FeatureFlags.Empty,\n        logger: logger,\n        errorLogLevel: LogLevel.Warning,\n        ct: ct);\n}\n",[32,2249,2250,2291,2295,2311,2318,2334,2346,2363,2374],{"__ignoreMap":59},[63,2251,2252,2254,2257,2259,2261,2264,2266,2269,2271,2273,2275,2277,2280,2283,2285,2287,2289],{"class":65,"line":66},[63,2253,440],{"class":439},[63,2255,2256],{"class":439}," async",[63,2258,486],{"class":102},[63,2260,99],{"class":91},[63,2262,2263],{"class":102},"FeatureFlags",[63,2265,1847],{"class":91},[63,2267,2268],{"class":95},"GetFlagsAsync",[63,2270,142],{"class":91},[63,2272,34],{"class":102},[63,2274,2013],{"class":87},[63,2276,508],{"class":91},[63,2278,2279],{"class":102},"ILogger",[63,2281,2282],{"class":87}," logger",[63,2284,508],{"class":91},[63,2286,511],{"class":102},[63,2288,514],{"class":87},[63,2290,474],{"class":91},[63,2292,2293],{"class":65,"line":115},[63,2294,118],{"class":91},[63,2296,2297,2299,2302,2304,2306,2309],{"class":65,"line":121},[63,2298,1890],{"class":439},[63,2300,2301],{"class":91}," await ",[63,2303,1902],{"class":87},[63,2305,92],{"class":91},[63,2307,2308],{"class":95},"GetWithErrorHandlingAsync",[63,2310,219],{"class":91},[63,2312,2313,2316],{"class":65,"line":152},[63,2314,2315],{"class":145},"        \"\u002Fflags\"",[63,2317,233],{"class":91},[63,2319,2320,2323,2325,2327,2329,2332],{"class":65,"line":253},[63,2321,2322],{"class":87},"        defaultResponse",[63,2324,227],{"class":91},[63,2326,2263],{"class":87},[63,2328,92],{"class":91},[63,2330,2331],{"class":87},"Empty",[63,2333,233],{"class":91},[63,2335,2336,2339,2341,2344],{"class":65,"line":277},[63,2337,2338],{"class":87},"        logger",[63,2340,227],{"class":91},[63,2342,2343],{"class":528},"logger",[63,2345,233],{"class":91},[63,2347,2348,2351,2353,2356,2358,2361],{"class":65,"line":295},[63,2349,2350],{"class":87},"        errorLogLevel",[63,2352,227],{"class":91},[63,2354,2355],{"class":87},"LogLevel",[63,2357,92],{"class":91},[63,2359,2360],{"class":87},"Warning",[63,2362,233],{"class":91},[63,2364,2365,2368,2370,2372],{"class":65,"line":301},[63,2366,2367],{"class":87},"        ct",[63,2369,227],{"class":91},[63,2371,614],{"class":528},[63,2373,149],{"class":91},[63,2375,2376],{"class":65,"line":313},[63,2377,626],{"class":91},[16,2379,2380],{},"This is not a general error strategy. It is for non-critical calls where a fallback value is genuinely acceptable, like feature flags or optional metadata. Caller cancellation is always propagated; only HTTP failures, JSON errors, and I\u002FO errors fall through to the default value, and they always get logged.",[11,2382,2384],{"id":2383},"out-of-scope-on-purpose","Out of scope, on purpose",[16,2386,2387],{},"It's worth being explicit about what the library deliberately stays away from.",[1789,2389,2390,2396,2403,2406,2409],{},[173,2391,2392,2393,2395],{},"No retries, exponential backoff, jitter, or circuit breaking. Use ",[32,2394,1369],{}," and let CloudHttp compose with it.",[173,2397,2398,2399,2402],{},"No distributed health state across caller pods. Each replica of your caller service tracks its own health-aware degradations. Pool ",[32,2400,2401],{},"#2"," being marked degraded on pod A does not propagate to pod B.",[173,2404,2405],{},"No service discovery beyond cluster DNS. No Consul, no Kubernetes API integration, no Eureka.",[173,2407,2408],{},"No guarantee of even distribution. The N-pool trick is best-effort. If you need actual L7 load balancing, a service mesh is the right tool.",[173,2410,2411,2412,508,2414,508,2417,508,2420,2423],{},"No automatic replay of mutating operations. ",[32,2413,1820],{},[32,2415,2416],{},"PUT",[32,2418,2419],{},"PATCH",[32,2421,2422],{},"DELETE"," will not retry across pools. That is by design.",[16,2425,2426],{},"These are not bugs. They are scope decisions. Doing fewer things means each one is easier to reason about, and Microsoft already ships a better retry \u002F circuit-breaker library that this one composes with.",[11,2428,2430],{"id":2429},"when-this-is-the-right-fit","When this is the right fit",[16,2432,2433],{},"You probably want CloudHttp if:",[1789,2435,2436,2439,2442,2445],{},[173,2437,2438],{},"Your .NET service calls another service through a cluster DNS name.",[173,2440,2441],{},"You see one TCP connection from each caller pod sticking to the same upstream pod across long lifetimes.",[173,2443,2444],{},"You do not have a service mesh handling L7 load balancing for you.",[173,2446,2447,2448,2450],{},"You want a ",[32,2449,38],{}," profile sane for cluster traffic without writing it from scratch every time.",[16,2452,2453],{},"Skip it when:",[1789,2455,2456,2459,2462,2465],{},[173,2457,2458],{},"You only call public internet APIs. The cluster-traffic defaults are not the right shape for slow, distant, less reliable upstreams.",[173,2460,2461],{},"A single connection pool is enough for your throughput.",[173,2463,2464],{},"You already run a service mesh sidecar that handles L7 balancing.",[173,2466,2467],{},"Your upstream already does client-side balancing (e.g. gRPC name resolvers, AWS SDK-style retries with discovery).",[11,2469,2471],{"id":2470},"try-it","Try it",[16,2473,2474],{},"The repo includes a Docker Compose sample that runs several upstream containers behind one DNS name and prints which backend handled each request:",[54,2476,2480],{"className":2477,"code":2478,"language":2479,"meta":59,"style":59},"language-powershell shiki shiki-themes one-light one-dark-pro","$env:REQUESTS = \"48\"\n$env:CLIENT_COUNT = \"8\"\n$env:DISTRIBUTION_MODE = \"RoundRobin\"\n\ndocker compose `\n  --file .\\samples\\CloudHttp.Sample\\compose.yaml `\n  up --build --abort-on-container-exit --exit-code-from client `\n  --scale upstream=4 client\n","powershell",[32,2481,2482,2487,2492,2497,2501,2506,2511,2516],{"__ignoreMap":59},[63,2483,2484],{"class":65,"line":66},[63,2485,2486],{},"$env:REQUESTS = \"48\"\n",[63,2488,2489],{"class":65,"line":115},[63,2490,2491],{},"$env:CLIENT_COUNT = \"8\"\n",[63,2493,2494],{"class":65,"line":121},[63,2495,2496],{},"$env:DISTRIBUTION_MODE = \"RoundRobin\"\n",[63,2498,2499],{"class":65,"line":152},[63,2500,588],{"emptyLinePlaceholder":587},[63,2502,2503],{"class":65,"line":253},[63,2504,2505],{},"docker compose `\n",[63,2507,2508],{"class":65,"line":277},[63,2509,2510],{},"  --file .\\samples\\CloudHttp.Sample\\compose.yaml `\n",[63,2512,2513],{"class":65,"line":295},[63,2514,2515],{},"  up --build --abort-on-container-exit --exit-code-from client `\n",[63,2517,2518],{"class":65,"line":301},[63,2519,2520],{},"  --scale upstream=4 client\n",[16,2522,2523],{},"After the run, the client prints a summary like this:",[54,2525,2530],{"className":2526,"code":2528,"language":2529,"meta":59},[2527],"language-text","Summary\n-------\ncloudhttp-sample-upstream-1: 11 responses\ncloudhttp-sample-upstream-2: 14 responses\ncloudhttp-sample-upstream-3: 12 responses\ncloudhttp-sample-upstream-4: 11 responses\nFailures observed by client: 0\n","text",[32,2531,2528],{"__ignoreMap":59},[16,2533,2534,2535,2537],{},"The distribution will not be perfectly even, since Docker DNS, connection pooling, and timing all interfere, but you can see a single logical upstream getting reached through several independent client pools. There is also an unstable variant that makes upstreams return 503 every Nth request, which is the cleanest way to watch ",[32,2536,600],{}," rotation in action.",[16,2539,2540,2541,92],{},"The full walkthrough, including environment variables, the health-aware and weighted variants, and a Docker-less local script, is in ",[20,2542,2545],{"href":2543,"rel":2544},"https:\u002F\u002Fgithub.com\u002Fhaiilong\u002FCloudHttp\u002Fblob\u002Fmain\u002Fsamples\u002FCloudHttp.Sample\u002FREADME.md",[52],"samples\u002FCloudHttp.Sample\u002FREADME.md",[11,2547,2549],{"id":2548},"wrapping-the-http-series","Wrapping the HTTP series",[16,2551,2552],{},"This is the third post on this blog about HTTP in .NET, and probably the last for a while. The 2024 piece described the underlying load-balancing problem in Kubernetes. The HttpClient connection lifetime post was about understanding the machinery. CloudHttp turns the workaround into a library you can install.",[16,2554,2555],{},"The package itself is small: a handful of extension methods, one selector type, three distribution strategies, a curated set of cloud defaults, a few JSON helpers. The README and the three docs files in the repo go further than this post for anyone who wants more.",[16,2557,2558,2559],{},"Repo: ",[20,2560,50],{"href":50,"rel":2561},[52],[2563,2564,2565],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .s7GmK, html code.shiki .s7GmK{--shiki-default:#383A42;--shiki-dark:#E5C07B}html pre.shiki code .s5ixo, html code.shiki .s5ixo{--shiki-default:#383A42;--shiki-dark:#ABB2BF}html pre.shiki code .sAdtL, html code.shiki .sAdtL{--shiki-default:#4078F2;--shiki-dark:#61AFEF}html pre.shiki code .sC09Y, html code.shiki .sC09Y{--shiki-default:#C18401;--shiki-dark:#E5C07B}html pre.shiki code .sknuh, html code.shiki .sknuh{--shiki-default:#383A42;--shiki-dark:#56B6C2}html pre.shiki code .sDhpE, html code.shiki .sDhpE{--shiki-default:#50A14F;--shiki-dark:#98C379}html pre.shiki code .sAGMh, html code.shiki .sAGMh{--shiki-default:#986801;--shiki-dark:#D19A66}html pre.shiki code .sLKXg, html code.shiki .sLKXg{--shiki-default:#A626A4;--shiki-dark:#C678DD}html pre.shiki code .sz0mV, html code.shiki .sz0mV{--shiki-default:#383A42;--shiki-dark:#E06C75}",{"title":59,"searchDepth":115,"depth":115,"links":2567},[2568,2569,2570,2571,2576,2577,2578,2579,2580,2581,2582,2583,2584],{"id":13,"depth":115,"text":14},{"id":71,"depth":115,"text":72},{"id":188,"depth":115,"text":189},{"id":717,"depth":115,"text":718,"children":2572},[2573,2574,2575],{"id":722,"depth":121,"text":723},{"id":806,"depth":121,"text":807},{"id":925,"depth":121,"text":926},{"id":1039,"depth":115,"text":1040},{"id":1052,"depth":115,"text":1053},{"id":1358,"depth":115,"text":1359},{"id":1780,"depth":115,"text":1781},{"id":2047,"depth":115,"text":2048},{"id":2383,"depth":115,"text":2384},{"id":2429,"depth":115,"text":2430},{"id":2470,"depth":115,"text":2471},{"id":2548,"depth":115,"text":2549},null,"2026-05-15","Cloud-friendly HttpClient extensions for .NET, independent connection pools, cloud-tuned defaults, and composition with Microsoft's resilience pipeline.","md",{},"\u002Fblog\u002Fwhy-i-built-cloudhttp",{"title":6,"description":2587},"blog\u002Fwhy-i-built-cloudhttp",[2594],"tech","G1SBLq5q3ZTrlDR70iEIfvtIMBS5Ap2M5aZ1wPtxdcU",{"id":2597,"title":2598,"body":2599,"book":2585,"date":4149,"description":4150,"extension":2588,"meta":4151,"navigation":587,"path":4152,"seo":4153,"stem":4154,"tags":4155,"__hash__":4156},"blog\u002Fblog\u002Fwhy-i-built-singletonjob.md","Why I built SingletonJob",{"type":8,"value":2600,"toc":4131},[2601,2603,2606,2609,2612,2615,2621,2625,2628,2631,2651,2655,2658,2661,2675,2678,2682,2685,2712,2715,2719,2722,3059,3064,3069,3074,3078,3081,3087,3094,3100,3114,3117,3149,3152,3155,3181,3185,3188,3191,3197,3212,3215,3221,3250,3266,3272,3276,3289,3292,3298,3302,3310,3324,3331,3334,3344,3348,3354,3500,3517,3520,3524,3541,3548,3745,3752,3781,3788,3799,3801,3804,3884,3890,3893,3950,3953,3957,3960,3980,3983,3987,4002,4015,4026,4036,4049,4051,4060,4063,4078,4087,4092,4103,4107,4125,4128],[11,2602,14],{"id":13},[16,2604,2605],{},"I work on a trading system. Without going into the specifics of what we trade, two things matter for the rest of this post.",[16,2607,2608],{},"First, prices tick. Every second. Sometimes faster than that.",[16,2610,2611],{},"Second, the prediction pipeline needs fresh data, and that data has to be pulled from a lot of different sources every 500 ms or so by background threads. If you zoom out far enough, the whole thing is basically \"is the data ready, and how stale is it?\" on a loop.",[16,2613,2614],{},"This is the story of a library I've wanted to write for years and finally did.",[16,2616,2617],{},[20,2618,2619],{"href":2619,"rel":2620},"https:\u002F\u002Fgithub.com\u002Fhaiilong\u002FSingletonJob",[52],[11,2622,2624],{"id":2623},"what-i-wanted","What I wanted",[16,2626,2627],{},"A way to run a periodic job across a few pods in Kubernetes, where exactly one pod runs it at any given moment. That's basically it.",[16,2629,2630],{},"But the requirements piled up:",[1789,2632,2633,2636,2639,2642,2645,2648],{},[173,2634,2635],{},"Sub-second frequency. Some jobs need to fire every 500 ms.",[173,2637,2638],{},"Drop-on-overlap. If the previous tick is still running, skip the next one. Don't queue it. Don't run two at once. Just drop.",[173,2640,2641],{},"No persistence overhead. I don't need a job history, a dashboard, retry policies, or a database table per job. Just \"exactly one pod is the leader, and that pod runs the loop\".",[173,2643,2644],{},"Failover in seconds, not minutes.",[173,2646,2647],{},"Cheap. Hundreds of bytes per job in Redis, not hundreds of megabytes of in-memory state per pod.",[173,2649,2650],{},"AOT compatible, because eventually I want everything trimmed and AOT'd anyway.",[11,2652,2654],{"id":2653},"why-not-hangfire","Why not Hangfire",[16,2656,2657],{},"Hangfire is great for what it was built for: durable, retryable, observable background jobs with a dashboard, like email queues and nightly reports. These are jobs where you come back tomorrow and see what happened.",[16,2659,2660],{},"But it isn't the right shape for what I needed:",[1789,2662,2663,2666,2669,2672],{},[173,2664,2665],{},"Cron has a one second minimum. That alone disqualifies it for tick driven work.",[173,2667,2668],{},"Overlapping runs queue. If the previous job runs long, the next one doesn't get skipped, it stacks. For price ticks, that's exactly backwards. You want the new tick to fire and the old one to die.",[173,2670,2671],{},"Memory and CPU spikes on startup. For a worker pod that already holds models in memory and runs hot loops, a Hangfire startup spike is not free.",[173,2673,2674],{},"The storage backend is structural overhead I don't need. A SQL Server schema with histories, retries, states, hash tables. For \"one pod runs this every second\", that is far too much machinery.",[16,2676,2677],{},"I'm not picking a fight with Hangfire. I just needed a different shape of tool, and the right shape happened to be small enough that nobody had bothered to publish it.",[11,2679,2681],{"id":2680},"why-redis","Why Redis",[16,2683,2684],{},"When I floated the idea, the first question I got was usually \"couldn't you do this with a SQL Server row, or etcd, or ZooKeeper?\" Yes, you can. All of those work. Here is why I went with Redis anyway.",[1789,2686,2687,2690,2700,2703,2709],{},[173,2688,2689],{},"Almost every .NET microservice I have worked on already had Redis somewhere: caching, pub\u002Fsub, rate limiting, locks for other things. Adding a 50 byte lock key per job is basically free.",[173,2691,2692,2695,2696,2699],{},[32,2693,2694],{},"SET NX PX"," is one command. The SQL Server equivalent is a transaction with ",[32,2697,2698],{},"WITH (UPDLOCK, HOLDLOCK)"," wrapped in a stored procedure. It works, but it's a lot more moving parts for the same outcome.",[173,2701,2702],{},"Lua scripts. The renewal and release patterns below are seven lines each. The SQL equivalents are not.",[173,2704,2705,2708],{},[32,2706,2707],{},"StackExchange.Redis"," is mature and well behaved under load. I have never once had to debug the client itself, which is more than I can say for some SQL drivers.",[173,2710,2711],{},"A lock key is around 50 bytes. Three replicas × five jobs, each heartbeat every three seconds → 15\u002F3 = 5 ops\u002Fsec. Cost isn't something I have to think about.",[16,2713,2714],{},"If your stack already has etcd or Consul, those work fine too. But for a typical .NET shop with Redis already in the picture, this is about as cheap as it gets.",[11,2716,2718],{"id":2717},"the-shape-of-the-thing","The shape of the thing",[16,2720,2721],{},"Three job types ended up covering pretty much every periodic workload I've had at work.",[54,2723,2725],{"className":78,"code":2724,"language":80,"meta":59,"style":59},"\u002F\u002F 1) Run, wait, run. \"At least N seconds between runs.\"\npublic sealed class HeartbeatJob(...) : SingletonIntervalJob(...)\n{\n    public override string JobName => \"heartbeat\";\n    protected override TimeSpan GetJobInterval() => TimeSpan.FromSeconds(1);\n    protected override Task ExecuteJobAsync(CancellationToken ct) { ... }\n}\n\n\u002F\u002F 2) Fire on a fixed rate. Drop the tick if the previous run is still in flight.\npublic sealed class PriceTickJob(...) : SingletonFixedRateJob(...)\n{\n    public override string JobName => \"price-tick\";\n    protected override TimeSpan GetJobInterval() => TimeSpan.FromMilliseconds(500);\n    protected override Task ExecuteJobAsync(CancellationToken ct) { ... }\n}\n\n\u002F\u002F 3) Cron schedule.\npublic sealed class DailyReportJob(...) : SingletonCronJob(...)\n{\n    private static readonly CronExpression Expr = CronExpression.Parse(\"0 3 * * *\"); \u002F\u002F from the Cronos library\n    public override string JobName => \"daily-report\";\n    protected override CronExpression GetCronExpression() => Expr;\n    protected override Task ExecuteJobAsync(CancellationToken ct) { ... }\n}\n",[32,2726,2727,2733,2753,2757,2778,2805,2825,2829,2833,2838,2856,2860,2877,2903,2921,2925,2929,2935,2954,2959,2998,3016,3035,3054],{"__ignoreMap":59},[63,2728,2729],{"class":65,"line":66},[63,2730,2732],{"class":2731},"sW2Sy","\u002F\u002F 1) Run, wait, run. \"At least N seconds between runs.\"\n",[63,2734,2735,2737,2739,2741,2744,2747,2750],{"class":65,"line":115},[63,2736,440],{"class":439},[63,2738,443],{"class":439},[63,2740,446],{"class":439},[63,2742,2743],{"class":102}," HeartbeatJob",[63,2745,2746],{"class":91},"(...) : ",[63,2748,2749],{"class":102},"SingletonIntervalJob",[63,2751,2752],{"class":91},"(...)\n",[63,2754,2755],{"class":65,"line":121},[63,2756,118],{"class":91},[63,2758,2759,2761,2764,2767,2771,2773,2776],{"class":65,"line":152},[63,2760,483],{"class":439},[63,2762,2763],{"class":439}," override",[63,2765,2766],{"class":439}," string",[63,2768,2770],{"class":2769},"siaei"," JobName",[63,2772,784],{"class":91},[63,2774,2775],{"class":145},"\"heartbeat\"",[63,2777,274],{"class":91},[63,2779,2780,2783,2785,2787,2790,2793,2795,2797,2799,2801,2803],{"class":65,"line":253},[63,2781,2782],{"class":439},"    protected",[63,2784,2763],{"class":439},[63,2786,682],{"class":102},[63,2788,2789],{"class":95}," GetJobInterval",[63,2791,2792],{"class":91},"() => ",[63,2794,979],{"class":87},[63,2796,92],{"class":91},[63,2798,687],{"class":95},[63,2800,142],{"class":91},[63,2802,880],{"class":289},[63,2804,149],{"class":91},[63,2806,2807,2809,2811,2813,2816,2818,2820,2822],{"class":65,"line":277},[63,2808,2782],{"class":439},[63,2810,2763],{"class":439},[63,2812,486],{"class":102},[63,2814,2815],{"class":95}," ExecuteJobAsync",[63,2817,142],{"class":91},[63,2819,511],{"class":102},[63,2821,514],{"class":87},[63,2823,2824],{"class":91},") { ... }\n",[63,2826,2827],{"class":65,"line":295},[63,2828,626],{"class":91},[63,2830,2831],{"class":65,"line":301},[63,2832,588],{"emptyLinePlaceholder":587},[63,2834,2835],{"class":65,"line":313},[63,2836,2837],{"class":2731},"\u002F\u002F 2) Fire on a fixed rate. Drop the tick if the previous run is still in flight.\n",[63,2839,2840,2842,2844,2846,2849,2851,2854],{"class":65,"line":318},[63,2841,440],{"class":439},[63,2843,443],{"class":439},[63,2845,446],{"class":439},[63,2847,2848],{"class":102}," PriceTickJob",[63,2850,2746],{"class":91},[63,2852,2853],{"class":102},"SingletonFixedRateJob",[63,2855,2752],{"class":91},[63,2857,2858],{"class":65,"line":340},[63,2859,118],{"class":91},[63,2861,2862,2864,2866,2868,2870,2872,2875],{"class":65,"line":369},[63,2863,483],{"class":439},[63,2865,2763],{"class":439},[63,2867,2766],{"class":439},[63,2869,2770],{"class":2769},[63,2871,784],{"class":91},[63,2873,2874],{"class":145},"\"price-tick\"",[63,2876,274],{"class":91},[63,2878,2879,2881,2883,2885,2887,2889,2891,2893,2896,2898,2901],{"class":65,"line":374},[63,2880,2782],{"class":439},[63,2882,2763],{"class":439},[63,2884,682],{"class":102},[63,2886,2789],{"class":95},[63,2888,2792],{"class":91},[63,2890,979],{"class":87},[63,2892,92],{"class":91},[63,2894,2895],{"class":95},"FromMilliseconds",[63,2897,142],{"class":91},[63,2899,2900],{"class":289},"500",[63,2902,149],{"class":91},[63,2904,2905,2907,2909,2911,2913,2915,2917,2919],{"class":65,"line":387},[63,2906,2782],{"class":439},[63,2908,2763],{"class":439},[63,2910,486],{"class":102},[63,2912,2815],{"class":95},[63,2914,142],{"class":91},[63,2916,511],{"class":102},[63,2918,514],{"class":87},[63,2920,2824],{"class":91},[63,2922,2923],{"class":65,"line":392},[63,2924,626],{"class":91},[63,2926,2927],{"class":65,"line":406},[63,2928,588],{"emptyLinePlaceholder":587},[63,2930,2932],{"class":65,"line":2931},17,[63,2933,2934],{"class":2731},"\u002F\u002F 3) Cron schedule.\n",[63,2936,2938,2940,2942,2944,2947,2949,2952],{"class":65,"line":2937},18,[63,2939,440],{"class":439},[63,2941,443],{"class":439},[63,2943,446],{"class":439},[63,2945,2946],{"class":102}," DailyReportJob",[63,2948,2746],{"class":91},[63,2950,2951],{"class":102},"SingletonCronJob",[63,2953,2752],{"class":91},[63,2955,2957],{"class":65,"line":2956},19,[63,2958,118],{"class":91},[63,2960,2962,2965,2968,2971,2974,2978,2980,2982,2984,2987,2989,2992,2995],{"class":65,"line":2961},20,[63,2963,2964],{"class":439},"    private",[63,2966,2967],{"class":439}," static",[63,2969,2970],{"class":439}," readonly",[63,2972,2973],{"class":102}," CronExpression",[63,2975,2977],{"class":2976},"sJa8x"," Expr",[63,2979,133],{"class":132},[63,2981,2973],{"class":87},[63,2983,92],{"class":91},[63,2985,2986],{"class":95},"Parse",[63,2988,142],{"class":91},[63,2990,2991],{"class":145},"\"0 3 * * *\"",[63,2993,2994],{"class":91},"); ",[63,2996,2997],{"class":2731},"\u002F\u002F from the Cronos library\n",[63,2999,3001,3003,3005,3007,3009,3011,3014],{"class":65,"line":3000},21,[63,3002,483],{"class":439},[63,3004,2763],{"class":439},[63,3006,2766],{"class":439},[63,3008,2770],{"class":2769},[63,3010,784],{"class":91},[63,3012,3013],{"class":145},"\"daily-report\"",[63,3015,274],{"class":91},[63,3017,3019,3021,3023,3025,3028,3030,3033],{"class":65,"line":3018},22,[63,3020,2782],{"class":439},[63,3022,2763],{"class":439},[63,3024,2973],{"class":102},[63,3026,3027],{"class":95}," GetCronExpression",[63,3029,2792],{"class":91},[63,3031,3032],{"class":528},"Expr",[63,3034,274],{"class":91},[63,3036,3038,3040,3042,3044,3046,3048,3050,3052],{"class":65,"line":3037},23,[63,3039,2782],{"class":439},[63,3041,2763],{"class":439},[63,3043,486],{"class":102},[63,3045,2815],{"class":95},[63,3047,142],{"class":91},[63,3049,511],{"class":102},[63,3051,514],{"class":87},[63,3053,2824],{"class":91},[63,3055,3057],{"class":65,"line":3056},24,[63,3058,626],{"class":91},[16,3060,3061,3063],{},[32,3062,2749],{}," is the simple one. Run, wait N seconds, run again. The time between iterations is bounded below, not above. If a job takes longer than the interval, the next start just gets pushed out.",[16,3065,3066,3068],{},[32,3067,2853],{}," is the one I actually wrote this library for. Ticks come at fixed wall-clock offsets. If the previous tick is still running when the next one fires, that next tick gets dropped on the floor. No queue, no overlap, no surprise stacking later when the load picks back up.",[16,3070,3071,3073],{},[32,3072,2951],{}," is for the boring stuff. Nightly reports, hourly cleanups, anything where the time of day matters. Cron expression in, callback out.",[11,3075,3077],{"id":3076},"how-leader-election-actually-works","How leader election actually works",[16,3079,3080],{},"Leader election comes down to a single Redis key per job.",[54,3082,3085],{"className":3083,"code":3084,"language":2529},[2527],"{ProjectName}:{JobName}:lock\n",[32,3086,3084],{"__ignoreMap":59},[16,3088,3089,3090,3093],{},"Every replica, every ",[32,3091,3092],{},"HeartbeatInterval"," (3 seconds by default), runs:",[54,3095,3098],{"className":3096,"code":3097,"language":2529},[2527],"SET {lockKey} {nodeId} NX PX {LockExpiry}\n",[32,3099,3097],{"__ignoreMap":59},[16,3101,3102,3105,3106,3109,3110,3113],{},[32,3103,3104],{},"NX"," means \"only set if absent\". ",[32,3107,3108],{},"PX"," is a TTL in milliseconds. The first pod to land that SET becomes the leader. Everyone else gets ",[32,3111,3112],{},"null"," back and stays a follower.",[16,3115,3116],{},"Renewal is a tiny Lua script the leader runs on every heartbeat:",[54,3118,3122],{"className":3119,"code":3120,"language":3121,"meta":59,"style":59},"language-lua shiki shiki-themes one-light one-dark-pro","if redis.call('GET', KEYS[1]) == ARGV[1] then\n    return redis.call('PEXPIRE', KEYS[1], ARGV[2])\nelse\n    return 0\nend\n","lua",[32,3123,3124,3129,3134,3139,3144],{"__ignoreMap":59},[63,3125,3126],{"class":65,"line":66},[63,3127,3128],{},"if redis.call('GET', KEYS[1]) == ARGV[1] then\n",[63,3130,3131],{"class":65,"line":115},[63,3132,3133],{},"    return redis.call('PEXPIRE', KEYS[1], ARGV[2])\n",[63,3135,3136],{"class":65,"line":121},[63,3137,3138],{},"else\n",[63,3140,3141],{"class":65,"line":152},[63,3142,3143],{},"    return 0\n",[63,3145,3146],{"class":65,"line":253},[63,3147,3148],{},"end\n",[16,3150,3151],{},"Only the holder can extend the TTL. If the script returns 0, we lost leadership (probably because too many heartbeats failed in a row and the key expired in between), and the loop drops back to follower mode.",[16,3153,3154],{},"On graceful shutdown, there's a third Lua script:",[54,3156,3158],{"className":3119,"code":3157,"language":3121,"meta":59,"style":59},"if redis.call('GET', KEYS[1]) == ARGV[1] then\n    return redis.call('DEL', KEYS[1])\nelse\n    return 0\nend\n",[32,3159,3160,3164,3169,3173,3177],{"__ignoreMap":59},[63,3161,3162],{"class":65,"line":66},[63,3163,3128],{},[63,3165,3166],{"class":65,"line":115},[63,3167,3168],{},"    return redis.call('DEL', KEYS[1])\n",[63,3170,3171],{"class":65,"line":121},[63,3172,3138],{},[63,3174,3175],{"class":65,"line":152},[63,3176,3143],{},[63,3178,3179],{"class":65,"line":253},[63,3180,3148],{},[11,3182,3184],{"id":3183},"why-all-three-of-these-are-one-redis-command","Why all three of these are one Redis command",[16,3186,3187],{},"You might notice that acquire, renew, and release are each a single Redis operation. That's deliberate. Anything that does \"check, then act\" against shared state across multiple round trips is a race waiting to happen.",[16,3189,3190],{},"Take acquire. The naive version would be:",[54,3192,3195],{"className":3193,"code":3194,"language":2529},[2527],"EXISTS lockKey      # returns 0, nobody owns it\nSET lockKey nodeId  # OK, I'll set it\n",[32,3196,3194],{"__ignoreMap":59},[16,3198,3199,3200,3203,3204,3207,3208,3211],{},"Between those two commands, another pod can also see ",[32,3201,3202],{},"EXISTS"," return 0 and also issue its own ",[32,3205,3206],{},"SET",". Now both pods think they're the leader, and you spend the next incident figuring out why two workers fought over the same tick. ",[32,3209,3210],{},"SET ... NX"," solves this by collapsing the check and the write into one operation that Redis runs as a single atomic step. There is no window for anyone to slip in.",[16,3213,3214],{},"Renewal has the same problem. The naive version is:",[54,3216,3219],{"className":3217,"code":3218,"language":2529},[2527],"GET lockKey            # is it still me?\nPEXPIRE lockKey 10000  # yes, extend the TTL\n",[32,3220,3218],{"__ignoreMap":59},[16,3222,3223,3224,3227,3228,3231,3232,3235,3236,3238,3239,3243,3244,3246,3247,3249],{},"Between the ",[32,3225,3226],{},"GET"," and the ",[32,3229,3230],{},"PEXPIRE",", the lock can expire on its own (a network blip, a few missed heartbeats), and another pod can ",[32,3233,3234],{},"SET NX"," and become the new leader. If we then run our ",[32,3237,3230],{},", we just extended ",[3240,3241,3242],"em",{},"their"," lock without realizing it. The new leader now holds a key with twice the TTL it should have, and we don't even know we lost leadership. The Lua script wraps ",[32,3245,3226],{}," and ",[32,3248,3230],{}," into one call. Redis runs the whole script atomically from every other client's perspective, so nothing can sneak in between the two steps.",[16,3251,3252,3253,3255,3256,3259,3260,3262,3263,3265],{},"Release is the same shape. ",[32,3254,3226],{}," then ",[32,3257,3258],{},"DEL"," is two commands and a race: if the lock expires and another pod acquires it between the two, our ",[32,3261,3258],{}," deletes ",[3240,3264,3242],{}," lock. The Lua version checks ownership and deletes in one step.",[16,3267,3268,3269,3271],{},"So the rule is: any operation whose correctness depends on the current state of the lock has to run on the Redis server in one shot. ",[32,3270,3234],{}," handles acquire. Lua handles the other two.",[11,3273,3275],{"id":3274},"why-explicit-release-matters","Why explicit release matters",[16,3277,3278,3279,3282,3283,3285,3286,3288],{},"Without an explicit release, peers have to wait up to ",[32,3280,3281],{},"LockExpiry"," (10 seconds by default) before a fresh ",[32,3284,3234],{}," can win. With release, the next pod takes over within one ",[32,3287,3092],{},", which is 3 seconds by default.",[16,3290,3291],{},"On a rolling deploy, that's the difference between \"10 seconds of nobody running the job\" and \"3 seconds of nobody running the job\". For a tick driven loop firing every 500 ms, that's the difference between a few stale ticks and around thirty of them.",[16,3293,3294,3295,3297],{},"On a hard kill (SIGKILL, OOM, the node dropping off the network), nothing graceful runs. The lock just expires after ",[32,3296,3281],{},". That's still fine. It's the worst case, and the worst case is bounded by your config.",[11,3299,3301],{"id":3300},"sizing-heartbeatinterval-and-lockexpiry","Sizing HeartbeatInterval and LockExpiry",[16,3303,3304,3305,3246,3307,3309],{},"There are really only two knobs to tune: ",[32,3306,3092],{},[32,3308,3281],{},". The relationship between them is what you actually care about.",[16,3311,3312,3314,3315,3317,3318,3320,3321,3323],{},[32,3313,3092],{}," is how often the leader tries to renew. ",[32,3316,3281],{}," is the TTL on the key. Once ",[32,3319,3281],{}," passes without a successful renewal, the key vanishes from Redis and whichever replica wins the next ",[32,3322,3234],{}," is the new leader.",[16,3325,3326,3327,3330],{},"Set them too close together (say 3s and 4s), and one slow round trip costs you leadership. Set them too far apart (3s and 60s), and a hard kill takes a full minute to fail over. The rule I land on is ",[32,3328,3329],{},"LockExpiry >= 3 * HeartbeatInterval",". Three missed renewals before we lose the lock. The defaults (3s and 10s) fit that rule.",[16,3332,3333],{},"For very fast jobs (every 500 ms, every 100 ms), the job loop tightens up, but the heartbeat doesn't have to match the job tick. The job loop and the election loop run in parallel inside the same hosted service, so you can run the job every 500 ms and still heartbeat at 3 s without anything fighting.",[16,3335,3336,3337,3340,3341,3343],{},"The library also logs a warning if a single iteration of ",[32,3338,3339],{},"ExecuteJobAsync"," runs longer than 80% of ",[32,3342,3281],{},". That's the canary for \"your job is so slow it's about to time out the lock and another pod will take it from you\". If you see that warning regularly, the sizing is wrong, not the job.",[11,3345,3347],{"id":3346},"the-drop-on-overlap-bit","The drop-on-overlap bit",[16,3349,3350,3351,3353],{},"This is what ",[32,3352,2853],{}," does. The iteration loop, simplified, is:",[54,3355,3357],{"className":78,"code":3356,"language":80,"meta":59,"style":59},"while (!ct.IsCancellationRequested)\n{\n    await _timer.WaitForNextTickAsync(ct);\n\n    if (!IsLeader) continue;\n    if (_isJobRunning) { \u002F* drop this tick *\u002F continue; }\n\n    _isJobRunning = true;\n    try { await ExecuteJobAsync(ct); }\n    finally { _isJobRunning = false; }\n}\n",[32,3358,3359,3379,3383,3402,3406,3426,3447,3451,3462,3479,3496],{"__ignoreMap":59},[63,3360,3361,3364,3367,3370,3372,3374,3377],{"class":65,"line":66},[63,3362,3363],{"class":439},"while",[63,3365,3366],{"class":91}," (",[63,3368,3369],{"class":132},"!",[63,3371,614],{"class":87},[63,3373,92],{"class":91},[63,3375,3376],{"class":87},"IsCancellationRequested",[63,3378,474],{"class":91},[63,3380,3381],{"class":65,"line":115},[63,3382,118],{"class":91},[63,3384,3385,3388,3391,3393,3396,3398,3400],{"class":65,"line":121},[63,3386,3387],{"class":91},"    await ",[63,3389,3390],{"class":87},"_timer",[63,3392,92],{"class":91},[63,3394,3395],{"class":95},"WaitForNextTickAsync",[63,3397,142],{"class":91},[63,3399,614],{"class":528},[63,3401,149],{"class":91},[63,3403,3404],{"class":65,"line":152},[63,3405,588],{"emptyLinePlaceholder":587},[63,3407,3408,3411,3413,3415,3418,3421,3424],{"class":65,"line":253},[63,3409,3410],{"class":439},"    if",[63,3412,3366],{"class":91},[63,3414,3369],{"class":132},[63,3416,3417],{"class":528},"IsLeader",[63,3419,3420],{"class":91},") ",[63,3422,3423],{"class":439},"continue",[63,3425,274],{"class":91},[63,3427,3428,3430,3432,3435,3438,3441,3444],{"class":65,"line":277},[63,3429,3410],{"class":439},[63,3431,3366],{"class":91},[63,3433,3434],{"class":528},"_isJobRunning",[63,3436,3437],{"class":91},") { ",[63,3439,3440],{"class":2731},"\u002F* drop this tick *\u002F",[63,3442,3443],{"class":439}," continue",[63,3445,3446],{"class":91},"; }\n",[63,3448,3449],{"class":65,"line":295},[63,3450,588],{"emptyLinePlaceholder":587},[63,3452,3453,3456,3458,3460],{"class":65,"line":301},[63,3454,3455],{"class":528},"    _isJobRunning",[63,3457,133],{"class":132},[63,3459,1515],{"class":289},[63,3461,274],{"class":91},[63,3463,3464,3467,3470,3472,3474,3476],{"class":65,"line":313},[63,3465,3466],{"class":439},"    try",[63,3468,3469],{"class":91}," { await ",[63,3471,3339],{"class":95},[63,3473,142],{"class":91},[63,3475,614],{"class":528},[63,3477,3478],{"class":91},"); }\n",[63,3480,3481,3484,3487,3489,3491,3494],{"class":65,"line":318},[63,3482,3483],{"class":439},"    finally",[63,3485,3486],{"class":91}," { ",[63,3488,3434],{"class":528},[63,3490,133],{"class":132},[63,3492,3493],{"class":289}," false",[63,3495,3446],{"class":91},[63,3497,3498],{"class":65,"line":340},[63,3499,626],{"class":91},[16,3501,3502,3505,3506,3509,3510,3512,3513,3516],{},[32,3503,3504],{},"PeriodicTimer.WaitForNextTickAsync"," gives you ticks at fixed wall-clock instants instead of drifting like ",[32,3507,3508],{},"Task.Delay"," would. The ",[32,3511,3434],{}," flag is just a ",[32,3514,3515],{},"volatile bool",". If a tick arrives while a previous run is still going, we drop it on the floor.",[16,3518,3519],{},"This is the semantic Hangfire's recurring job runner doesn't give you. Hangfire queues overlapping runs. Mine drops them. For \"run prediction every 500 ms\" workloads, drop is the correct default. A stale prediction is worse than a missed one.",[11,3521,3523],{"id":3522},"aot-and-the-source-generator","AOT and the source generator",[16,3525,3526,3527,3246,3530,3533,3534,3246,3537,3540],{},"The library targets ",[32,3528,3529],{},"net8.0",[32,3531,3532],{},"net10.0",", and it's marked ",[32,3535,3536],{},"IsAotCompatible=true",[32,3538,3539],{},"IsTrimmable=true",". Those flags only mean something if you actually avoid reflection at startup, so I shipped a Roslyn source generator inside the package.",[16,3542,3543,3544,3547],{},"The generator scans your compilation, finds every non-abstract subclass of ",[32,3545,3546],{},"SingletonBackgroundJob",", and emits an extension method directly into your assembly:",[54,3549,3551],{"className":78,"code":3550,"language":80,"meta":59,"style":59},"internal static class SingletonJobGeneratedRegistration\n{\n    internal static IServiceCollection AddSingletonJobs(this IServiceCollection services, IConfiguration? configuration = null)\n    {\n        services.ConfigureSingletonJobOptions(configuration);\n        services.TryAddEnumerable(ServiceDescriptor.Singleton\u003CIHostedService, MyApp.DailyReportJob>());\n        services.TryAddEnumerable(ServiceDescriptor.Singleton\u003CIHostedService, MyApp.HeartbeatJob>());\n        services.TryAddEnumerable(ServiceDescriptor.Singleton\u003CIHostedService, MyApp.PriceTickJob>());\n        return services;\n    }\n}\n",[32,3552,3553,3565,3569,3610,3614,3630,3667,3698,3729,3737,3741],{"__ignoreMap":59},[63,3554,3555,3558,3560,3562],{"class":65,"line":66},[63,3556,3557],{"class":439},"internal",[63,3559,2967],{"class":439},[63,3561,446],{"class":439},[63,3563,3564],{"class":102}," SingletonJobGeneratedRegistration\n",[63,3566,3567],{"class":65,"line":115},[63,3568,118],{"class":91},[63,3570,3571,3574,3576,3579,3582,3584,3587,3589,3592,3594,3597,3600,3603,3605,3608],{"class":65,"line":121},[63,3572,3573],{"class":439},"    internal",[63,3575,2967],{"class":439},[63,3577,3578],{"class":102}," IServiceCollection",[63,3580,3581],{"class":95}," AddSingletonJobs",[63,3583,142],{"class":91},[63,3585,3586],{"class":439},"this",[63,3588,3578],{"class":102},[63,3590,3591],{"class":87}," services",[63,3593,508],{"class":91},[63,3595,3596],{"class":102},"IConfiguration",[63,3598,3599],{"class":91},"? ",[63,3601,3602],{"class":87},"configuration",[63,3604,133],{"class":132},[63,3606,3607],{"class":289}," null",[63,3609,474],{"class":91},[63,3611,3612],{"class":65,"line":152},[63,3613,250],{"class":91},[63,3615,3616,3619,3621,3624,3626,3628],{"class":65,"line":253},[63,3617,3618],{"class":87},"        services",[63,3620,92],{"class":91},[63,3622,3623],{"class":95},"ConfigureSingletonJobOptions",[63,3625,142],{"class":91},[63,3627,3602],{"class":528},[63,3629,149],{"class":91},[63,3631,3632,3634,3636,3639,3641,3644,3646,3649,3651,3654,3656,3659,3661,3664],{"class":65,"line":277},[63,3633,3618],{"class":87},[63,3635,92],{"class":91},[63,3637,3638],{"class":95},"TryAddEnumerable",[63,3640,142],{"class":91},[63,3642,3643],{"class":87},"ServiceDescriptor",[63,3645,92],{"class":91},[63,3647,3648],{"class":95},"Singleton",[63,3650,99],{"class":91},[63,3652,3653],{"class":102},"IHostedService",[63,3655,508],{"class":91},[63,3657,3658],{"class":102},"MyApp",[63,3660,92],{"class":91},[63,3662,3663],{"class":102},"DailyReportJob",[63,3665,3666],{"class":91},">());\n",[63,3668,3669,3671,3673,3675,3677,3679,3681,3683,3685,3687,3689,3691,3693,3696],{"class":65,"line":295},[63,3670,3618],{"class":87},[63,3672,92],{"class":91},[63,3674,3638],{"class":95},[63,3676,142],{"class":91},[63,3678,3643],{"class":87},[63,3680,92],{"class":91},[63,3682,3648],{"class":95},[63,3684,99],{"class":91},[63,3686,3653],{"class":102},[63,3688,508],{"class":91},[63,3690,3658],{"class":102},[63,3692,92],{"class":91},[63,3694,3695],{"class":102},"HeartbeatJob",[63,3697,3666],{"class":91},[63,3699,3700,3702,3704,3706,3708,3710,3712,3714,3716,3718,3720,3722,3724,3727],{"class":65,"line":301},[63,3701,3618],{"class":87},[63,3703,92],{"class":91},[63,3705,3638],{"class":95},[63,3707,142],{"class":91},[63,3709,3643],{"class":87},[63,3711,92],{"class":91},[63,3713,3648],{"class":95},[63,3715,99],{"class":91},[63,3717,3653],{"class":102},[63,3719,508],{"class":91},[63,3721,3658],{"class":102},[63,3723,92],{"class":91},[63,3725,3726],{"class":102},"PriceTickJob",[63,3728,3666],{"class":91},[63,3730,3731,3733,3735],{"class":65,"line":313},[63,3732,593],{"class":439},[63,3734,3591],{"class":528},[63,3736,274],{"class":91},[63,3738,3739],{"class":65,"line":318},[63,3740,621],{"class":91},[63,3742,3743],{"class":65,"line":340},[63,3744,626],{"class":91},[16,3746,3747,3748,3751],{},"So in ",[32,3749,3750],{},"Program.cs"," you write:",[54,3753,3755],{"className":78,"code":3754,"language":80,"meta":59,"style":59},"builder.Services.AddSingletonJobs(builder.Configuration);\n",[32,3756,3757],{"__ignoreMap":59},[63,3758,3759,3761,3763,3765,3767,3770,3772,3774,3776,3779],{"class":65,"line":66},[63,3760,206],{"class":87},[63,3762,92],{"class":91},[63,3764,211],{"class":87},[63,3766,92],{"class":91},[63,3768,3769],{"class":95},"AddSingletonJobs",[63,3771,142],{"class":91},[63,3773,206],{"class":87},[63,3775,92],{"class":91},[63,3777,3778],{"class":87},"Configuration",[63,3780,149],{"class":91},[16,3782,3783,3784,3787],{},"That call expands at compile time into the class above. No ",[32,3785,3786],{},"Assembly.GetTypes()",", no reflection, no trim warnings at publish. I’ve built several source generators before, so this was familiar territory.",[16,3789,3790,3791,3794,3795,3798],{},"One small catch: the generator only runs as part of a build. On a fresh checkout your IDE will scream ",[32,3792,3793],{},"CS1061: 'IServiceCollection' does not contain a definition for 'AddSingletonJobs'"," until you ",[32,3796,3797],{},"dotnet build"," once. After that it resolves and stays resolved.",[11,3800,3778],{"id":3602},[16,3802,3803],{},"Defaults look like:",[54,3805,3809],{"className":3806,"code":3807,"language":3808,"meta":59,"style":59},"language-json shiki shiki-themes one-light one-dark-pro","{\n  \"ConnectionStrings\": { \"Redis\": \"localhost:6379\" },\n  \"SingletonJob\": {\n    \"ProjectName\": \"myapp\",\n    \"HeartbeatInterval\": \"00:00:03\",\n    \"LockExpiry\": \"00:00:10\"\n  }\n}\n","json",[32,3810,3811,3815,3833,3841,3853,3865,3875,3880],{"__ignoreMap":59},[63,3812,3813],{"class":65,"line":66},[63,3814,118],{"class":91},[63,3816,3817,3820,3823,3826,3828,3831],{"class":65,"line":115},[63,3818,3819],{"class":2976},"  \"ConnectionStrings\"",[63,3821,3822],{"class":91},": { ",[63,3824,3825],{"class":2976},"\"Redis\"",[63,3827,227],{"class":91},[63,3829,3830],{"class":145},"\"localhost:6379\"",[63,3832,890],{"class":91},[63,3834,3835,3838],{"class":65,"line":121},[63,3836,3837],{"class":2976},"  \"SingletonJob\"",[63,3839,3840],{"class":91},": {\n",[63,3842,3843,3846,3848,3851],{"class":65,"line":152},[63,3844,3845],{"class":2976},"    \"ProjectName\"",[63,3847,227],{"class":91},[63,3849,3850],{"class":145},"\"myapp\"",[63,3852,233],{"class":91},[63,3854,3855,3858,3860,3863],{"class":65,"line":253},[63,3856,3857],{"class":2976},"    \"HeartbeatInterval\"",[63,3859,227],{"class":91},[63,3861,3862],{"class":145},"\"00:00:03\"",[63,3864,233],{"class":91},[63,3866,3867,3870,3872],{"class":65,"line":277},[63,3868,3869],{"class":2976},"    \"LockExpiry\"",[63,3871,227],{"class":91},[63,3873,3874],{"class":145},"\"00:00:10\"\n",[63,3876,3877],{"class":65,"line":295},[63,3878,3879],{"class":91},"  }\n",[63,3881,3882],{"class":65,"line":301},[63,3883,626],{"class":91},[16,3885,3886,3887,3889],{},"The relationship that actually matters is ",[32,3888,3329],{},". A single dropped network call shouldn't cost you leadership. Three in a row, sure.",[16,3891,3892],{},"Per-job override if you have one heavy job that needs a longer lock:",[54,3894,3896],{"className":78,"code":3895,"language":80,"meta":59,"style":59},"services.PostConfigureSingletonJob(\"heavy-job\", o =>\n{\n    o.LockExpiry = TimeSpan.FromMinutes(5);\n});\n",[32,3897,3898,3918,3922,3946],{"__ignoreMap":59},[63,3899,3900,3902,3904,3907,3909,3912,3914,3916],{"class":65,"line":66},[63,3901,88],{"class":87},[63,3903,92],{"class":91},[63,3905,3906],{"class":95},"PostConfigureSingletonJob",[63,3908,142],{"class":91},[63,3910,3911],{"class":145},"\"heavy-job\"",[63,3913,508],{"class":91},[63,3915,1469],{"class":87},[63,3917,112],{"class":91},[63,3919,3920],{"class":65,"line":115},[63,3921,118],{"class":91},[63,3923,3924,3927,3929,3931,3933,3935,3937,3940,3942,3944],{"class":65,"line":121},[63,3925,3926],{"class":87},"    o",[63,3928,92],{"class":91},[63,3930,3281],{"class":87},[63,3932,133],{"class":132},[63,3934,682],{"class":87},[63,3936,92],{"class":91},[63,3938,3939],{"class":95},"FromMinutes",[63,3941,142],{"class":91},[63,3943,1596],{"class":289},[63,3945,149],{"class":91},[63,3947,3948],{"class":65,"line":152},[63,3949,155],{"class":91},[16,3951,3952],{},"Per-job options are frozen at startup. If you need to change them, redeploy. I went back and forth on whether to support hot reload and eventually convinced myself that hot reloading leader election config is a great way to invent a heisenbug.",[11,3954,3956],{"id":3955},"what-it-does-not-do","What it does not do",[16,3958,3959],{},"Libraries that try to do too much are how you end up rebuilding Hangfire, so the non-goals matter here.",[1789,3961,3962,3968,3971,3974,3977],{},[173,3963,3964,3965,3967],{},"No retries. If ",[32,3966,3339],{}," throws, it's logged and the next tick runs. Want retries? Write them in your handler.",[173,3969,3970],{},"No history. Ticks aren't persisted anywhere. Want a record? Log it yourself.",[173,3972,3973],{},"No dashboard. There is no UI. There never will be.",[173,3975,3976],{},"No cross-pod work distribution. Exactly one pod runs the job, the others sit idle. If you want round-robin or sharded execution, that's a different problem and a different library.",[173,3978,3979],{},"No durability. Jobs are in-memory loops. A pod restart means the loop restarts. That is on purpose.",[16,3981,3982],{},"What's left is the one thing the library actually does: make sure exactly one pod across N replicas runs a given periodic loop, with fast and bounded failover.",[11,3984,3986],{"id":3985},"a-few-things-i-learned-along-the-way","A few things I learned along the way",[16,3988,3989,3997,3998,4001],{},[2055,3990,3991,3994,3995,92],{},[32,3992,3993],{},"volatile"," is the right primitive for ",[32,3996,3417],{}," Single writer (the election loop), many readers (the job loop, the release path). Eventually consistent publication is fine here, because losing leadership only delays a single iteration check by one tick at worst. Reaching for ",[32,3999,4000],{},"Interlocked"," or a lock would be cargo-cult programming, more complexity than the problem needs.",[16,4003,4004,4010,4011,4014],{},[2055,4005,4006,4009],{},[32,4007,4008],{},"PeriodicTimer"," is the right primitive for fixed rate ticks."," It produces ticks at fixed wall-clock instants. ",[32,4012,4013],{},"await Task.Delay(interval)"," does not. The drift adds up over a few hours, and you only notice when you check the timestamps in the logs and realize you've quietly lost a beat.",[16,4016,4017,4020,4021,3255,4023,4025],{},[2055,4018,4019],{},"Lua scripts make Redis atomic for free."," ",[32,4022,3226],{},[32,4024,3230],{}," is two round trips and a race. The Lua version is one round trip and atomic. After writing the first script, the other two were easy: renew, release, and a no-op ownership check.",[16,4027,4028,4031,4032,4035],{},[2055,4029,4030],{},"Backoff with jitter."," When Redis comes back after an outage, you don't want N replicas to all retry at the exact same moment and dogpile the server. The formula is ",[32,4033,4034],{},"delay = min(HeartbeatInterval * 2^failures, MaxBackoffDelay)"," plus or minus 20% jitter. Four lines of code that save you an entire class of follow-on incident.",[16,4037,4038,4041,4042,4045,4046,4048],{},[2055,4039,4040],{},"Cron without a time zone is UTC."," Cronos is great, but the default for \"cron with no time zone\" is UTC. We're in Singapore (UTC+8), so a daily 3 AM job actually fires at 11 AM local. After running into this in a test deployment, I added the optional ",[32,4043,4044],{},"TimeZone"," override on ",[32,4047,2951],{},". If you only ever run in UTC, ignore. If not, set it explicitly, most likely to your server's own local time.",[11,4050,2471],{"id":2470},[54,4052,4054],{"className":56,"code":4053,"language":58,"meta":59,"style":59},"dotnet add package SingletonJob\n",[32,4055,4056],{"__ignoreMap":59},[63,4057,4058],{"class":65,"line":66},[63,4059,4053],{},[16,4061,4062],{},"Or clone the repo and spin up three workers locally:",[54,4064,4066],{"className":56,"code":4065,"language":58,"meta":59,"style":59},"cd samples\ndocker compose up --build --scale worker=3\n",[32,4067,4068,4073],{"__ignoreMap":59},[63,4069,4070],{"class":65,"line":66},[63,4071,4072],{},"cd samples\n",[63,4074,4075],{"class":65,"line":115},[63,4076,4077],{},"docker compose up --build --scale worker=3\n",[16,4079,4080,4081,4084,4085,92],{},"Exactly one of them prints ",[32,4082,4083],{},"became LEADER",". Kill it, and another takes over within ",[32,4086,3092],{},[16,4088,2558,4089],{},[20,4090,2619],{"href":2619,"rel":4091},[52],[16,4093,4094,4095,4098,4099,4102],{},"If you are interested, please study the ",[32,4096,4097],{},"README.md"," and all the technical documents under ",[32,4100,4101],{},".\u002Fdoc",". I'm happy to hear any feedbacks.",[11,4104,4106],{"id":4105},"closing","Closing",[16,4108,4109,4110,4113,4114,4116,4117,4120,4121,4124],{},"I've wanted a library like this to exist for years. Every team I've been on has eventually written some version of it: a half broken ",[32,4111,4112],{},"try\u002Ffinally"," around a Redis ",[32,4115,3234],{},", a hand rolled scheduler that quietly queues runs it should have dropped, and worst of all: a ",[32,4118,4119],{},"Quartz","\u002F",[32,4122,4123],{},"BackgroundService"," that runs in every pod or configured to \"only run on a pod 0\" (extremely brittle). None of them were ever good enough to pull out into a package.",[16,4126,4127],{},"This one I think actually is. The code is short enough to read in one sitting. The surface area is small enough that I keep failing to find new things to add to it. And the design has held up across enough rewrites at work that I'm not nervous about it anymore. If you have a tick driven workload in .NET and you've been fighting Hangfire about it, give this a try.",[2563,4129,4130],{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sW2Sy, html code.shiki .sW2Sy{--shiki-default:#A0A1A7;--shiki-default-font-style:italic;--shiki-dark:#7F848E;--shiki-dark-font-style:italic}html pre.shiki code .sLKXg, html code.shiki .sLKXg{--shiki-default:#A626A4;--shiki-dark:#C678DD}html pre.shiki code .sC09Y, html code.shiki .sC09Y{--shiki-default:#C18401;--shiki-dark:#E5C07B}html pre.shiki code .s5ixo, html code.shiki .s5ixo{--shiki-default:#383A42;--shiki-dark:#ABB2BF}html pre.shiki code .siaei, html code.shiki .siaei{--shiki-default:#4078F2;--shiki-dark:#ABB2BF}html pre.shiki code .sDhpE, html code.shiki .sDhpE{--shiki-default:#50A14F;--shiki-dark:#98C379}html pre.shiki code .sAdtL, html code.shiki .sAdtL{--shiki-default:#4078F2;--shiki-dark:#61AFEF}html pre.shiki code .s7GmK, html code.shiki .s7GmK{--shiki-default:#383A42;--shiki-dark:#E5C07B}html pre.shiki code .sAGMh, html code.shiki .sAGMh{--shiki-default:#986801;--shiki-dark:#D19A66}html pre.shiki code .sJa8x, html code.shiki .sJa8x{--shiki-default:#E45649;--shiki-dark:#E06C75}html pre.shiki code .sknuh, html code.shiki .sknuh{--shiki-default:#383A42;--shiki-dark:#56B6C2}html pre.shiki code .sz0mV, html code.shiki .sz0mV{--shiki-default:#383A42;--shiki-dark:#E06C75}",{"title":59,"searchDepth":115,"depth":115,"links":4132},[4133,4134,4135,4136,4137,4138,4139,4140,4141,4142,4143,4144,4145,4146,4147,4148],{"id":13,"depth":115,"text":14},{"id":2623,"depth":115,"text":2624},{"id":2653,"depth":115,"text":2654},{"id":2680,"depth":115,"text":2681},{"id":2717,"depth":115,"text":2718},{"id":3076,"depth":115,"text":3077},{"id":3183,"depth":115,"text":3184},{"id":3274,"depth":115,"text":3275},{"id":3300,"depth":115,"text":3301},{"id":3346,"depth":115,"text":3347},{"id":3522,"depth":115,"text":3523},{"id":3602,"depth":115,"text":3778},{"id":3955,"depth":115,"text":3956},{"id":3985,"depth":115,"text":3986},{"id":2470,"depth":115,"text":2471},{"id":4105,"depth":115,"text":4106},"2026-05-11","A Redis-backed singleton background job library for high-frequency .NET workloads.",{},"\u002Fblog\u002Fwhy-i-built-singletonjob",{"title":2598,"description":4150},"blog\u002Fwhy-i-built-singletonjob",[2594],"NHt5gVis_1dYUi9Qx7uyTrBCed34BWQPMg5sCG3U0Pc",{"id":4158,"title":4159,"body":4160,"book":2585,"date":4207,"description":4208,"extension":2588,"meta":4209,"navigation":587,"path":4210,"seo":4211,"stem":4212,"tags":4213,"__hash__":4214},"blog\u002Fblog\u002Fbuilding-ml-inference-part-5.md","Building an ML Inference API, Part V",{"type":8,"value":4161,"toc":4205},[4162,4165,4176,4179,4190,4193,4196,4202],[16,4163,4164],{},"Continuing from part IV, I have a few things I want to do:",[170,4166,4167,4170,4173],{},[173,4168,4169],{},"Extend this to XGBoost and CatBoost. These 3 should cover almost all gradient boosting cases.",[173,4171,4172],{},"Make the prompt interactive in terminal (ask for model path, output name, etc.)",[173,4174,4175],{},"Extend to Go because besides C# which I use, Go is a common language used by microservices in cloud.",[16,4177,4178],{},"Since the requirements are quite clear and there is already existing working codes for LightGBM as well as the explanation in part IV, I decided to just pass this to Claude and let it finish (also I'm not that good with Go anyways).",[16,4180,4181,4182,4185,4186,4189],{},"But before that, I created 3 different models based on a small dataset (sklearn ",[32,4183,4184],{},"make_classification",", 10 features, 200 rows). I also created a dotnet test using XUnit to assert that .NET classes called LgbmModel, XgbModel and CbModel (placeholders with ",[32,4187,4188],{},"Predict()"," function) would match the predictions by Python. As long as Claude can make sure all the tests pass, I'm quite confident the task is finished (because of the nature of the work, there is no way it would pass if there is a bug in classification prediction for 200 data).",[16,4191,4192],{},"Anyway, there were some hiccups but Claude managed to fix them by itself after like 20 minutes lol. The TDD approach worked well here (you should not do TDD if you don't know the usefulness, because in fact it's not that useful most of the time).",[16,4194,4195],{},"Here is the git repo if it's useful:",[16,4197,4198],{},[20,4199,4200],{"href":4200,"rel":4201},"https:\u002F\u002Fgithub.com\u002Fhaiilong\u002Fboostexport",[52],[16,4203,4204],{},"With that, I'll close this long chapter on ML inference.",{"title":59,"searchDepth":115,"depth":115,"links":4206},[],"2026-05-07","Extending native inference to other boosting frameworks",{},"\u002Fblog\u002Fbuilding-ml-inference-part-5",{"title":4159,"description":4208},"blog\u002Fbuilding-ml-inference-part-5",[2594],"WmrwX6LQgG7VBn3SDlQX0pjoR6e_CGaZFTMzNmChCOQ",{"id":4216,"title":4217,"body":4218,"book":2585,"date":6665,"description":6666,"extension":2588,"meta":6667,"navigation":587,"path":6668,"seo":6669,"stem":6670,"tags":6671,"__hash__":6672},"blog\u002Fblog\u002Fbuilding-ml-inference-part-4.md","Building an ML Inference API, Part IV",{"type":8,"value":4219,"toc":6640},[4220,4222,4230,4233,4236,4250,4257,4260,4283,4286,4290,4293,4304,4307,4333,4336,4391,4397,4400,4404,4407,4413,4416,4419,4535,4542,4545,4549,4552,4558,4561,4653,4656,4718,4721,4725,4728,4734,4737,4855,4858,4861,4865,4868,5045,5048,5054,5057,5061,5064,5132,5135,5151,5154,5564,5567,5657,5660,5664,5668,5671,5675,5678,5689,5693,5696,5700,5703,5707,5710,5721,5724,5730,5733,5736,5750,5753,5756,5760,5763,5777,5780,5946,5949,6075,6078,6082,6085,6102,6105,6109,6113,6116,6122,6187,6190,6194,6197,6203,6206,6298,6301,6416,6419,6423,6426,6497,6501,6504,6510,6528,6537,6565,6571,6581,6587,6593,6599,6605,6613,6616,6622,6625,6629,6637],[11,4221,14],{"id":13},[16,4223,4224,4225,4229],{},"By the time the FastAPI inference service from ",[20,4226,4228],{"href":4227},"\u002Fblog\u002Fbuilding-ml-inference-part-2","Part II"," was in production, the .NET side around it had all the usual machinery: HTTP clients, retries, timeouts, circuit breakers, tracing. All necessary, but still plumbing. And in some downstream systems, even a clean network call was too expensive.",[16,4231,4232],{},"Most of what we were serving was LightGBM regression. Training, feature engineering, and retraining schedules all belonged in Python. Inference itself was just trees, which means comparisons and additions. There's no reason that has to sit behind another network call.",[16,4234,4235],{},"So I started experimenting with translating trained LightGBM models directly into C#. Then callers could invoke a method instead of making an HTTP request, and the runtime path lost a few things:",[1789,4237,4238,4241,4244,4247],{},[173,4239,4240],{},"the network hop",[173,4242,4243],{},"retry logic around inference",[173,4245,4246],{},"the Python process",[173,4248,4249],{},"a separate service to scale",[16,4251,4252,4253,4256],{},"For regression, the mapping from tree structure to code is almost literal. You generate C# source from the model, compile it into a DLL, and do inference natively inside .NET with no model file at runtime. The catch is that giant generated ",[32,4254,4255],{},"if\u002Felse"," trees get ugly fast, and at some point it's cleaner to stop generating so much branching and represent the trees as data instead, then walk them with a small runtime. That ended up being the approach I preferred.",[16,4258,4259],{},"This post walks through both:",[170,4261,4262,4265,4268,4271,4274,4277,4280],{},[173,4263,4264],{},"Why trees can be represented directly as code",[173,4266,4267],{},"How LightGBM trees map to C#",[173,4269,4270],{},"A tiny concrete example",[173,4272,4273],{},"Generating the if\u002Felse version",[173,4275,4276],{},"Where that starts to break down",[173,4278,4279],{},"An evaluator that treats the model as data",[173,4281,4282],{},"Extending the same idea to classification",[16,4284,4285],{},"Regression first, because it makes the idea easiest to see.",[11,4287,4289],{"id":4288},"a-regression-tree-is-already-code","A regression tree is already code",[16,4291,4292],{},"One mental shift helps a lot:",[16,4294,4295,4296,4298,4299,4020,4302,92],{},"A decision tree is not really something you translate into ",[32,4297,4255],{},". It already ",[3240,4300,4301],{},"is",[32,4303,4255],{},[16,4305,4306],{},"Take a toy tree:",[1789,4308,4309,4323],{},[173,4310,4311,4312,4315],{},"If ",[32,4313,4314],{},"strength_diff \u003C= -2.11",[1789,4316,4317],{},[173,4318,4319,4320],{},"predict ",[32,4321,4322],{},"0.3145",[173,4324,4325,4326],{},"Else\n",[1789,4327,4328],{},[173,4329,4319,4330],{},[32,4331,4332],{},"0.7237",[16,4334,4335],{},"That is literally:",[54,4337,4339],{"className":78,"code":4338,"language":80,"meta":59,"style":59},"if (features[5] \u003C= -2.11)\n    return 0.3145;\nelse\n    return 0.7237;\n",[32,4340,4341,4369,4378,4382],{"__ignoreMap":59},[63,4342,4343,4346,4348,4351,4354,4356,4358,4361,4364,4367],{"class":65,"line":66},[63,4344,4345],{"class":439},"if",[63,4347,3366],{"class":91},[63,4349,4350],{"class":87},"features",[63,4352,4353],{"class":91},"[",[63,4355,1596],{"class":289},[63,4357,574],{"class":91},[63,4359,4360],{"class":132},"\u003C=",[63,4362,4363],{"class":132}," -",[63,4365,4366],{"class":289},"2.11",[63,4368,474],{"class":91},[63,4370,4371,4373,4376],{"class":65,"line":115},[63,4372,1890],{"class":439},[63,4374,4375],{"class":289}," 0.3145",[63,4377,274],{"class":91},[63,4379,4380],{"class":65,"line":121},[63,4381,3138],{"class":439},[63,4383,4384,4386,4389],{"class":65,"line":152},[63,4385,1890],{"class":439},[63,4387,4388],{"class":289}," 0.7237",[63,4390,274],{"class":91},[16,4392,4393,4394,4396],{},"That is not an approximation. That ",[3240,4395,4301],{}," the model.",[16,4398,4399],{},"And that’s why tree models are unusually portable compared with a lot of other ML models.",[11,4401,4403],{"id":4402},"boosting-is-just-many-trees-adding-corrections","Boosting is just many trees adding corrections",[16,4405,4406],{},"A LightGBM regression model is an ensemble:",[54,4408,4411],{"className":4409,"code":4410,"language":2529,"meta":59},[2527],"prediction =\n    tree1(x)\n  + tree2(x)\n  + tree3(x)\n  + ...\n",[32,4412,4410],{"__ignoreMap":59},[16,4414,4415],{},"Each tree contributes a little correction.",[16,4417,4418],{},"In code:",[54,4420,4422],{"className":78,"code":4421,"language":80,"meta":59,"style":59},"public static double Predict(double[] f)\n{\n    double score = 0;\n\n    score += Tree0(f);\n    score += Tree1(f);\n    score += Tree2(f);\n\n    return score;\n}\n",[32,4423,4424,4448,4452,4467,4471,4489,4504,4519,4523,4531],{"__ignoreMap":59},[63,4425,4426,4428,4430,4433,4436,4438,4440,4443,4446],{"class":65,"line":66},[63,4427,440],{"class":439},[63,4429,2967],{"class":439},[63,4431,4432],{"class":439}," double",[63,4434,4435],{"class":95}," Predict",[63,4437,142],{"class":91},[63,4439,861],{"class":439},[63,4441,4442],{"class":91},"[] ",[63,4444,4445],{"class":87},"f",[63,4447,474],{"class":91},[63,4449,4450],{"class":65,"line":115},[63,4451,118],{"class":91},[63,4453,4454,4457,4460,4462,4465],{"class":65,"line":121},[63,4455,4456],{"class":439},"    double",[63,4458,4459],{"class":528}," score",[63,4461,133],{"class":132},[63,4463,4464],{"class":289}," 0",[63,4466,274],{"class":91},[63,4468,4469],{"class":65,"line":152},[63,4470,588],{"emptyLinePlaceholder":587},[63,4472,4473,4476,4480,4483,4485,4487],{"class":65,"line":253},[63,4474,4475],{"class":528},"    score",[63,4477,4479],{"class":4478},"sblXP"," +=",[63,4481,4482],{"class":95}," Tree0",[63,4484,142],{"class":91},[63,4486,4445],{"class":528},[63,4488,149],{"class":91},[63,4490,4491,4493,4495,4498,4500,4502],{"class":65,"line":277},[63,4492,4475],{"class":528},[63,4494,4479],{"class":4478},[63,4496,4497],{"class":95}," Tree1",[63,4499,142],{"class":91},[63,4501,4445],{"class":528},[63,4503,149],{"class":91},[63,4505,4506,4508,4510,4513,4515,4517],{"class":65,"line":295},[63,4507,4475],{"class":528},[63,4509,4479],{"class":4478},[63,4511,4512],{"class":95}," Tree2",[63,4514,142],{"class":91},[63,4516,4445],{"class":528},[63,4518,149],{"class":91},[63,4520,4521],{"class":65,"line":301},[63,4522,588],{"emptyLinePlaceholder":587},[63,4524,4525,4527,4529],{"class":65,"line":313},[63,4526,1890],{"class":439},[63,4528,4459],{"class":528},[63,4530,274],{"class":91},[63,4532,4533],{"class":65,"line":318},[63,4534,626],{"class":91},[16,4536,4537,4538,4541],{},"Each ",[32,4539,4540],{},"TreeN"," is nested branching logic.",[16,4543,4544],{},"That is the whole inference loop for regression.",[11,4546,4548],{"id":4547},"reading-a-lightgbm-tree","Reading a LightGBM tree",[16,4550,4551],{},"A tree dump might look something like:",[54,4553,4556],{"className":4554,"code":4555,"language":2529,"meta":59},[2527],"split_feature: strength_diff\nthreshold: -1.18\n\nleft_child:\n   split_feature: minute\n   threshold: 15\n   left_child: leaf=0.48\n   right_child: leaf=0.62\n\nright_child:\n   leaf=1.03\n",[32,4557,4555],{"__ignoreMap":59},[16,4559,4560],{},"Which maps directly to:",[54,4562,4564],{"className":78,"code":4563,"language":80,"meta":59,"style":59},"if (strengthDiff \u003C= -1.18)\n{\n    if (minute \u003C= 15)\n        return 0.48;\n    else\n        return 0.62;\n}\nelse\n{\n    return 1.03;\n}\n",[32,4565,4566,4585,4589,4605,4614,4619,4628,4632,4636,4640,4649],{"__ignoreMap":59},[63,4567,4568,4570,4572,4575,4578,4580,4583],{"class":65,"line":66},[63,4569,4345],{"class":439},[63,4571,3366],{"class":91},[63,4573,4574],{"class":528},"strengthDiff",[63,4576,4577],{"class":132}," \u003C=",[63,4579,4363],{"class":132},[63,4581,4582],{"class":289},"1.18",[63,4584,474],{"class":91},[63,4586,4587],{"class":65,"line":115},[63,4588,118],{"class":91},[63,4590,4591,4593,4595,4598,4600,4603],{"class":65,"line":121},[63,4592,3410],{"class":439},[63,4594,3366],{"class":91},[63,4596,4597],{"class":528},"minute",[63,4599,4577],{"class":132},[63,4601,4602],{"class":289}," 15",[63,4604,474],{"class":91},[63,4606,4607,4609,4612],{"class":65,"line":152},[63,4608,593],{"class":439},[63,4610,4611],{"class":289}," 0.48",[63,4613,274],{"class":91},[63,4615,4616],{"class":65,"line":253},[63,4617,4618],{"class":439},"    else\n",[63,4620,4621,4623,4626],{"class":65,"line":277},[63,4622,593],{"class":439},[63,4624,4625],{"class":289}," 0.62",[63,4627,274],{"class":91},[63,4629,4630],{"class":65,"line":295},[63,4631,626],{"class":91},[63,4633,4634],{"class":65,"line":301},[63,4635,3138],{"class":439},[63,4637,4638],{"class":65,"line":313},[63,4639,118],{"class":91},[63,4641,4642,4644,4647],{"class":65,"line":318},[63,4643,1890],{"class":439},[63,4645,4646],{"class":289}," 1.03",[63,4648,274],{"class":91},[63,4650,4651],{"class":65,"line":340},[63,4652,626],{"class":91},[16,4654,4655],{},"Pretty much one to one:",[1121,4657,4658,4668],{},[1124,4659,4660],{},[1127,4661,4662,4665],{},[1130,4663,4664],{},"LightGBM",[1130,4666,4667],{},"C#",[1140,4669,4670,4678,4686,4694,4702,4710],{},[1127,4671,4672,4675],{},[1145,4673,4674],{},"split_feature",[1145,4676,4677],{},"feature index",[1127,4679,4680,4683],{},[1145,4681,4682],{},"threshold",[1145,4684,4685],{},"comparison",[1127,4687,4688,4691],{},[1145,4689,4690],{},"left child",[1145,4692,4693],{},"if branch",[1127,4695,4696,4699],{},[1145,4697,4698],{},"right child",[1145,4700,4701],{},"else branch",[1127,4703,4704,4707],{},[1145,4705,4706],{},"leaf value",[1145,4708,4709],{},"returned value",[1127,4711,4712,4715],{},[1145,4713,4714],{},"ensemble",[1145,4716,4717],{},"sum of trees",[16,4719,4720],{},"Once that clicks, everything else follows naturally.",[11,4722,4724],{"id":4723},"tiny-example","Tiny example",[16,4726,4727],{},"Suppose a model has:",[54,4729,4732],{"className":4730,"code":4731,"language":2529,"meta":59},[2527],"Tree=0\nnum_leaves=3\nsplit_feature=5 7\nthreshold=-1.18 15\nleft_child=1 -1\nright_child=2 -2\nleaf_value=0.48 0.62 1.03\n",[32,4733,4731],{"__ignoreMap":59},[16,4735,4736],{},"Generated C#:",[54,4738,4740],{"className":78,"code":4739,"language":80,"meta":59,"style":59},"static double Tree0(double[] f)\n{\n    if (f[5] \u003C= -1.18)\n    {\n        if (f[7] \u003C= 15)\n            return 0.48;\n        else\n            return 0.62;\n    }\n\n    return 1.03;\n}\n",[32,4741,4742,4761,4765,4787,4791,4813,4822,4827,4835,4839,4843,4851],{"__ignoreMap":59},[63,4743,4744,4747,4749,4751,4753,4755,4757,4759],{"class":65,"line":66},[63,4745,4746],{"class":439},"static",[63,4748,4432],{"class":439},[63,4750,4482],{"class":95},[63,4752,142],{"class":91},[63,4754,861],{"class":439},[63,4756,4442],{"class":91},[63,4758,4445],{"class":87},[63,4760,474],{"class":91},[63,4762,4763],{"class":65,"line":115},[63,4764,118],{"class":91},[63,4766,4767,4769,4771,4773,4775,4777,4779,4781,4783,4785],{"class":65,"line":121},[63,4768,3410],{"class":439},[63,4770,3366],{"class":91},[63,4772,4445],{"class":87},[63,4774,4353],{"class":91},[63,4776,1596],{"class":289},[63,4778,574],{"class":91},[63,4780,4360],{"class":132},[63,4782,4363],{"class":132},[63,4784,4582],{"class":289},[63,4786,474],{"class":91},[63,4788,4789],{"class":65,"line":152},[63,4790,250],{"class":91},[63,4792,4793,4796,4798,4800,4802,4805,4807,4809,4811],{"class":65,"line":253},[63,4794,4795],{"class":439},"        if",[63,4797,3366],{"class":91},[63,4799,4445],{"class":87},[63,4801,4353],{"class":91},[63,4803,4804],{"class":289},"7",[63,4806,574],{"class":91},[63,4808,4360],{"class":132},[63,4810,4602],{"class":289},[63,4812,474],{"class":91},[63,4814,4815,4818,4820],{"class":65,"line":277},[63,4816,4817],{"class":439},"            return",[63,4819,4611],{"class":289},[63,4821,274],{"class":91},[63,4823,4824],{"class":65,"line":295},[63,4825,4826],{"class":439},"        else\n",[63,4828,4829,4831,4833],{"class":65,"line":301},[63,4830,4817],{"class":439},[63,4832,4625],{"class":289},[63,4834,274],{"class":91},[63,4836,4837],{"class":65,"line":313},[63,4838,621],{"class":91},[63,4840,4841],{"class":65,"line":318},[63,4842,588],{"emptyLinePlaceholder":587},[63,4844,4845,4847,4849],{"class":65,"line":340},[63,4846,1890],{"class":439},[63,4848,4646],{"class":289},[63,4850,274],{"class":91},[63,4852,4853],{"class":65,"line":369},[63,4854,626],{"class":91},[16,4856,4857],{},"That is the tree.",[16,4859,4860],{},"If you have 300 trees, generate 300 methods. And this is surprisingly workable.",[11,4862,4864],{"id":4863},"verifying-inference","Verifying inference",[16,4866,4867],{},"Using one row from my model:",[54,4869,4871],{"className":78,"code":4870,"language":80,"meta":59,"style":59},"var result = Model.Predict([\n1.2699999809265137,\n3.380000114440918,\n0.7300000190734863,\n1.6299999952316284,\n4.650000095367432,\n-2.1100001335144043,\n1.0,\n28.0,\n0.0,\n0.0,\n2.0,\n2.0,\n-2.0,\n0.0,\n2.0,\n2.0,\n-2.0,\n7.0,\n-0.40880000591278076,\n1.0871999263763428,\n427465.0\n]);\n",[32,4872,4873,4893,4900,4907,4914,4921,4928,4938,4945,4952,4959,4965,4972,4978,4986,4992,4998,5004,5012,5019,5028,5035,5040],{"__ignoreMap":59},[63,4874,4875,4877,4880,4882,4885,4887,4890],{"class":65,"line":66},[63,4876,2067],{"class":439},[63,4878,4879],{"class":528}," result",[63,4881,133],{"class":132},[63,4883,4884],{"class":87}," Model",[63,4886,92],{"class":91},[63,4888,4889],{"class":95},"Predict",[63,4891,4892],{"class":91},"([\n",[63,4894,4895,4898],{"class":65,"line":115},[63,4896,4897],{"class":289},"1.2699999809265137",[63,4899,233],{"class":91},[63,4901,4902,4905],{"class":65,"line":121},[63,4903,4904],{"class":289},"3.380000114440918",[63,4906,233],{"class":91},[63,4908,4909,4912],{"class":65,"line":152},[63,4910,4911],{"class":289},"0.7300000190734863",[63,4913,233],{"class":91},[63,4915,4916,4919],{"class":65,"line":253},[63,4917,4918],{"class":289},"1.6299999952316284",[63,4920,233],{"class":91},[63,4922,4923,4926],{"class":65,"line":277},[63,4924,4925],{"class":289},"4.650000095367432",[63,4927,233],{"class":91},[63,4929,4930,4933,4936],{"class":65,"line":295},[63,4931,4932],{"class":132},"-",[63,4934,4935],{"class":289},"2.1100001335144043",[63,4937,233],{"class":91},[63,4939,4940,4943],{"class":65,"line":301},[63,4941,4942],{"class":289},"1.0",[63,4944,233],{"class":91},[63,4946,4947,4950],{"class":65,"line":313},[63,4948,4949],{"class":289},"28.0",[63,4951,233],{"class":91},[63,4953,4954,4957],{"class":65,"line":318},[63,4955,4956],{"class":289},"0.0",[63,4958,233],{"class":91},[63,4960,4961,4963],{"class":65,"line":340},[63,4962,4956],{"class":289},[63,4964,233],{"class":91},[63,4966,4967,4970],{"class":65,"line":369},[63,4968,4969],{"class":289},"2.0",[63,4971,233],{"class":91},[63,4973,4974,4976],{"class":65,"line":374},[63,4975,4969],{"class":289},[63,4977,233],{"class":91},[63,4979,4980,4982,4984],{"class":65,"line":387},[63,4981,4932],{"class":132},[63,4983,4969],{"class":289},[63,4985,233],{"class":91},[63,4987,4988,4990],{"class":65,"line":392},[63,4989,4956],{"class":289},[63,4991,233],{"class":91},[63,4993,4994,4996],{"class":65,"line":406},[63,4995,4969],{"class":289},[63,4997,233],{"class":91},[63,4999,5000,5002],{"class":65,"line":2931},[63,5001,4969],{"class":289},[63,5003,233],{"class":91},[63,5005,5006,5008,5010],{"class":65,"line":2937},[63,5007,4932],{"class":132},[63,5009,4969],{"class":289},[63,5011,233],{"class":91},[63,5013,5014,5017],{"class":65,"line":2956},[63,5015,5016],{"class":289},"7.0",[63,5018,233],{"class":91},[63,5020,5021,5023,5026],{"class":65,"line":2961},[63,5022,4932],{"class":132},[63,5024,5025],{"class":289},"0.40880000591278076",[63,5027,233],{"class":91},[63,5029,5030,5033],{"class":65,"line":3000},[63,5031,5032],{"class":289},"1.0871999263763428",[63,5034,233],{"class":91},[63,5036,5037],{"class":65,"line":3018},[63,5038,5039],{"class":289},"427465.0\n",[63,5041,5042],{"class":65,"line":3037},[63,5043,5044],{"class":91},"]);\n",[16,5046,5047],{},"Expected:",[54,5049,5052],{"className":5050,"code":5051,"language":2529,"meta":59},[2527],"0.31458735\n",[32,5053,5051],{"__ignoreMap":59},[16,5055,5056],{},"Generated predictor matched exactly.",[11,5058,5060],{"id":5059},"generating-the-c","Generating the C#",[16,5062,5063],{},"This should absolutely be generated. Do not hand write trees. For any real model, it gets impossible very quickly. I used Python to generate the C# code, but the language of the exporter does not matter much.",[54,5065,5069],{"className":5066,"code":5067,"language":5068,"meta":59,"style":59},"language-python shiki shiki-themes one-light one-dark-pro","import lightgbm as lgb\n\nbooster = lgb.Booster(model_file=\"model.txt\")\nmodel_dump = booster.dump_model()\n","python",[32,5070,5071,5085,5089,5116],{"__ignoreMap":59},[63,5072,5073,5076,5079,5082],{"class":65,"line":66},[63,5074,5075],{"class":439},"import",[63,5077,5078],{"class":91}," lightgbm ",[63,5080,5081],{"class":439},"as",[63,5083,5084],{"class":91}," lgb\n",[63,5086,5087],{"class":65,"line":115},[63,5088,588],{"emptyLinePlaceholder":587},[63,5090,5091,5094,5096,5099,5103,5105,5109,5111,5114],{"class":65,"line":121},[63,5092,5093],{"class":91},"booster ",[63,5095,577],{"class":132},[63,5097,5098],{"class":91}," lgb.",[63,5100,5102],{"class":5101},"slOjB","Booster",[63,5104,142],{"class":91},[63,5106,5108],{"class":5107},"sp7wS","model_file",[63,5110,577],{"class":132},[63,5112,5113],{"class":145},"\"model.txt\"",[63,5115,474],{"class":91},[63,5117,5118,5121,5123,5126,5129],{"class":65,"line":152},[63,5119,5120],{"class":91},"model_dump ",[63,5122,577],{"class":132},[63,5124,5125],{"class":91}," booster.",[63,5127,5128],{"class":5101},"dump_model",[63,5130,5131],{"class":91},"()\n",[16,5133,5134],{},"Trees live in:",[54,5136,5138],{"className":5066,"code":5137,"language":5068,"meta":59,"style":59},"model_dump[\"tree_info\"]\n",[32,5139,5140],{"__ignoreMap":59},[63,5141,5142,5145,5148],{"class":65,"line":66},[63,5143,5144],{"class":91},"model_dump[",[63,5146,5147],{"class":145},"\"tree_info\"",[63,5149,5150],{"class":91},"]\n",[16,5152,5153],{},"Recursive emitter:",[54,5155,5157],{"className":5066,"code":5156,"language":5068,"meta":59,"style":59},"def emit_node(node, indent=1):\n    pad = \"    \" * indent\n\n    if \"leaf_value\" in node:\n        return f\"{pad}return {node['leaf_value']};\\n\"\n\n    feature = node[\"split_feature\"]\n    threshold = node[\"threshold\"]\n\n    code = []\n\n    code.append(\n        f\"{pad}if (features[{feature}] \u003C= {threshold})\\n\"\n        f\"{pad}{{\\n\"\n    )\n\n    code.append(emit_node(node[\"left_child\"], indent+1))\n\n    code.append(\n        f\"{pad}}}\\n\"\n        f\"{pad}else\\n\"\n        f\"{pad}{{\\n\"\n    )\n\n    code.append(emit_node(node[\"right_child\"], indent+1))\n\n    code.append(f\"{pad}}}\\n\")\n\n    return \"\".join(code)\n",[32,5158,5159,5186,5202,5206,5219,5264,5268,5283,5297,5301,5311,5315,5325,5364,5381,5386,5390,5417,5421,5429,5446,5465,5481,5485,5489,5513,5518,5543,5548],{"__ignoreMap":59},[63,5160,5161,5164,5167,5169,5173,5176,5179,5181,5183],{"class":65,"line":66},[63,5162,5163],{"class":439},"def",[63,5165,5166],{"class":95}," emit_node",[63,5168,142],{"class":91},[63,5170,5172],{"class":5171},"so_Uh","node",[63,5174,5175],{"class":91},",",[63,5177,5178],{"class":5171}," indent",[63,5180,577],{"class":91},[63,5182,880],{"class":289},[63,5184,5185],{"class":91},"):\n",[63,5187,5188,5191,5193,5196,5199],{"class":65,"line":115},[63,5189,5190],{"class":91},"    pad ",[63,5192,577],{"class":132},[63,5194,5195],{"class":145}," \"    \"",[63,5197,5198],{"class":132}," *",[63,5200,5201],{"class":91}," indent\n",[63,5203,5204],{"class":65,"line":121},[63,5205,588],{"emptyLinePlaceholder":587},[63,5207,5208,5210,5213,5216],{"class":65,"line":152},[63,5209,3410],{"class":439},[63,5211,5212],{"class":145}," \"leaf_value\"",[63,5214,5215],{"class":439}," in",[63,5217,5218],{"class":91}," node:\n",[63,5220,5221,5223,5226,5229,5232,5235,5238,5241,5243,5246,5249,5252,5254,5257,5261],{"class":65,"line":253},[63,5222,593],{"class":439},[63,5224,5225],{"class":439}," f",[63,5227,5228],{"class":145},"\"",[63,5230,5231],{"class":289},"{",[63,5233,5234],{"class":91},"pad",[63,5236,5237],{"class":289},"}",[63,5239,5240],{"class":145},"return ",[63,5242,5231],{"class":289},[63,5244,5245],{"class":91},"node[",[63,5247,5248],{"class":145},"'leaf_value'",[63,5250,5251],{"class":91},"]",[63,5253,5237],{"class":289},[63,5255,5256],{"class":145},";",[63,5258,5260],{"class":5259},"s_Sar","\\n",[63,5262,5263],{"class":145},"\"\n",[63,5265,5266],{"class":65,"line":277},[63,5267,588],{"emptyLinePlaceholder":587},[63,5269,5270,5273,5275,5278,5281],{"class":65,"line":295},[63,5271,5272],{"class":91},"    feature ",[63,5274,577],{"class":132},[63,5276,5277],{"class":91}," node[",[63,5279,5280],{"class":145},"\"split_feature\"",[63,5282,5150],{"class":91},[63,5284,5285,5288,5290,5292,5295],{"class":65,"line":301},[63,5286,5287],{"class":91},"    threshold ",[63,5289,577],{"class":132},[63,5291,5277],{"class":91},[63,5293,5294],{"class":145},"\"threshold\"",[63,5296,5150],{"class":91},[63,5298,5299],{"class":65,"line":313},[63,5300,588],{"emptyLinePlaceholder":587},[63,5302,5303,5306,5308],{"class":65,"line":318},[63,5304,5305],{"class":91},"    code ",[63,5307,577],{"class":132},[63,5309,5310],{"class":91}," []\n",[63,5312,5313],{"class":65,"line":340},[63,5314,588],{"emptyLinePlaceholder":587},[63,5316,5317,5320,5323],{"class":65,"line":369},[63,5318,5319],{"class":91},"    code.",[63,5321,5322],{"class":5101},"append",[63,5324,219],{"class":91},[63,5326,5327,5330,5332,5334,5336,5338,5341,5343,5346,5348,5351,5353,5355,5357,5360,5362],{"class":65,"line":374},[63,5328,5329],{"class":439},"        f",[63,5331,5228],{"class":145},[63,5333,5231],{"class":289},[63,5335,5234],{"class":91},[63,5337,5237],{"class":289},[63,5339,5340],{"class":145},"if (features[",[63,5342,5231],{"class":289},[63,5344,5345],{"class":91},"feature",[63,5347,5237],{"class":289},[63,5349,5350],{"class":145},"] \u003C= ",[63,5352,5231],{"class":289},[63,5354,4682],{"class":91},[63,5356,5237],{"class":289},[63,5358,5359],{"class":145},")",[63,5361,5260],{"class":5259},[63,5363,5263],{"class":145},[63,5365,5366,5368,5370,5372,5374,5376,5379],{"class":65,"line":387},[63,5367,5329],{"class":439},[63,5369,5228],{"class":145},[63,5371,5231],{"class":289},[63,5373,5234],{"class":91},[63,5375,5237],{"class":289},[63,5377,5378],{"class":5259},"{{\\n",[63,5380,5263],{"class":145},[63,5382,5383],{"class":65,"line":392},[63,5384,5385],{"class":91},"    )\n",[63,5387,5388],{"class":65,"line":406},[63,5389,588],{"emptyLinePlaceholder":587},[63,5391,5392,5394,5396,5398,5401,5404,5407,5410,5413,5415],{"class":65,"line":2931},[63,5393,5319],{"class":91},[63,5395,5322],{"class":5101},[63,5397,142],{"class":91},[63,5399,5400],{"class":5101},"emit_node",[63,5402,5403],{"class":91},"(node[",[63,5405,5406],{"class":145},"\"left_child\"",[63,5408,5409],{"class":91},"], indent",[63,5411,5412],{"class":132},"+",[63,5414,880],{"class":289},[63,5416,1106],{"class":91},[63,5418,5419],{"class":65,"line":2937},[63,5420,588],{"emptyLinePlaceholder":587},[63,5422,5423,5425,5427],{"class":65,"line":2956},[63,5424,5319],{"class":91},[63,5426,5322],{"class":5101},[63,5428,219],{"class":91},[63,5430,5431,5433,5435,5437,5439,5441,5444],{"class":65,"line":2961},[63,5432,5329],{"class":439},[63,5434,5228],{"class":145},[63,5436,5231],{"class":289},[63,5438,5234],{"class":91},[63,5440,5237],{"class":289},[63,5442,5443],{"class":5259},"}}\\n",[63,5445,5263],{"class":145},[63,5447,5448,5450,5452,5454,5456,5458,5461,5463],{"class":65,"line":3000},[63,5449,5329],{"class":439},[63,5451,5228],{"class":145},[63,5453,5231],{"class":289},[63,5455,5234],{"class":91},[63,5457,5237],{"class":289},[63,5459,5460],{"class":145},"else",[63,5462,5260],{"class":5259},[63,5464,5263],{"class":145},[63,5466,5467,5469,5471,5473,5475,5477,5479],{"class":65,"line":3018},[63,5468,5329],{"class":439},[63,5470,5228],{"class":145},[63,5472,5231],{"class":289},[63,5474,5234],{"class":91},[63,5476,5237],{"class":289},[63,5478,5378],{"class":5259},[63,5480,5263],{"class":145},[63,5482,5483],{"class":65,"line":3037},[63,5484,5385],{"class":91},[63,5486,5487],{"class":65,"line":3056},[63,5488,588],{"emptyLinePlaceholder":587},[63,5490,5492,5494,5496,5498,5500,5502,5505,5507,5509,5511],{"class":65,"line":5491},25,[63,5493,5319],{"class":91},[63,5495,5322],{"class":5101},[63,5497,142],{"class":91},[63,5499,5400],{"class":5101},[63,5501,5403],{"class":91},[63,5503,5504],{"class":145},"\"right_child\"",[63,5506,5409],{"class":91},[63,5508,5412],{"class":132},[63,5510,880],{"class":289},[63,5512,1106],{"class":91},[63,5514,5516],{"class":65,"line":5515},26,[63,5517,588],{"emptyLinePlaceholder":587},[63,5519,5521,5523,5525,5527,5529,5531,5533,5535,5537,5539,5541],{"class":65,"line":5520},27,[63,5522,5319],{"class":91},[63,5524,5322],{"class":5101},[63,5526,142],{"class":91},[63,5528,4445],{"class":439},[63,5530,5228],{"class":145},[63,5532,5231],{"class":289},[63,5534,5234],{"class":91},[63,5536,5237],{"class":289},[63,5538,5443],{"class":5259},[63,5540,5228],{"class":145},[63,5542,474],{"class":91},[63,5544,5546],{"class":65,"line":5545},28,[63,5547,588],{"emptyLinePlaceholder":587},[63,5549,5551,5553,5556,5558,5561],{"class":65,"line":5550},29,[63,5552,1890],{"class":439},[63,5554,5555],{"class":145}," \"\"",[63,5557,92],{"class":91},[63,5559,5560],{"class":5101},"join",[63,5562,5563],{"class":91},"(code)\n",[16,5565,5566],{},"Generate methods:",[54,5568,5570],{"className":5066,"code":5569,"language":5068,"meta":59,"style":59},"for i, tree in enumerate(model_dump[\"tree_info\"]):\n    print(f\"static double Tree{i}(double[] features)\")\n    print(\"{\")\n    print(emit_node(tree[\"tree_structure\"]))\n    print(\"}\")\n",[32,5571,5572,5594,5618,5629,5646],{"__ignoreMap":59},[63,5573,5574,5577,5580,5583,5586,5589,5591],{"class":65,"line":66},[63,5575,5576],{"class":439},"for",[63,5578,5579],{"class":91}," i, tree ",[63,5581,5582],{"class":439},"in",[63,5584,5585],{"class":5259}," enumerate",[63,5587,5588],{"class":91},"(model_dump[",[63,5590,5147],{"class":145},[63,5592,5593],{"class":91},"]):\n",[63,5595,5596,5599,5601,5603,5606,5608,5611,5613,5616],{"class":65,"line":115},[63,5597,5598],{"class":5259},"    print",[63,5600,142],{"class":91},[63,5602,4445],{"class":439},[63,5604,5605],{"class":145},"\"static double Tree",[63,5607,5231],{"class":289},[63,5609,5610],{"class":91},"i",[63,5612,5237],{"class":289},[63,5614,5615],{"class":145},"(double[] features)\"",[63,5617,474],{"class":91},[63,5619,5620,5622,5624,5627],{"class":65,"line":121},[63,5621,5598],{"class":5259},[63,5623,142],{"class":91},[63,5625,5626],{"class":145},"\"{\"",[63,5628,474],{"class":91},[63,5630,5631,5633,5635,5637,5640,5643],{"class":65,"line":152},[63,5632,5598],{"class":5259},[63,5634,142],{"class":91},[63,5636,5400],{"class":5101},[63,5638,5639],{"class":91},"(tree[",[63,5641,5642],{"class":145},"\"tree_structure\"",[63,5644,5645],{"class":91},"]))\n",[63,5647,5648,5650,5652,5655],{"class":65,"line":253},[63,5649,5598],{"class":5259},[63,5651,142],{"class":91},[63,5653,5654],{"class":145},"\"}\"",[63,5656,474],{"class":91},[16,5658,5659],{},"The basic version is quite small.",[11,5661,5663],{"id":5662},"why-i-liked-this","Why I liked this",[720,5665,5667],{"id":5666},"native-inference","Native inference",[16,5669,5670],{},"The generated predictor does not need a Python runtime or a LightGBM dependency. It is just .NET code.",[720,5672,5674],{"id":5673},"speed","Speed",[16,5676,5677],{},"Inference becomes a small set of cheap operations:",[1789,5679,5680,5683,5686],{},[173,5681,5682],{},"comparisons",[173,5684,5685],{},"branches",[173,5687,5688],{},"additions",[720,5690,5692],{"id":5691},"deployment","Deployment",[16,5694,5695],{},"The model becomes source. Ship a DLL and you've shipped the model.",[720,5697,5699],{"id":5698},"debugging","Debugging",[16,5701,5702],{},"You can inspect actual decision paths, which is useful in pricing or risk sensitive systems.",[11,5704,5706],{"id":5705},"where-it-starts-breaking-down","Where it starts breaking down",[16,5708,5709],{},"Say:",[1789,5711,5712,5715,5718],{},[173,5713,5714],{},"500 trees",[173,5716,5717],{},"depth 8",[173,5719,5720],{},"~255 nodes each",[16,5722,5723],{},"That’s potentially:",[54,5725,5728],{"className":5726,"code":5727,"language":2529,"meta":59},[2527],"127,500 node checks\n",[32,5729,5727],{"__ignoreMap":59},[16,5731,5732],{},"Now generated code gets huge.",[16,5734,5735],{},"You start getting:",[1789,5737,5738,5741,5744,5747],{},[173,5739,5740],{},"giant files",[173,5742,5743],{},"ugly diffs",[173,5745,5746],{},"slower compile times",[173,5748,5749],{},"questionable JIT behavior",[16,5751,5752],{},"It still works. For example, one of my models had 150 trees and 25 features, and the generated C# was about 130k lines. If you use Rider or another IDE that parses C#, you will feel it slow down.",[16,5754,5755],{},"That pushed me toward a better representation.",[11,5757,5759],{"id":5758},"represent-the-model-as-data","Represent the model as data",[16,5761,5762],{},"Instead of generating giant branch forests, export the model the way it already exists internally:",[1789,5764,5765,5768,5771,5774],{},[173,5766,5767],{},"feature arrays",[173,5769,5770],{},"thresholds",[173,5772,5773],{},"child pointers",[173,5775,5776],{},"leaf values",[16,5778,5779],{},"Then use a tiny evaluator:",[54,5781,5783],{"className":78,"code":5782,"language":80,"meta":59,"style":59},"private static double Eval(int node, ReadOnlySpan\u003Cdouble> f)\n{\n    while (true)\n    {\n        if (IsLeaf[node])\n            return Value[node];\n\n        if (f[Feature[node]] \u003C= Threshold[node])\n            node = Left[node];\n        else\n            node = Right[node];\n    }\n}\n",[32,5784,5785,5819,5823,5834,5838,5854,5868,5872,5903,5919,5923,5938,5942],{"__ignoreMap":59},[63,5786,5787,5790,5792,5794,5797,5799,5801,5804,5806,5809,5811,5813,5815,5817],{"class":65,"line":66},[63,5788,5789],{"class":439},"private",[63,5791,2967],{"class":439},[63,5793,4432],{"class":439},[63,5795,5796],{"class":95}," Eval",[63,5798,142],{"class":91},[63,5800,856],{"class":439},[63,5802,5803],{"class":87}," node",[63,5805,508],{"class":91},[63,5807,5808],{"class":102},"ReadOnlySpan",[63,5810,99],{"class":91},[63,5812,861],{"class":439},[63,5814,1847],{"class":91},[63,5816,4445],{"class":87},[63,5818,474],{"class":91},[63,5820,5821],{"class":65,"line":115},[63,5822,118],{"class":91},[63,5824,5825,5828,5830,5832],{"class":65,"line":121},[63,5826,5827],{"class":439},"    while",[63,5829,3366],{"class":91},[63,5831,1206],{"class":289},[63,5833,474],{"class":91},[63,5835,5836],{"class":65,"line":152},[63,5837,250],{"class":91},[63,5839,5840,5842,5844,5847,5849,5851],{"class":65,"line":253},[63,5841,4795],{"class":439},[63,5843,3366],{"class":91},[63,5845,5846],{"class":87},"IsLeaf",[63,5848,4353],{"class":91},[63,5850,5172],{"class":528},[63,5852,5853],{"class":91},"])\n",[63,5855,5856,5858,5861,5863,5865],{"class":65,"line":277},[63,5857,4817],{"class":439},[63,5859,5860],{"class":87}," Value",[63,5862,4353],{"class":91},[63,5864,5172],{"class":528},[63,5866,5867],{"class":91},"];\n",[63,5869,5870],{"class":65,"line":295},[63,5871,588],{"emptyLinePlaceholder":587},[63,5873,5874,5876,5878,5880,5882,5885,5887,5889,5892,5894,5897,5899,5901],{"class":65,"line":301},[63,5875,4795],{"class":439},[63,5877,3366],{"class":91},[63,5879,4445],{"class":87},[63,5881,4353],{"class":91},[63,5883,5884],{"class":87},"Feature",[63,5886,4353],{"class":91},[63,5888,5172],{"class":528},[63,5890,5891],{"class":91},"]] ",[63,5893,4360],{"class":132},[63,5895,5896],{"class":87}," Threshold",[63,5898,4353],{"class":91},[63,5900,5172],{"class":528},[63,5902,5853],{"class":91},[63,5904,5905,5908,5910,5913,5915,5917],{"class":65,"line":313},[63,5906,5907],{"class":528},"            node",[63,5909,133],{"class":132},[63,5911,5912],{"class":87}," Left",[63,5914,4353],{"class":91},[63,5916,5172],{"class":528},[63,5918,5867],{"class":91},[63,5920,5921],{"class":65,"line":318},[63,5922,4826],{"class":439},[63,5924,5925,5927,5929,5932,5934,5936],{"class":65,"line":340},[63,5926,5907],{"class":528},[63,5928,133],{"class":132},[63,5930,5931],{"class":87}," Right",[63,5933,4353],{"class":91},[63,5935,5172],{"class":528},[63,5937,5867],{"class":91},[63,5939,5940],{"class":65,"line":369},[63,5941,621],{"class":91},[63,5943,5944],{"class":65,"line":374},[63,5945,626],{"class":91},[16,5947,5948],{},"Boosting:",[54,5950,5952],{"className":78,"code":5951,"language":80,"meta":59,"style":59},"public static double Predict(ReadOnlySpan\u003Cdouble> f)\n{\n    double score = 0;\n\n    for (int i = 0; i \u003C TreeCount; i++)\n        score += Eval(Roots[i], f);\n\n    return score;\n}\n",[32,5953,5954,5978,5982,5994,5998,6034,6059,6063,6071],{"__ignoreMap":59},[63,5955,5956,5958,5960,5962,5964,5966,5968,5970,5972,5974,5976],{"class":65,"line":66},[63,5957,440],{"class":439},[63,5959,2967],{"class":439},[63,5961,4432],{"class":439},[63,5963,4435],{"class":95},[63,5965,142],{"class":91},[63,5967,5808],{"class":102},[63,5969,99],{"class":91},[63,5971,861],{"class":439},[63,5973,1847],{"class":91},[63,5975,4445],{"class":87},[63,5977,474],{"class":91},[63,5979,5980],{"class":65,"line":115},[63,5981,118],{"class":91},[63,5983,5984,5986,5988,5990,5992],{"class":65,"line":121},[63,5985,4456],{"class":439},[63,5987,4459],{"class":528},[63,5989,133],{"class":132},[63,5991,4464],{"class":289},[63,5993,274],{"class":91},[63,5995,5996],{"class":65,"line":152},[63,5997,588],{"emptyLinePlaceholder":587},[63,5999,6000,6003,6005,6007,6010,6012,6014,6017,6019,6022,6025,6027,6029,6032],{"class":65,"line":253},[63,6001,6002],{"class":439},"    for",[63,6004,3366],{"class":91},[63,6006,856],{"class":439},[63,6008,6009],{"class":528}," i",[63,6011,133],{"class":132},[63,6013,4464],{"class":289},[63,6015,6016],{"class":91},"; ",[63,6018,5610],{"class":528},[63,6020,6021],{"class":132}," \u003C",[63,6023,6024],{"class":528}," TreeCount",[63,6026,6016],{"class":91},[63,6028,5610],{"class":528},[63,6030,6031],{"class":132},"++",[63,6033,474],{"class":91},[63,6035,6036,6039,6041,6043,6045,6048,6050,6052,6055,6057],{"class":65,"line":277},[63,6037,6038],{"class":528},"        score",[63,6040,4479],{"class":4478},[63,6042,5796],{"class":95},[63,6044,142],{"class":91},[63,6046,6047],{"class":87},"Roots",[63,6049,4353],{"class":91},[63,6051,5610],{"class":528},[63,6053,6054],{"class":91},"], ",[63,6056,4445],{"class":528},[63,6058,149],{"class":91},[63,6060,6061],{"class":65,"line":295},[63,6062,588],{"emptyLinePlaceholder":587},[63,6064,6065,6067,6069],{"class":65,"line":301},[63,6066,1890],{"class":439},[63,6068,4459],{"class":528},[63,6070,274],{"class":91},[63,6072,6073],{"class":65,"line":313},[63,6074,626],{"class":91},[16,6076,6077],{},"Still exact inference. Just much cleaner.",[11,6079,6081],{"id":6080},"why-i-ended-up-preferring-this","Why I ended up preferring this",[16,6083,6084],{},"Compared with giant generated if\u002Felse:",[1789,6086,6087,6090,6093,6096,6099],{},[173,6088,6089],{},"much smaller generated source",[173,6091,6092],{},"one evaluator method",[173,6094,6095],{},"cleaner diffs",[173,6097,6098],{},"easier codegen",[173,6100,6101],{},"friendlier for JIT",[16,6103,6104],{},"And source size grows mostly with model data, not duplicated branching syntax.",[11,6106,6108],{"id":6107},"classification-extends-naturally","Classification extends naturally",[720,6110,6112],{"id":6111},"binary-classification","Binary classification",[16,6114,6115],{},"Same trees. Usually sum the logits, then apply sigmoid:",[54,6117,6120],{"className":6118,"code":6119,"language":2529,"meta":59},[2527],"probability = sigmoid(sum)\n",[32,6121,6119],{"__ignoreMap":59},[54,6123,6125],{"className":78,"code":6124,"language":80,"meta":59,"style":59},"static double Sigmoid(double x)\n{\n   return 1 \u002F (1 + Math.Exp(-x));\n}\n",[32,6126,6127,6145,6149,6183],{"__ignoreMap":59},[63,6128,6129,6131,6133,6136,6138,6140,6143],{"class":65,"line":66},[63,6130,4746],{"class":439},[63,6132,4432],{"class":439},[63,6134,6135],{"class":95}," Sigmoid",[63,6137,142],{"class":91},[63,6139,861],{"class":439},[63,6141,6142],{"class":87}," x",[63,6144,474],{"class":91},[63,6146,6147],{"class":65,"line":115},[63,6148,118],{"class":91},[63,6150,6151,6154,6156,6159,6161,6163,6166,6169,6171,6174,6176,6178,6181],{"class":65,"line":121},[63,6152,6153],{"class":439},"   return",[63,6155,887],{"class":289},[63,6157,6158],{"class":132}," \u002F",[63,6160,3366],{"class":91},[63,6162,880],{"class":289},[63,6164,6165],{"class":132}," +",[63,6167,6168],{"class":87}," Math",[63,6170,92],{"class":91},[63,6172,6173],{"class":95},"Exp",[63,6175,142],{"class":91},[63,6177,4932],{"class":132},[63,6179,6180],{"class":528},"x",[63,6182,366],{"class":91},[63,6184,6185],{"class":65,"line":152},[63,6186,626],{"class":91},[16,6188,6189],{},"Then threshold. Same traversal, different output transform.",[720,6191,6193],{"id":6192},"multiclass","Multiclass",[16,6195,6196],{},"Often:",[54,6198,6201],{"className":6199,"code":6200,"language":2529,"meta":59},[2527],"num_classes × boosting_rounds trees\n",[32,6202,6200],{"__ignoreMap":59},[16,6204,6205],{},"Accumulate per class:",[54,6207,6209],{"className":78,"code":6208,"language":80,"meta":59,"style":59},"double[] scores = new double[3];\n\nscores[0] += Tree0(f);\nscores[1] += Tree1(f);\nscores[2] += Tree2(f);\n",[32,6210,6211,6232,6236,6257,6277],{"__ignoreMap":59},[63,6212,6213,6215,6217,6220,6222,6224,6226,6228,6230],{"class":65,"line":66},[63,6214,861],{"class":439},[63,6216,4442],{"class":91},[63,6218,6219],{"class":528},"scores",[63,6221,133],{"class":132},[63,6223,136],{"class":91},[63,6225,861],{"class":439},[63,6227,4353],{"class":91},[63,6229,692],{"class":289},[63,6231,5867],{"class":91},[63,6233,6234],{"class":65,"line":115},[63,6235,588],{"emptyLinePlaceholder":587},[63,6237,6238,6240,6242,6244,6246,6249,6251,6253,6255],{"class":65,"line":121},[63,6239,6219],{"class":87},[63,6241,4353],{"class":91},[63,6243,867],{"class":289},[63,6245,574],{"class":91},[63,6247,6248],{"class":4478},"+=",[63,6250,4482],{"class":95},[63,6252,142],{"class":91},[63,6254,4445],{"class":528},[63,6256,149],{"class":91},[63,6258,6259,6261,6263,6265,6267,6269,6271,6273,6275],{"class":65,"line":152},[63,6260,6219],{"class":87},[63,6262,4353],{"class":91},[63,6264,880],{"class":289},[63,6266,574],{"class":91},[63,6268,6248],{"class":4478},[63,6270,4497],{"class":95},[63,6272,142],{"class":91},[63,6274,4445],{"class":528},[63,6276,149],{"class":91},[63,6278,6279,6281,6283,6286,6288,6290,6292,6294,6296],{"class":65,"line":253},[63,6280,6219],{"class":87},[63,6282,4353],{"class":91},[63,6284,6285],{"class":289},"2",[63,6287,574],{"class":91},[63,6289,6248],{"class":4478},[63,6291,4512],{"class":95},[63,6293,142],{"class":91},[63,6295,4445],{"class":528},[63,6297,149],{"class":91},[16,6299,6300],{},"Then softmax:",[54,6302,6304],{"className":78,"code":6303,"language":80,"meta":59,"style":59},"static double[] Softmax(double[] x)\n{\n    var exp = x.Select(Math.Exp).ToArray();\n    var sum = exp.Sum();\n    return exp.Select(v => v \u002F sum).ToArray();\n}\n",[32,6305,6306,6327,6331,6365,6383,6412],{"__ignoreMap":59},[63,6307,6308,6310,6312,6314,6317,6319,6321,6323,6325],{"class":65,"line":66},[63,6309,4746],{"class":439},[63,6311,4432],{"class":439},[63,6313,4442],{"class":91},[63,6315,6316],{"class":95},"Softmax",[63,6318,142],{"class":91},[63,6320,861],{"class":439},[63,6322,4442],{"class":91},[63,6324,6180],{"class":87},[63,6326,474],{"class":91},[63,6328,6329],{"class":65,"line":115},[63,6330,118],{"class":91},[63,6332,6333,6336,6339,6341,6343,6345,6348,6350,6353,6355,6357,6360,6363],{"class":65,"line":121},[63,6334,6335],{"class":439},"    var",[63,6337,6338],{"class":528}," exp",[63,6340,133],{"class":132},[63,6342,6142],{"class":87},[63,6344,92],{"class":91},[63,6346,6347],{"class":95},"Select",[63,6349,142],{"class":91},[63,6351,6352],{"class":87},"Math",[63,6354,92],{"class":91},[63,6356,6173],{"class":87},[63,6358,6359],{"class":91},").",[63,6361,6362],{"class":95},"ToArray",[63,6364,403],{"class":91},[63,6366,6367,6369,6372,6374,6376,6378,6381],{"class":65,"line":152},[63,6368,6335],{"class":439},[63,6370,6371],{"class":528}," sum",[63,6373,133],{"class":132},[63,6375,6338],{"class":87},[63,6377,92],{"class":91},[63,6379,6380],{"class":95},"Sum",[63,6382,403],{"class":91},[63,6384,6385,6387,6389,6391,6393,6395,6398,6400,6402,6404,6406,6408,6410],{"class":65,"line":253},[63,6386,1890],{"class":439},[63,6388,6338],{"class":87},[63,6390,92],{"class":91},[63,6392,6347],{"class":95},[63,6394,142],{"class":91},[63,6396,6397],{"class":87},"v",[63,6399,784],{"class":91},[63,6401,6397],{"class":528},[63,6403,6158],{"class":132},[63,6405,6371],{"class":528},[63,6407,6359],{"class":91},[63,6409,6362],{"class":95},[63,6411,403],{"class":91},[63,6413,6414],{"class":65,"line":277},[63,6415,626],{"class":91},[16,6417,6418],{},"Same trees, different aggregation.",[11,6420,6422],{"id":6421},"validate-everything","Validate everything",[16,6424,6425],{},"Before optimizing, verify generated inference against the original model. Use multiple test cases, not just one row.",[54,6427,6429],{"className":78,"code":6428,"language":80,"meta":59,"style":59},"Assert.True(\n    diff \u003C 1e-4,\n    $\"Row {i} mismatch. Expected={expected}, Actual={actual}\"\n);\n",[32,6430,6431,6443,6459,6493],{"__ignoreMap":59},[63,6432,6433,6436,6438,6441],{"class":65,"line":66},[63,6434,6435],{"class":87},"Assert",[63,6437,92],{"class":91},[63,6439,6440],{"class":95},"True",[63,6442,219],{"class":91},[63,6444,6445,6448,6450,6453,6455,6457],{"class":65,"line":115},[63,6446,6447],{"class":528},"    diff",[63,6449,6021],{"class":132},[63,6451,6452],{"class":289}," 1e",[63,6454,4932],{"class":132},[63,6456,771],{"class":289},[63,6458,233],{"class":91},[63,6460,6461,6464,6467,6469,6471,6474,6476,6479,6481,6484,6486,6489,6491],{"class":65,"line":121},[63,6462,6463],{"class":145},"    $\"Row ",[63,6465,5231],{"class":6466},"sMj0N",[63,6468,5610],{"class":528},[63,6470,5237],{"class":6466},[63,6472,6473],{"class":145}," mismatch. Expected=",[63,6475,5231],{"class":6466},[63,6477,6478],{"class":528},"expected",[63,6480,5237],{"class":6466},[63,6482,6483],{"class":145},", Actual=",[63,6485,5231],{"class":6466},[63,6487,6488],{"class":528},"actual",[63,6490,5237],{"class":6466},[63,6492,5263],{"class":145},[63,6494,6495],{"class":65,"line":152},[63,6496,149],{"class":91},[11,6498,6500],{"id":6499},"whats-in-a-lightgbm-dump-and-what-makes-it-into-c","What's in a LightGBM dump, and what makes it into C#",[16,6502,6503],{},"Here is one real LightGBM tree from a text dump. Arrays are shortened for readability, but the shape is the same:",[54,6505,6508],{"className":6506,"code":6507,"language":2529,"meta":59},[2527],"Tree=1\nnum_leaves=5\nsplit_feature=11 2 17 18\nsplit_gain=437375 65423.8 30686.3 15052.8\nthreshold=1.0000000180025095e-35 0.94499999284744274 15.500000000000002 0.47699999809265142\ndecision_type=2 2 2 2\nleft_child=1 -1 -2 -3\nright_child=2 3 -4 -5\nleaf_value=0.0017963732109250652 -0.0029226866697364827 0.012002031587390959 -0.0066260868638778623 0.0074389293400878229\nleaf_weight=419486 548345 401639 1175784 300693\nleaf_count=419486 548345 401639 1175784 300693\ninternal_value=3.11874e-06 0.00803391 -0.00396084 0.00239001\ninternal_weight=5.49575e+06 1.8162e+06 3.67954e+06 617181\ninternal_count=5495746 1816203 3679543 617181\nis_linear=0\nshrinkage=0.02\n",[32,6509,6507],{"__ignoreMap":59},[16,6511,6512,6513,508,6515,508,6517,508,6520,6523,6524,6527],{},"The dump contains more than the C# runtime needs. The generated code keeps the fields used to walk the tree and return a prediction: ",[32,6514,4674],{},[32,6516,4682],{},[32,6518,6519],{},"left_child",[32,6521,6522],{},"right_child",", and ",[32,6525,6526],{},"leaf_value",". A few other fields are useful to understand, but they do not show up in the final arrays.",[16,6529,6530,6533,6534,6536],{},[32,6531,6532],{},"shrinkage"," is the tree learning rate. In many LightGBM text dumps, the shrinkage has already been folded into ",[32,6535,6526],{},", so the C# predictor can just add the tree result directly:",[54,6538,6540],{"className":78,"code":6539,"language":80,"meta":59,"style":59},"score += Eval(Roots[i], f);\n",[32,6541,6542],{"__ignoreMap":59},[63,6543,6544,6547,6549,6551,6553,6555,6557,6559,6561,6563],{"class":65,"line":66},[63,6545,6546],{"class":528},"score",[63,6548,4479],{"class":4478},[63,6550,5796],{"class":95},[63,6552,142],{"class":91},[63,6554,6047],{"class":87},[63,6556,4353],{"class":91},[63,6558,5610],{"class":528},[63,6560,6054],{"class":91},[63,6562,4445],{"class":528},[63,6564,149],{"class":91},[16,6566,6567,6568,6570],{},"If shrinkage were not already folded into the leaves, the runtime would need to multiply each tree output by a per tree shrinkage value. For the dumped models I was working with, ignoring the explicit ",[32,6569,6532],{}," field was correct because the leaf values already contained it.",[16,6572,6573,6576,6577,6580],{},[32,6574,6575],{},"is_linear"," tells you whether the tree uses normal constant leaves or linear leaves. The exporter assumes ",[32,6578,6579],{},"is_linear=0",", where each leaf returns one scalar value. That matches the usual LightGBM tree:",[54,6582,6585],{"className":6583,"code":6584,"language":2529,"meta":59},[2527],"if feature \u003C= threshold:\n    return leaf_value\n",[32,6586,6584],{"__ignoreMap":59},[16,6588,4311,6589,6592],{},[32,6590,6591],{},"is_linear=1",", each leaf contains a small linear model instead of a single number. That needs a different inference engine. This exporter does not support that case.",[16,6594,6595,6598],{},[32,6596,6597],{},"internal_value"," is the value stored at an internal split node. You can think of it as the prediction at that point if the tree stopped there. It is useful for diagnostics, but inference does not return from internal nodes, so the C# code does not need it.",[16,6600,6601,6604],{},[32,6602,6603],{},"internal_weight"," is the weighted amount of training data that reached an internal node. LightGBM uses it while training for split decisions, regularization, and pruning. Once the tree is trained, inference only needs to know which branch to take.",[16,6606,6607,6610,6611,92],{},[32,6608,6609],{},"leaf_weight"," is similar, but for a leaf. It tells you how much weighted training data ended up in that leaf. It can be useful when inspecting the model, but prediction only needs ",[32,6612,6526],{},[16,6614,6615],{},"So the flat C# representation is intentionally small:",[54,6617,6620],{"className":6618,"code":6619,"language":2529,"meta":59},[2527],"Feature\nThreshold\nLeft\nRight\nValue\nIsLeaf\nRoots\n",[32,6621,6619],{"__ignoreMap":59},[16,6623,6624],{},"That is basically a tiny runtime for executing tree bytecode. The original LightGBM dump has training metadata too, but the generated predictor only carries what it needs to reproduce inference.",[11,6626,6628],{"id":6627},"the-complete-code","The complete code",[16,6630,6631,6632,92],{},"The full exporter is on GitHub: ",[20,6633,6636],{"href":6634,"rel":6635},"https:\u002F\u002Fgithub.com\u002Fhaiilong\u002Fexport_lgbm_universal_cs",[52],"haiilong\u002Fexport_lgbm_universal_cs",[2563,6638,6639],{},"html pre.shiki code .sLKXg, html code.shiki .sLKXg{--shiki-default:#A626A4;--shiki-dark:#C678DD}html pre.shiki code .s5ixo, html code.shiki .s5ixo{--shiki-default:#383A42;--shiki-dark:#ABB2BF}html pre.shiki code .s7GmK, html code.shiki .s7GmK{--shiki-default:#383A42;--shiki-dark:#E5C07B}html pre.shiki code .sAGMh, html code.shiki .sAGMh{--shiki-default:#986801;--shiki-dark:#D19A66}html pre.shiki code .sknuh, html code.shiki .sknuh{--shiki-default:#383A42;--shiki-dark:#56B6C2}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sAdtL, html code.shiki .sAdtL{--shiki-default:#4078F2;--shiki-dark:#61AFEF}html pre.shiki code .sz0mV, html code.shiki .sz0mV{--shiki-default:#383A42;--shiki-dark:#E06C75}html pre.shiki code .sblXP, html code.shiki .sblXP{--shiki-default:#383A42;--shiki-dark:#C678DD}html pre.shiki code .slOjB, html code.shiki .slOjB{--shiki-default:#383A42;--shiki-dark:#61AFEF}html pre.shiki code .sp7wS, html code.shiki .sp7wS{--shiki-default:#986801;--shiki-default-font-style:inherit;--shiki-dark:#E06C75;--shiki-dark-font-style:italic}html pre.shiki code .sDhpE, html code.shiki .sDhpE{--shiki-default:#50A14F;--shiki-dark:#98C379}html pre.shiki code .so_Uh, html code.shiki .so_Uh{--shiki-default:#986801;--shiki-default-font-style:inherit;--shiki-dark:#D19A66;--shiki-dark-font-style:italic}html pre.shiki code .s_Sar, html code.shiki .s_Sar{--shiki-default:#0184BC;--shiki-dark:#56B6C2}html pre.shiki code .sC09Y, html code.shiki .sC09Y{--shiki-default:#C18401;--shiki-dark:#E5C07B}html pre.shiki code .sMj0N, html code.shiki .sMj0N{--shiki-default:#50A14F;--shiki-dark:#ABB2BF}",{"title":59,"searchDepth":115,"depth":115,"links":6641},[6642,6643,6644,6645,6646,6647,6648,6649,6655,6656,6657,6658,6662,6663,6664],{"id":13,"depth":115,"text":14},{"id":4288,"depth":115,"text":4289},{"id":4402,"depth":115,"text":4403},{"id":4547,"depth":115,"text":4548},{"id":4723,"depth":115,"text":4724},{"id":4863,"depth":115,"text":4864},{"id":5059,"depth":115,"text":5060},{"id":5662,"depth":115,"text":5663,"children":6650},[6651,6652,6653,6654],{"id":5666,"depth":121,"text":5667},{"id":5673,"depth":121,"text":5674},{"id":5691,"depth":121,"text":5692},{"id":5698,"depth":121,"text":5699},{"id":5705,"depth":115,"text":5706},{"id":5758,"depth":115,"text":5759},{"id":6080,"depth":115,"text":6081},{"id":6107,"depth":115,"text":6108,"children":6659},[6660,6661],{"id":6111,"depth":121,"text":6112},{"id":6192,"depth":121,"text":6193},{"id":6421,"depth":115,"text":6422},{"id":6499,"depth":115,"text":6500},{"id":6627,"depth":115,"text":6628},"2026-04-28","Converting LightGBM models into native .NET inference",{},"\u002Fblog\u002Fbuilding-ml-inference-part-4",{"title":4217,"description":6666},"blog\u002Fbuilding-ml-inference-part-4",[2594],"R6mQdvNAEOnn4JOLS_Ong_EaXeyTDq8h1aWEXtRMX0I",{"id":6674,"title":6675,"body":6676,"book":2585,"date":6975,"description":6976,"extension":2588,"meta":6977,"navigation":587,"path":6978,"seo":6979,"stem":6980,"tags":6981,"__hash__":6982},"blog\u002Fblog\u002Ftwo-dotnet-claude-skills.md","Two Claude Code skills I wrote for .NET",{"type":8,"value":6677,"toc":6968},[6678,6680,6683,6705,6708,6711,6717,6720,6804,6807,6810,6816,6819,6822,6844,6855,6861,6901,6904,6907,6911,6914,6947,6954,6957,6959,6962,6965],[11,6679,14],{"id":13},[16,6681,6682],{},"I use Claude Code a lot for .NET work. The code it produces is fine by default, just not in my style. A few patterns kept coming up that I'd manually rewrite every time:",[1789,6684,6685,6688,6695],{},[173,6686,6687],{},"Services that should use primary constructors but don't.",[173,6689,6690,6691,6694],{},"DTOs and records without ",[32,6692,6693],{},"required"," + init-only properties.",[173,6696,6697,6698,6700,6701,6704],{},"Minimal API endpoints registered one by one in ",[32,6699,3750],{}," instead of being auto-scanned via an ",[32,6702,6703],{},"IEndpoint"," interface.",[16,6706,6707],{},"None of those are bugs. They're reasonable defaults, just not mine. So I wrote two skills to nudge Claude toward the way I'd actually write the code.",[11,6709,6710],{"id":6710},"dotnet-skills",[16,6712,6713],{},[20,6714,6715],{"href":6715,"rel":6716},"https:\u002F\u002Fgithub.com\u002Fhaiilong\u002Fdotnet-skills",[52],[16,6718,6719],{},"A collection of opinionated .NET 10 \u002F C# 14 conventions, packaged as Markdown skill files. Roughly the stuff that comes up on a normal day at work:",[1789,6721,6722,6733,6736,6743,6755,6765,6771,6778,6787,6792,6795,6801],{},[173,6723,6724,6725,6728,6729,6732],{},"C# coding standards (records with ",[32,6726,6727],{},"required { get; init; }",", primary constructors for services, ",[32,6730,6731],{},"sealed"," by default)",[173,6734,6735],{},"Type design (class vs record vs struct vs readonly record struct)",[173,6737,6738,6739,6742],{},"Value objects (",[32,6740,6741],{},"readonly record struct"," patterns)",[173,6744,6745,6746,508,6749,6751,6752,5359],{},"Concurrency (",[32,6747,6748],{},"TimeProvider",[32,6750,4008],{},", bounded ",[32,6753,6754],{},"Channel\u003CT>",[173,6756,6757,6758,6761,6762,5359],{},"Error handling (",[32,6759,6760],{},"Result\u003CT>",", RFC 9457 ",[32,6763,6764],{},"ProblemDetails",[173,6766,6767,6768,6770],{},"ASP.NET Core minimal APIs with auto-registered ",[32,6769,6703],{}," implementations",[173,6772,6773,6774,6777],{},"Dependency injection with ",[32,6775,6776],{},"extension(IServiceCollection)"," blocks",[173,6779,6780,6781,3246,6784],{},"Configuration with ",[32,6782,6783],{},"IOptions\u003CT>",[32,6785,6786],{},"ValidateOnStart",[173,6788,6789,6790],{},"Resilient HTTP clients via ",[32,6791,1369],{},[173,6793,6794],{},"Serialization (System.Text.Json source gen, MessagePack)",[173,6796,6797,6798],{},"Structured logging with ",[32,6799,6800],{},"[LoggerMessage]",[173,6802,6803],{},"Testing (xUnit, NSubstitute, FluentAssertions, TestContainers, Verify)",[16,6805,6806],{},"Very opinionated, and the opinions are good ones (in my opinion). The other reason: it doubles as onboarding material for the team. \"Read this folder\" is a faster answer than explaining the same conventions one PR at a time.",[11,6808,6809],{"id":6809},"dotnet-performance-skill",[16,6811,6812],{},[20,6813,6814],{"href":6814,"rel":6815},"https:\u002F\u002Fgithub.com\u002Fhaiilong\u002Fdotnet-performance-skill",[52],[16,6817,6818],{},"The second skill has a different goal: it runs through code that already exists and flags performance problems.",[16,6820,6821],{},"It walks a catalog of around 90 anti-patterns, grouped by severity:",[1789,6823,6824,6831,6834],{},[173,6825,6826,6827,6830],{},"High: thread pool starvation, sync over async (",[32,6828,6829],{},".Result","), N+1 queries, LOH allocation. The stuff that takes a service down as soon as real load shows up.",[173,6832,6833],{},"Medium: missing pooling, cancellation propagation, middleware ordering.",[173,6835,6836,6837,6839,6840,6843],{},"Low: micro-optimizations like ",[32,6838,6731],{},", SIMD, ",[32,6841,6842],{},"stackalloc",". Only worth touching inside a measured hot path.",[16,6845,6846,6847,6850,6851,6854],{},"Each entry has a name, a short paragraph on why it matters, and a ",[32,6848,6849],{},"\u002F\u002F BAD"," \u002F ",[32,6852,6853],{},"\u002F\u002F GOOD"," code pair you can copy from. When you ask the skill to apply a fix, it pulls from the catalog instead of inventing one, which keeps the output predictable run to run.",[16,6856,6857,6858,92],{},"The goal is to find places where the existing code is doing something dumb, not to lecture about architecture. The clearest example is multiple enumeration on ",[32,6859,6860],{},"IEnumerable",[54,6862,6864],{"className":78,"code":6863,"language":80,"meta":59,"style":59},"\u002F\u002F BAD: enumerates the IEnumerable twice (and the source might be a database query)\nif (items.Any()) return items.Count();\n",[32,6865,6866,6871],{"__ignoreMap":59},[63,6867,6868],{"class":65,"line":66},[63,6869,6870],{"class":2731},"\u002F\u002F BAD: enumerates the IEnumerable twice (and the source might be a database query)\n",[63,6872,6873,6875,6877,6880,6882,6885,6888,6891,6894,6896,6899],{"class":65,"line":115},[63,6874,4345],{"class":439},[63,6876,3366],{"class":91},[63,6878,6879],{"class":87},"items",[63,6881,92],{"class":91},[63,6883,6884],{"class":95},"Any",[63,6886,6887],{"class":91},"()) ",[63,6889,6890],{"class":439},"return",[63,6892,6893],{"class":87}," items",[63,6895,92],{"class":91},[63,6897,6898],{"class":95},"Count",[63,6900,403],{"class":91},[16,6902,6903],{},"One thing that's still unclear: it sometimes surfaces stuff that isn't really a performance issue at all. Style nits, design smells, that kind of thing. The catalog and the skill description are both pretty explicit about scope, so it's not obvious why this happens. For now, read the High tier closely and treat the rest as suggestions.",[16,6905,6906],{},"EF Core isn't covered either, because it's not something I've used in production, so the catalog doesn't include it. Fork and add your own if you want it.",[11,6908,6910],{"id":6909},"install","Install",[16,6912,6913],{},"Both are manual clone. No plugin marketplace entry, and I don't plan to publish one.",[54,6915,6919],{"className":6916,"code":6917,"language":6918,"meta":59,"style":59},"language-bash shiki shiki-themes one-light one-dark-pro","git clone https:\u002F\u002Fgithub.com\u002Fhaiilong\u002Fdotnet-skills ~\u002F.claude\u002Fskills\u002Fdotnet-skills\ngit clone https:\u002F\u002Fgithub.com\u002Fhaiilong\u002Fdotnet-performance-skill ~\u002F.claude\u002Fskills\u002Fdotnet-performance\n","bash",[32,6920,6921,6935],{"__ignoreMap":59},[63,6922,6923,6926,6929,6932],{"class":65,"line":66},[63,6924,6925],{"class":95},"git",[63,6927,6928],{"class":145}," clone",[63,6930,6931],{"class":145}," https:\u002F\u002Fgithub.com\u002Fhaiilong\u002Fdotnet-skills",[63,6933,6934],{"class":145}," ~\u002F.claude\u002Fskills\u002Fdotnet-skills\n",[63,6936,6937,6939,6941,6944],{"class":65,"line":115},[63,6938,6925],{"class":95},[63,6940,6928],{"class":145},[63,6942,6943],{"class":145}," https:\u002F\u002Fgithub.com\u002Fhaiilong\u002Fdotnet-performance-skill",[63,6945,6946],{"class":145}," ~\u002F.claude\u002Fskills\u002Fdotnet-performance\n",[16,6948,6949,6950,6953],{},"Restart Claude Code or ",[32,6951,6952],{},"\u002Freload"," and they show up.",[16,6955,6956],{},"I'm not publishing as a plugin because these aren't broad enough for a general .NET audience. No ORM (EF Core, Dapper, Marten), no Razor, no Blazor, no MVC, plus a handful of other slices of the ecosystem I just don't touch at work. They reflect my own habits in the corner of .NET I actually live in. If your taste happens to overlap, fine. If it doesn't, the skills will spend their time fighting your defaults instead of helping, which is worse than not installing them at all.",[11,6958,4106],{"id":4105},[16,6960,6961],{},"Honestly the main audience here is me and my team. Me, because I want Claude Code to produce .NET that already passes my own review. The team, because \"go read this folder\" turns out to be a faster onboarding answer than \"watch me review your first ten PRs and pick it up by osmosis\".",[16,6963,6964],{},"If you happen to share the taste, take a look. Otherwise fork it and rewrite the bits you disagree with. The files are short enough that this is realistic, not a project.",[2563,6966,6967],{},"html pre.shiki code .sW2Sy, html code.shiki .sW2Sy{--shiki-default:#A0A1A7;--shiki-default-font-style:italic;--shiki-dark:#7F848E;--shiki-dark-font-style:italic}html pre.shiki code .sLKXg, html code.shiki .sLKXg{--shiki-default:#A626A4;--shiki-dark:#C678DD}html pre.shiki code .s5ixo, html code.shiki .s5ixo{--shiki-default:#383A42;--shiki-dark:#ABB2BF}html pre.shiki code .s7GmK, html code.shiki .s7GmK{--shiki-default:#383A42;--shiki-dark:#E5C07B}html pre.shiki code .sAdtL, html code.shiki .sAdtL{--shiki-default:#4078F2;--shiki-dark:#61AFEF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sDhpE, html code.shiki .sDhpE{--shiki-default:#50A14F;--shiki-dark:#98C379}",{"title":59,"searchDepth":115,"depth":115,"links":6969},[6970,6971,6972,6973,6974],{"id":13,"depth":115,"text":14},{"id":6710,"depth":115,"text":6710},{"id":6809,"depth":115,"text":6809},{"id":6909,"depth":115,"text":6910},{"id":4105,"depth":115,"text":4106},"2026-04-22","A pair of opinionated .NET skills for Claude Code, one for coding conventions, one for performance review.",{},"\u002Fblog\u002Ftwo-dotnet-claude-skills",{"title":6675,"description":6976},"blog\u002Ftwo-dotnet-claude-skills",[2594],"fLktajBLLWzDSUexby1GQwpzY-A1sEKpRjmahBoJRec",{"id":6984,"title":6985,"body":6986,"book":2585,"date":8721,"description":8722,"extension":2588,"meta":8723,"navigation":587,"path":8724,"seo":8725,"stem":8726,"tags":8727,"__hash__":8728},"blog\u002Fblog\u002Fjwt-with-auto-refresh-in-cookies.md","JWT with auto-refresh in cookies",{"type":8,"value":6987,"toc":8708},[6988,6990,6993,7004,7010,7016,7023,7027,7034,7048,7051,7054,7058,7061,7075,7078,7288,7291,7311,7327,7334,7338,7348,7354,7527,7536,7539,7543,7546,7553,7807,7810,7832,7835,7839,7842,7845,8331,8334,8373,8377,8380,8420,8426,8447,8453,8456,8460,8487,8493,8496,8535,8559,8565,8569,8572,8610,8616,8620,8626,8695,8697,8700,8705],[11,6989,14],{"id":13},[16,6991,6992],{},"JWT tutorials usually stop at \"here's an access token, put it in the Authorization header\". That's enough to get an API rejecting unauthenticated requests, but it's not enough for a real app. A production setup needs to deal with:",[1789,6994,6995,6998,7001],{},[173,6996,6997],{},"Where the token lives in the browser",[173,6999,7000],{},"What happens when the access token expires",[173,7002,7003],{},"How the client refreshes without forcing the user to log back in",[16,7005,7006,7007,7009],{},"I built a small demo that covers all three in the shape you'd actually use in production. It's a single ",[32,7008,3750],{}," of about 240 lines.",[16,7011,7012],{},[20,7013,7014],{"href":7014,"rel":7015},"https:\u002F\u002Fgithub.com\u002Fhaiilong\u002Fjwt-auth-demo",[52],[16,7017,7018,7019,7022],{},"A note on shape before we start: this isn't trying to be a \"minimum to demonstrate the concept\" sample. The patterns in it (cookie config, the ",[32,7020,7021],{},"OnMessageReceived"," hook, the auto-refresh middleware, the order in the pipeline) are the same patterns I'd use in a real ASP.NET Core API. The handful of toy parts (in-memory user store, hardcoded signing key, in-memory refresh-token dictionary) are clearly marked as toy and easy to swap. I'll list the production swaps near the end of the post.",[11,7024,7026],{"id":7025},"two-tokens-not-one","Two tokens, not one",[16,7028,7029,7030,7033],{},"The first thing the demo does is hand out ",[3240,7031,7032],{},"two"," tokens on login:",[1789,7035,7036,7042],{},[173,7037,7038,7041],{},[2055,7039,7040],{},"Access token",", a JWT signed with HS256, very short lived (1 minute in the demo so the refresh is easy to test). Carried on every request to prove who you are.",[173,7043,7044,7047],{},[2055,7045,7046],{},"Refresh token",", a cryptographically random 32 bytes, longer lived (7 days). Used only to ask for a new access token.",[16,7049,7050],{},"The point of the split is blast radius. If the access token leaks (someone reads your network traffic, an extension scrapes it, the client gets compromised), the damage window is one minute. The refresh token is the long-lived secret and you keep it locked down because it only goes back to the server when you want to refresh.",[16,7052,7053],{},"For the demo I made the access token 1 minute long so you don't have to wait around to see the refresh fire. Production values are usually 5 to 15 minutes for the access token and 7 to 30 days for the refresh token.",[11,7055,7057],{"id":7056},"where-the-tokens-live","Where the tokens live",[16,7059,7060],{},"Two common options:",[170,7062,7063,7069],{},[173,7064,7065,7068],{},[32,7066,7067],{},"localStorage"," in the browser. Easy to read from your JS code. Vulnerable to XSS: any script on your origin can read it.",[173,7070,7071,7072,6359],{},"HttpOnly cookies. Cannot be read from JS at all. Set by the server, sent automatically by the browser. Vulnerable to CSRF unless you mitigate (which is why the cookies in the demo are ",[32,7073,7074],{},"SameSite=Strict",[16,7076,7077],{},"The demo uses option 2. The login endpoint sets three cookies:",[54,7079,7081],{"className":78,"code":7080,"language":80,"meta":59,"style":59},"http.Response.Cookies.Append(\"X-Access-Token\", accessToken,\n    new CookieOptions\n    {\n        HttpOnly = true,\n        Secure = true,\n        SameSite = SameSiteMode.Strict,\n        Expires = DateTime.UtcNow.AddMinutes(jwtSettings.AccessTokenMinutes)\n    });\n\nhttp.Response.Cookies.Append(\"X-Refresh-Token\", refreshToken,\n    new CookieOptions { \u002F* same options, 7 day expiry *\u002F });\n\nhttp.Response.Cookies.Append(\"X-Username\", user.Username, cookieOpts);\n",[32,7082,7083,7114,7121,7125,7136,7147,7164,7196,7200,7204,7232,7246,7250],{"__ignoreMap":59},[63,7084,7085,7087,7089,7092,7094,7097,7099,7102,7104,7107,7109,7112],{"class":65,"line":66},[63,7086,1742],{"class":87},[63,7088,92],{"class":91},[63,7090,7091],{"class":87},"Response",[63,7093,92],{"class":91},[63,7095,7096],{"class":87},"Cookies",[63,7098,92],{"class":91},[63,7100,7101],{"class":95},"Append",[63,7103,142],{"class":91},[63,7105,7106],{"class":145},"\"X-Access-Token\"",[63,7108,508],{"class":91},[63,7110,7111],{"class":528},"accessToken",[63,7113,233],{"class":91},[63,7115,7116,7118],{"class":65,"line":115},[63,7117,2091],{"class":91},[63,7119,7120],{"class":102},"CookieOptions\n",[63,7122,7123],{"class":65,"line":121},[63,7124,250],{"class":91},[63,7126,7127,7130,7132,7134],{"class":65,"line":152},[63,7128,7129],{"class":528},"        HttpOnly",[63,7131,133],{"class":132},[63,7133,1515],{"class":289},[63,7135,233],{"class":91},[63,7137,7138,7141,7143,7145],{"class":65,"line":253},[63,7139,7140],{"class":528},"        Secure",[63,7142,133],{"class":132},[63,7144,1515],{"class":289},[63,7146,233],{"class":91},[63,7148,7149,7152,7154,7157,7159,7162],{"class":65,"line":277},[63,7150,7151],{"class":528},"        SameSite",[63,7153,133],{"class":132},[63,7155,7156],{"class":87}," SameSiteMode",[63,7158,92],{"class":91},[63,7160,7161],{"class":87},"Strict",[63,7163,233],{"class":91},[63,7165,7166,7169,7171,7174,7176,7179,7181,7184,7186,7189,7191,7194],{"class":65,"line":295},[63,7167,7168],{"class":528},"        Expires",[63,7170,133],{"class":132},[63,7172,7173],{"class":87}," DateTime",[63,7175,92],{"class":91},[63,7177,7178],{"class":87},"UtcNow",[63,7180,92],{"class":91},[63,7182,7183],{"class":95},"AddMinutes",[63,7185,142],{"class":91},[63,7187,7188],{"class":87},"jwtSettings",[63,7190,92],{"class":91},[63,7192,7193],{"class":87},"AccessTokenMinutes",[63,7195,474],{"class":91},[63,7197,7198],{"class":65,"line":301},[63,7199,409],{"class":91},[63,7201,7202],{"class":65,"line":313},[63,7203,588],{"emptyLinePlaceholder":587},[63,7205,7206,7208,7210,7212,7214,7216,7218,7220,7222,7225,7227,7230],{"class":65,"line":318},[63,7207,1742],{"class":87},[63,7209,92],{"class":91},[63,7211,7091],{"class":87},[63,7213,92],{"class":91},[63,7215,7096],{"class":87},[63,7217,92],{"class":91},[63,7219,7101],{"class":95},[63,7221,142],{"class":91},[63,7223,7224],{"class":145},"\"X-Refresh-Token\"",[63,7226,508],{"class":91},[63,7228,7229],{"class":528},"refreshToken",[63,7231,233],{"class":91},[63,7233,7234,7236,7239,7241,7244],{"class":65,"line":340},[63,7235,2091],{"class":91},[63,7237,7238],{"class":102},"CookieOptions",[63,7240,3486],{"class":91},[63,7242,7243],{"class":2731},"\u002F* same options, 7 day expiry *\u002F",[63,7245,582],{"class":91},[63,7247,7248],{"class":65,"line":369},[63,7249,588],{"emptyLinePlaceholder":587},[63,7251,7252,7254,7256,7258,7260,7262,7264,7266,7268,7271,7273,7276,7278,7281,7283,7286],{"class":65,"line":374},[63,7253,1742],{"class":87},[63,7255,92],{"class":91},[63,7257,7091],{"class":87},[63,7259,92],{"class":91},[63,7261,7096],{"class":87},[63,7263,92],{"class":91},[63,7265,7101],{"class":95},[63,7267,142],{"class":91},[63,7269,7270],{"class":145},"\"X-Username\"",[63,7272,508],{"class":91},[63,7274,7275],{"class":87},"user",[63,7277,92],{"class":91},[63,7279,7280],{"class":87},"Username",[63,7282,508],{"class":91},[63,7284,7285],{"class":528},"cookieOpts",[63,7287,149],{"class":91},[16,7289,7290],{},"The flags do real work:",[1789,7292,7293,7299,7305],{},[173,7294,7295,7298],{},[32,7296,7297],{},"HttpOnly = true",": JS in the browser cannot read or write this cookie. Closes the door on XSS-based token theft.",[173,7300,7301,7304],{},[32,7302,7303],{},"Secure = true",": cookie is only sent over HTTPS. In production this is non-negotiable.",[173,7306,7307,7310],{},[32,7308,7309],{},"SameSite = SameSiteMode.Strict",": cookie is not sent on cross-site requests at all. Closes the door on most CSRF attack patterns.",[16,7312,7313,7314,7317,7318,3246,7321,7324,7325,92],{},"After hitting ",[32,7315,7316],{},"\u002Flogin",", open the browser's DevTools, Application tab, Cookies, and the localhost entry. The three cookies should be there with ",[32,7319,7320],{},"HttpOnly",[32,7322,7323],{},"Secure"," both checked and ",[32,7326,7074],{},[16,7328,7329],{},[7330,7331],"img",{"alt":7332,"src":7333},"Cookies in DevTools after login","\u002Fblog\u002Fjwt-with-auto-refresh-in-cookies\u002Fcookies-after-login.png",[11,7335,7337],{"id":7336},"reading-the-jwt-from-a-cookie","Reading the JWT from a cookie",[16,7339,7340,7343,7344,7347],{},[32,7341,7342],{},"AddJwtBearer"," defaults to looking for the token in the ",[32,7345,7346],{},"Authorization: Bearer \u003Ctoken>"," header. If your token lives in a cookie, the default doesn't help.",[16,7349,7350,7351,429],{},"The hook is ",[32,7352,7353],{},"JwtBearerEvents.OnMessageReceived",[54,7355,7357],{"className":78,"code":7356,"language":80,"meta":59,"style":59},".AddJwtBearer(options =>\n{\n    options.TokenValidationParameters = new TokenValidationParameters { ... };\n\n    options.Events = new JwtBearerEvents\n    {\n        OnMessageReceived = context =>\n        {\n            if (context.Request.Cookies.TryGetValue(\"X-Access-Token\", out var token))\n            {\n                context.Token = token;\n            }\n            return Task.CompletedTask;\n        }\n    };\n});\n",[32,7358,7359,7372,7376,7395,7399,7415,7419,7431,7435,7475,7480,7495,7500,7513,7518,7523],{"__ignoreMap":59},[63,7360,7361,7363,7365,7367,7370],{"class":65,"line":66},[63,7362,92],{"class":91},[63,7364,7342],{"class":95},[63,7366,142],{"class":91},[63,7368,7369],{"class":87},"options",[63,7371,112],{"class":91},[63,7373,7374],{"class":65,"line":115},[63,7375,118],{"class":91},[63,7377,7378,7381,7383,7386,7388,7390,7392],{"class":65,"line":121},[63,7379,7380],{"class":87},"    options",[63,7382,92],{"class":91},[63,7384,7385],{"class":87},"TokenValidationParameters",[63,7387,133],{"class":132},[63,7389,136],{"class":91},[63,7391,7385],{"class":102},[63,7393,7394],{"class":91}," { ... };\n",[63,7396,7397],{"class":65,"line":152},[63,7398,588],{"emptyLinePlaceholder":587},[63,7400,7401,7403,7405,7408,7410,7412],{"class":65,"line":253},[63,7402,7380],{"class":87},[63,7404,92],{"class":91},[63,7406,7407],{"class":87},"Events",[63,7409,133],{"class":132},[63,7411,136],{"class":91},[63,7413,7414],{"class":102},"JwtBearerEvents\n",[63,7416,7417],{"class":65,"line":277},[63,7418,250],{"class":91},[63,7420,7421,7424,7426,7429],{"class":65,"line":295},[63,7422,7423],{"class":528},"        OnMessageReceived",[63,7425,133],{"class":132},[63,7427,7428],{"class":87}," context",[63,7430,112],{"class":91},[63,7432,7433],{"class":65,"line":301},[63,7434,1953],{"class":91},[63,7436,7437,7440,7442,7445,7447,7450,7452,7454,7456,7459,7461,7463,7465,7468,7470,7473],{"class":65,"line":313},[63,7438,7439],{"class":439},"            if",[63,7441,3366],{"class":91},[63,7443,7444],{"class":87},"context",[63,7446,92],{"class":91},[63,7448,7449],{"class":87},"Request",[63,7451,92],{"class":91},[63,7453,7096],{"class":87},[63,7455,92],{"class":91},[63,7457,7458],{"class":95},"TryGetValue",[63,7460,142],{"class":91},[63,7462,7106],{"class":145},[63,7464,508],{"class":91},[63,7466,7467],{"class":439},"out",[63,7469,1690],{"class":439},[63,7471,7472],{"class":528}," token",[63,7474,1106],{"class":91},[63,7476,7477],{"class":65,"line":318},[63,7478,7479],{"class":91},"            {\n",[63,7481,7482,7485,7487,7489,7491,7493],{"class":65,"line":340},[63,7483,7484],{"class":87},"                context",[63,7486,92],{"class":91},[63,7488,1765],{"class":87},[63,7490,133],{"class":132},[63,7492,7472],{"class":528},[63,7494,274],{"class":91},[63,7496,7497],{"class":65,"line":369},[63,7498,7499],{"class":91},"            }\n",[63,7501,7502,7504,7506,7508,7511],{"class":65,"line":374},[63,7503,4817],{"class":439},[63,7505,486],{"class":87},[63,7507,92],{"class":91},[63,7509,7510],{"class":87},"CompletedTask",[63,7512,274],{"class":91},[63,7514,7515],{"class":65,"line":387},[63,7516,7517],{"class":91},"        }\n",[63,7519,7520],{"class":65,"line":392},[63,7521,7522],{"class":91},"    };\n",[63,7524,7525],{"class":65,"line":406},[63,7526,155],{"class":91},[16,7528,7529,7531,7532,7535],{},[32,7530,7021],{}," runs at the start of authentication on every request. Set ",[32,7533,7534],{},"context.Token",", and the rest of the bearer pipeline (signature validation, claims extraction, lifetime check) just uses that value as if it had come from the Authorization header. You don't have to touch anything else in ASP.NET Core's auth machinery.",[16,7537,7538],{},"This is the right hook for any \"the JWT lives somewhere other than the standard header\" requirement: cookie, custom header, a query string for SSE or WebSocket connections, anywhere.",[11,7540,7542],{"id":7541},"the-naive-refresh-flow","The naive refresh flow",[16,7544,7545],{},"Before getting to the auto-refresh, here's what the manual flow looks like.",[16,7547,7548,7549,7552],{},"The demo exposes a ",[32,7550,7551],{},"\u002Frefresh"," endpoint:",[54,7554,7556],{"className":78,"code":7555,"language":80,"meta":59,"style":59},"app.MapPost(\"\u002Frefresh\", (HttpContext http) =>\n{\n    http.Request.Cookies.TryGetValue(\"X-Refresh-Token\", out var cookieRefreshToken);\n    http.Request.Cookies.TryGetValue(\"X-Username\", out var cookieUsername);\n\n    if (string.IsNullOrEmpty(cookieRefreshToken) || string.IsNullOrEmpty(cookieUsername))\n        return Results.Unauthorized();\n\n    if (!userRefreshTokenDict.TryGetValue(cookieUsername, out var storedRefreshToken)\n        || storedRefreshToken != cookieRefreshToken)\n        return Results.Forbid();\n\n    \u002F\u002F generate new tokens, set new cookies, rotate the server-side refresh-token entry\n    \u002F\u002F ...\n    return Results.Ok(new { message = \"Refreshed via Cookies\" });\n});\n",[32,7557,7558,7583,7587,7619,7650,7654,7690,7704,7708,7738,7752,7765,7769,7774,7779,7803],{"__ignoreMap":59},[63,7559,7560,7563,7565,7568,7570,7573,7576,7579,7581],{"class":65,"line":66},[63,7561,7562],{"class":87},"app",[63,7564,92],{"class":91},[63,7566,7567],{"class":95},"MapPost",[63,7569,142],{"class":91},[63,7571,7572],{"class":145},"\"\u002Frefresh\"",[63,7574,7575],{"class":91},", (",[63,7577,7578],{"class":102},"HttpContext",[63,7580,471],{"class":87},[63,7582,1910],{"class":91},[63,7584,7585],{"class":65,"line":115},[63,7586,118],{"class":91},[63,7588,7589,7592,7594,7596,7598,7600,7602,7604,7606,7608,7610,7612,7614,7617],{"class":65,"line":121},[63,7590,7591],{"class":87},"    http",[63,7593,92],{"class":91},[63,7595,7449],{"class":87},[63,7597,92],{"class":91},[63,7599,7096],{"class":87},[63,7601,92],{"class":91},[63,7603,7458],{"class":95},[63,7605,142],{"class":91},[63,7607,7224],{"class":145},[63,7609,508],{"class":91},[63,7611,7467],{"class":439},[63,7613,1690],{"class":439},[63,7615,7616],{"class":528}," cookieRefreshToken",[63,7618,149],{"class":91},[63,7620,7621,7623,7625,7627,7629,7631,7633,7635,7637,7639,7641,7643,7645,7648],{"class":65,"line":152},[63,7622,7591],{"class":87},[63,7624,92],{"class":91},[63,7626,7449],{"class":87},[63,7628,92],{"class":91},[63,7630,7096],{"class":87},[63,7632,92],{"class":91},[63,7634,7458],{"class":95},[63,7636,142],{"class":91},[63,7638,7270],{"class":145},[63,7640,508],{"class":91},[63,7642,7467],{"class":439},[63,7644,1690],{"class":439},[63,7646,7647],{"class":528}," cookieUsername",[63,7649,149],{"class":91},[63,7651,7652],{"class":65,"line":253},[63,7653,588],{"emptyLinePlaceholder":587},[63,7655,7656,7658,7660,7662,7664,7667,7669,7672,7674,7677,7679,7681,7683,7685,7688],{"class":65,"line":277},[63,7657,3410],{"class":439},[63,7659,3366],{"class":91},[63,7661,502],{"class":439},[63,7663,92],{"class":91},[63,7665,7666],{"class":95},"IsNullOrEmpty",[63,7668,142],{"class":91},[63,7670,7671],{"class":528},"cookieRefreshToken",[63,7673,3420],{"class":91},[63,7675,7676],{"class":132},"||",[63,7678,2766],{"class":439},[63,7680,92],{"class":91},[63,7682,7666],{"class":95},[63,7684,142],{"class":91},[63,7686,7687],{"class":528},"cookieUsername",[63,7689,1106],{"class":91},[63,7691,7692,7694,7697,7699,7702],{"class":65,"line":295},[63,7693,593],{"class":439},[63,7695,7696],{"class":87}," Results",[63,7698,92],{"class":91},[63,7700,7701],{"class":95},"Unauthorized",[63,7703,403],{"class":91},[63,7705,7706],{"class":65,"line":301},[63,7707,588],{"emptyLinePlaceholder":587},[63,7709,7710,7712,7714,7716,7719,7721,7723,7725,7727,7729,7731,7733,7736],{"class":65,"line":313},[63,7711,3410],{"class":439},[63,7713,3366],{"class":91},[63,7715,3369],{"class":132},[63,7717,7718],{"class":87},"userRefreshTokenDict",[63,7720,92],{"class":91},[63,7722,7458],{"class":95},[63,7724,142],{"class":91},[63,7726,7687],{"class":528},[63,7728,508],{"class":91},[63,7730,7467],{"class":439},[63,7732,1690],{"class":439},[63,7734,7735],{"class":528}," storedRefreshToken",[63,7737,474],{"class":91},[63,7739,7740,7743,7745,7748,7750],{"class":65,"line":318},[63,7741,7742],{"class":132},"        ||",[63,7744,7735],{"class":528},[63,7746,7747],{"class":132}," !=",[63,7749,7616],{"class":528},[63,7751,474],{"class":91},[63,7753,7754,7756,7758,7760,7763],{"class":65,"line":340},[63,7755,593],{"class":439},[63,7757,7696],{"class":87},[63,7759,92],{"class":91},[63,7761,7762],{"class":95},"Forbid",[63,7764,403],{"class":91},[63,7766,7767],{"class":65,"line":369},[63,7768,588],{"emptyLinePlaceholder":587},[63,7770,7771],{"class":65,"line":374},[63,7772,7773],{"class":2731},"    \u002F\u002F generate new tokens, set new cookies, rotate the server-side refresh-token entry\n",[63,7775,7776],{"class":65,"line":387},[63,7777,7778],{"class":2731},"    \u002F\u002F ...\n",[63,7780,7781,7783,7785,7787,7790,7793,7796,7798,7801],{"class":65,"line":392},[63,7782,1890],{"class":439},[63,7784,7696],{"class":87},[63,7786,92],{"class":91},[63,7788,7789],{"class":95},"Ok",[63,7791,7792],{"class":91},"(new { ",[63,7794,7795],{"class":528},"message",[63,7797,133],{"class":132},[63,7799,7800],{"class":145}," \"Refreshed via Cookies\"",[63,7802,582],{"class":91},[63,7804,7805],{"class":65,"line":406},[63,7806,155],{"class":91},[16,7808,7809],{},"In a manual flow, the client side looks like:",[170,7811,7812,7818,7821,7827],{},[173,7813,7814,7815,92],{},"Client requests ",[32,7816,7817],{},"\u002Fdashboard",[173,7819,7820],{},"Server returns 401 because the access token expired.",[173,7822,7823,7824,7826],{},"Client sees the 401, calls ",[32,7825,7551],{}," to get new tokens.",[173,7828,7829,7830,92],{},"Client retries ",[32,7831,7817],{},[16,7833,7834],{},"This works. It's three round trips for every protected request that catches an expired token. Every page open in the user's browser at the moment the token expires has to do its own version of this dance.",[11,7836,7838],{"id":7837},"auto-refresh-in-middleware","Auto-refresh, in middleware",[16,7840,7841],{},"The better version: the server handles the refresh transparently in the middleware pipeline, before authentication runs. The client only ever sees a successful response. The 401 dance disappears.",[16,7843,7844],{},"Here's the middleware:",[54,7846,7848],{"className":78,"code":7847,"language":80,"meta":59,"style":59},"app.Use(async (context, next) =>\n{\n    var accessToken = context.Request.Cookies[\"X-Access-Token\"];\n\n    if (IsTokenExpired(accessToken)\n        && context.Request.Cookies.TryGetValue(\"X-Refresh-Token\", out var refreshToken)\n        && context.Request.Cookies.TryGetValue(\"X-Username\", out var username))\n    {\n        if (userRefreshTokenDict.TryGetValue(username!, out var storedRefreshToken)\n            && storedRefreshToken == refreshToken)\n        {\n            var user = users.FirstOrDefault(u => u.Username == username);\n            if (user != null)\n            {\n                \u002F\u002F 1. Generate new tokens\n                var newAccessToken = GenerateAccessToken(...);\n                var newRefreshToken = GenerateRefreshToken();\n\n                \u002F\u002F 2. Update the server-side refresh-token store\n                userRefreshTokenDict[username!] = newRefreshToken;\n\n                \u002F\u002F 3. Write the new tokens to the response as fresh cookies\n                context.Response.Cookies.Append(\"X-Access-Token\", newAccessToken, ...);\n                context.Response.Cookies.Append(\"X-Refresh-Token\", newRefreshToken, ...);\n\n                \u002F\u002F 4. IMPORTANT: inject the new token into the CURRENT REQUEST's headers\n                \u002F\u002F so UseAuthentication sees a valid token and lets this request through\n                context.Request.Headers.Append(\"Authorization\", \"Bearer \" + newAccessToken);\n            }\n        }\n    }\n\n    await next();\n});\n\napp.UseAuthentication();\napp.UseAuthorization();\n",[32,7849,7850,7875,7879,7904,7908,7923,7957,7990,7994,8023,8037,8041,8078,8092,8096,8101,8117,8131,8135,8140,8159,8163,8168,8196,8223,8227,8232,8237,8269,8273,8278,8283,8288,8297,8302,8307,8319],{"__ignoreMap":59},[63,7851,7852,7854,7856,7859,7861,7864,7866,7868,7870,7873],{"class":65,"line":66},[63,7853,7562],{"class":87},[63,7855,92],{"class":91},[63,7857,7858],{"class":95},"Use",[63,7860,142],{"class":91},[63,7862,7863],{"class":439},"async",[63,7865,3366],{"class":91},[63,7867,7444],{"class":87},[63,7869,508],{"class":91},[63,7871,7872],{"class":87},"next",[63,7874,1910],{"class":91},[63,7876,7877],{"class":65,"line":115},[63,7878,118],{"class":91},[63,7880,7881,7883,7886,7888,7890,7892,7894,7896,7898,7900,7902],{"class":65,"line":121},[63,7882,6335],{"class":439},[63,7884,7885],{"class":528}," accessToken",[63,7887,133],{"class":132},[63,7889,7428],{"class":87},[63,7891,92],{"class":91},[63,7893,7449],{"class":87},[63,7895,92],{"class":91},[63,7897,7096],{"class":87},[63,7899,4353],{"class":91},[63,7901,7106],{"class":145},[63,7903,5867],{"class":91},[63,7905,7906],{"class":65,"line":152},[63,7907,588],{"emptyLinePlaceholder":587},[63,7909,7910,7912,7914,7917,7919,7921],{"class":65,"line":253},[63,7911,3410],{"class":439},[63,7913,3366],{"class":91},[63,7915,7916],{"class":95},"IsTokenExpired",[63,7918,142],{"class":91},[63,7920,7111],{"class":528},[63,7922,474],{"class":91},[63,7924,7925,7928,7930,7932,7934,7936,7938,7940,7942,7944,7946,7948,7950,7952,7955],{"class":65,"line":277},[63,7926,7927],{"class":132},"        &&",[63,7929,7428],{"class":87},[63,7931,92],{"class":91},[63,7933,7449],{"class":87},[63,7935,92],{"class":91},[63,7937,7096],{"class":87},[63,7939,92],{"class":91},[63,7941,7458],{"class":95},[63,7943,142],{"class":91},[63,7945,7224],{"class":145},[63,7947,508],{"class":91},[63,7949,7467],{"class":439},[63,7951,1690],{"class":439},[63,7953,7954],{"class":528}," refreshToken",[63,7956,474],{"class":91},[63,7958,7959,7961,7963,7965,7967,7969,7971,7973,7975,7977,7979,7981,7983,7985,7988],{"class":65,"line":295},[63,7960,7927],{"class":132},[63,7962,7428],{"class":87},[63,7964,92],{"class":91},[63,7966,7449],{"class":87},[63,7968,92],{"class":91},[63,7970,7096],{"class":87},[63,7972,92],{"class":91},[63,7974,7458],{"class":95},[63,7976,142],{"class":91},[63,7978,7270],{"class":145},[63,7980,508],{"class":91},[63,7982,7467],{"class":439},[63,7984,1690],{"class":439},[63,7986,7987],{"class":528}," username",[63,7989,1106],{"class":91},[63,7991,7992],{"class":65,"line":301},[63,7993,250],{"class":91},[63,7995,7996,7998,8000,8002,8004,8006,8008,8011,8013,8015,8017,8019,8021],{"class":65,"line":313},[63,7997,4795],{"class":439},[63,7999,3366],{"class":91},[63,8001,7718],{"class":87},[63,8003,92],{"class":91},[63,8005,7458],{"class":95},[63,8007,142],{"class":91},[63,8009,8010],{"class":528},"username",[63,8012,3369],{"class":132},[63,8014,508],{"class":91},[63,8016,7467],{"class":439},[63,8018,1690],{"class":439},[63,8020,7735],{"class":528},[63,8022,474],{"class":91},[63,8024,8025,8028,8030,8033,8035],{"class":65,"line":318},[63,8026,8027],{"class":132},"            &&",[63,8029,7735],{"class":528},[63,8031,8032],{"class":132}," ==",[63,8034,7954],{"class":528},[63,8036,474],{"class":91},[63,8038,8039],{"class":65,"line":340},[63,8040,1953],{"class":91},[63,8042,8043,8046,8049,8051,8054,8056,8059,8061,8064,8066,8068,8070,8072,8074,8076],{"class":65,"line":369},[63,8044,8045],{"class":439},"            var",[63,8047,8048],{"class":528}," user",[63,8050,133],{"class":132},[63,8052,8053],{"class":87}," users",[63,8055,92],{"class":91},[63,8057,8058],{"class":95},"FirstOrDefault",[63,8060,142],{"class":91},[63,8062,8063],{"class":87},"u",[63,8065,784],{"class":91},[63,8067,8063],{"class":87},[63,8069,92],{"class":91},[63,8071,7280],{"class":87},[63,8073,8032],{"class":132},[63,8075,7987],{"class":528},[63,8077,149],{"class":91},[63,8079,8080,8082,8084,8086,8088,8090],{"class":65,"line":374},[63,8081,7439],{"class":439},[63,8083,3366],{"class":91},[63,8085,7275],{"class":528},[63,8087,7747],{"class":132},[63,8089,3607],{"class":289},[63,8091,474],{"class":91},[63,8093,8094],{"class":65,"line":387},[63,8095,7479],{"class":91},[63,8097,8098],{"class":65,"line":392},[63,8099,8100],{"class":2731},"                \u002F\u002F 1. Generate new tokens\n",[63,8102,8103,8106,8109,8111,8114],{"class":65,"line":406},[63,8104,8105],{"class":439},"                var",[63,8107,8108],{"class":528}," newAccessToken",[63,8110,133],{"class":132},[63,8112,8113],{"class":95}," GenerateAccessToken",[63,8115,8116],{"class":91},"(...);\n",[63,8118,8119,8121,8124,8126,8129],{"class":65,"line":2931},[63,8120,8105],{"class":439},[63,8122,8123],{"class":528}," newRefreshToken",[63,8125,133],{"class":132},[63,8127,8128],{"class":95}," GenerateRefreshToken",[63,8130,403],{"class":91},[63,8132,8133],{"class":65,"line":2937},[63,8134,588],{"emptyLinePlaceholder":587},[63,8136,8137],{"class":65,"line":2956},[63,8138,8139],{"class":2731},"                \u002F\u002F 2. Update the server-side refresh-token store\n",[63,8141,8142,8145,8147,8149,8151,8153,8155,8157],{"class":65,"line":2961},[63,8143,8144],{"class":87},"                userRefreshTokenDict",[63,8146,4353],{"class":91},[63,8148,8010],{"class":528},[63,8150,3369],{"class":132},[63,8152,574],{"class":91},[63,8154,577],{"class":132},[63,8156,8123],{"class":528},[63,8158,274],{"class":91},[63,8160,8161],{"class":65,"line":3000},[63,8162,588],{"emptyLinePlaceholder":587},[63,8164,8165],{"class":65,"line":3018},[63,8166,8167],{"class":2731},"                \u002F\u002F 3. Write the new tokens to the response as fresh cookies\n",[63,8169,8170,8172,8174,8176,8178,8180,8182,8184,8186,8188,8190,8193],{"class":65,"line":3037},[63,8171,7484],{"class":87},[63,8173,92],{"class":91},[63,8175,7091],{"class":87},[63,8177,92],{"class":91},[63,8179,7096],{"class":87},[63,8181,92],{"class":91},[63,8183,7101],{"class":95},[63,8185,142],{"class":91},[63,8187,7106],{"class":145},[63,8189,508],{"class":91},[63,8191,8192],{"class":528},"newAccessToken",[63,8194,8195],{"class":91},", ...);\n",[63,8197,8198,8200,8202,8204,8206,8208,8210,8212,8214,8216,8218,8221],{"class":65,"line":3056},[63,8199,7484],{"class":87},[63,8201,92],{"class":91},[63,8203,7091],{"class":87},[63,8205,92],{"class":91},[63,8207,7096],{"class":87},[63,8209,92],{"class":91},[63,8211,7101],{"class":95},[63,8213,142],{"class":91},[63,8215,7224],{"class":145},[63,8217,508],{"class":91},[63,8219,8220],{"class":528},"newRefreshToken",[63,8222,8195],{"class":91},[63,8224,8225],{"class":65,"line":5491},[63,8226,588],{"emptyLinePlaceholder":587},[63,8228,8229],{"class":65,"line":5515},[63,8230,8231],{"class":2731},"                \u002F\u002F 4. IMPORTANT: inject the new token into the CURRENT REQUEST's headers\n",[63,8233,8234],{"class":65,"line":5520},[63,8235,8236],{"class":2731},"                \u002F\u002F so UseAuthentication sees a valid token and lets this request through\n",[63,8238,8239,8241,8243,8245,8247,8249,8251,8253,8255,8258,8260,8263,8265,8267],{"class":65,"line":5545},[63,8240,7484],{"class":87},[63,8242,92],{"class":91},[63,8244,7449],{"class":87},[63,8246,92],{"class":91},[63,8248,1990],{"class":87},[63,8250,92],{"class":91},[63,8252,7101],{"class":95},[63,8254,142],{"class":91},[63,8256,8257],{"class":145},"\"Authorization\"",[63,8259,508],{"class":91},[63,8261,8262],{"class":145},"\"Bearer \"",[63,8264,6165],{"class":132},[63,8266,8108],{"class":528},[63,8268,149],{"class":91},[63,8270,8271],{"class":65,"line":5550},[63,8272,7499],{"class":91},[63,8274,8276],{"class":65,"line":8275},30,[63,8277,7517],{"class":91},[63,8279,8281],{"class":65,"line":8280},31,[63,8282,621],{"class":91},[63,8284,8286],{"class":65,"line":8285},32,[63,8287,588],{"emptyLinePlaceholder":587},[63,8289,8291,8293,8295],{"class":65,"line":8290},33,[63,8292,3387],{"class":91},[63,8294,7872],{"class":95},[63,8296,403],{"class":91},[63,8298,8300],{"class":65,"line":8299},34,[63,8301,155],{"class":91},[63,8303,8305],{"class":65,"line":8304},35,[63,8306,588],{"emptyLinePlaceholder":587},[63,8308,8310,8312,8314,8317],{"class":65,"line":8309},36,[63,8311,7562],{"class":87},[63,8313,92],{"class":91},[63,8315,8316],{"class":95},"UseAuthentication",[63,8318,403],{"class":91},[63,8320,8322,8324,8326,8329],{"class":65,"line":8321},37,[63,8323,7562],{"class":87},[63,8325,92],{"class":91},[63,8327,8328],{"class":95},"UseAuthorization",[63,8330,403],{"class":91},[16,8332,8333],{},"Four steps:",[170,8335,8336,8342,8351,8357],{},[173,8337,8338,8341],{},[2055,8339,8340],{},"Check if a refresh is needed."," Look at the access token cookie. If it's missing or expired, and there's a refresh token cookie and a username cookie, proceed.",[173,8343,8344,8347,8348,8350],{},[2055,8345,8346],{},"Validate the refresh token against the server-side store."," This is the part that makes refresh tokens useful: the client doesn't get to declare what's valid. The server has the last say. If the client's refresh token doesn't match what we have on file, the refresh fails silently and the request falls through to be rejected by ",[32,8349,8316],{}," normally.",[173,8352,8353,8356],{},[2055,8354,8355],{},"Issue new tokens and write them as cookies on the response."," The browser picks them up and uses them on the next request automatically. The user is now \"logged in again\" without clicking anything.",[173,8358,8359,8366,8367,8369,8370,8372],{},[2055,8360,8361,8362,8365],{},"Inject the new access token into the current request's ",[32,8363,8364],{},"Authorization"," header."," This is the bit that makes the refresh transparent. Without it, the middleware would refresh the token for future requests, but the current request would still hit ",[32,8368,8316],{}," with the expired (or missing) cookie token, fail, and return 401. The injection puts the new token where ",[32,8371,8316],{}," looks for it, so the request that triggered the refresh also succeeds.",[11,8374,8376],{"id":8375},"order-of-operations","Order of operations",[16,8378,8379],{},"Pipeline order is doing real work here:",[54,8381,8383],{"className":78,"code":8382,"language":80,"meta":59,"style":59},"app.Use(\u002F* auto-refresh *\u002F);\napp.UseAuthentication();\napp.UseAuthorization();\n",[32,8384,8385,8400,8410],{"__ignoreMap":59},[63,8386,8387,8389,8391,8393,8395,8398],{"class":65,"line":66},[63,8388,7562],{"class":87},[63,8390,92],{"class":91},[63,8392,7858],{"class":95},[63,8394,142],{"class":91},[63,8396,8397],{"class":2731},"\u002F* auto-refresh *\u002F",[63,8399,149],{"class":91},[63,8401,8402,8404,8406,8408],{"class":65,"line":115},[63,8403,7562],{"class":87},[63,8405,92],{"class":91},[63,8407,8316],{"class":95},[63,8409,403],{"class":91},[63,8411,8412,8414,8416,8418],{"class":65,"line":121},[63,8413,7562],{"class":87},[63,8415,92],{"class":91},[63,8417,8328],{"class":95},[63,8419,403],{"class":91},[16,8421,8422,8423,8425],{},"The auto-refresh middleware has to run BEFORE ",[32,8424,8316],{},", because:",[1789,8427,8428,8436,8441],{},[173,8429,8430,8432,8433,92],{},[32,8431,8316],{}," is what reads the JWT, validates the signature, and builds the ",[32,8434,8435],{},"ClaimsPrincipal",[173,8437,4311,8438,8440],{},[32,8439,8316],{}," runs first with an expired token, it rejects the request. There's no opportunity to refresh.",[173,8442,8443,8444,8446],{},"The auto-refresh middleware needs to have rewritten the request before ",[32,8445,8316],{}," reads it.",[16,8448,8449,8450,8452],{},"Put the refresh middleware after ",[32,8451,8316],{}," and every protected request with an expired token returns 401, which puts you back on the manual flow.",[16,8454,8455],{},"There's a subtle point in step 4 above. Just setting the cookie on the response doesn't help the current request, because cookies are read from the request, not the response. The browser hasn't seen the new cookie yet; it's still attached to the old one for this round trip. The injection into the request headers is what lets the current request go through with the new token.",[11,8457,8459],{"id":8458},"walking-through-it","Walking through it",[16,8461,8462,8463,8466,8467,8470,8471,3246,8473,8475,8476,8478,8479,8482,8483,8486],{},"Run the demo with ",[32,8464,8465],{},"dotnet run",", then navigate to the URL the console shows (something like ",[32,8468,8469],{},"http:\u002F\u002Flocalhost:5265\u002Fscalar","). The Scalar UI lists all four endpoints. ",[32,8472,7316],{},[32,8474,7551],{}," are public. ",[32,8477,7817],{}," requires any authenticated user, and ",[32,8480,8481],{},"\u002Fadmin-only"," requires the ",[32,8484,8485],{},"Admin"," role on top of that.",[16,8488,8489],{},[7330,8490],{"alt":8491,"src":8492},"Scalar UI listing the endpoints","\u002Fblog\u002Fjwt-with-auto-refresh-in-cookies\u002Fscalar-endpoints.png",[16,8494,8495],{},"The walkthrough:",[170,8497,8498,8511,8520,8523],{},[173,8499,8500,8501,8503,8504,8507,8508,92],{},"Hit ",[32,8502,7316],{}," with ",[32,8505,8506],{},"{\"username\":\"admin\",\"password\":\"password\"}",". You're logged in. Cookies are set. The response body is ",[32,8509,8510],{},"{\"message\":\"Logged in via Cookies\"}",[173,8512,8500,8513,8515,8516,8519],{},[32,8514,7817],{},". Returns ",[32,8517,8518],{},"Hello User!",". Same connection, same cookies, no fuss.",[173,8521,8522],{},"Wait one minute. The access token is now expired.",[173,8524,8500,8525,8527,8528,8530,8531,8534],{},[32,8526,7817],{}," again. Returns ",[32,8529,8518],{},". ",[2055,8532,8533],{},"No 401, no client-side retry, no client involvement."," The auto-refresh middleware noticed the access token was expired, validated the refresh token, issued a new access token, set fresh cookies on the response, injected the new token into the request, and let it flow through. From the client's side, the request just worked.",[16,8536,8537,8538,8540,8541,8544,8545,3246,8548,8551,8552,8555,8556,8558],{},"That last request, the post-expiry ",[32,8539,7817],{}," call, is the one to watch. Open DevTools, Network tab, before you make it. After the request, click the entry, and look at the Response Headers panel. You should see two fresh ",[32,8542,8543],{},"Set-Cookie"," lines (",[32,8546,8547],{},"X-Access-Token=...",[32,8549,8550],{},"X-Refresh-Token=...",") on a ",[32,8553,8554],{},"200 OK"," response for ",[32,8557,7817],{},". That's the auto-refresh: the server issued new credentials inside a request that, from the client's side, was a normal call.",[16,8560,8561],{},[7330,8562],{"alt":8563,"src":8564},"Network panel showing Set-Cookie on a 200 OK dashboard response","\u002Fblog\u002Fjwt-with-auto-refresh-in-cookies\u002Fauto-refresh-network.png",[11,8566,8568],{"id":8567},"what-happens-on-failure","What happens on failure",[16,8570,8571],{},"Scenarios the middleware needs to handle, and how the demo handles them:",[1789,8573,8574,8589,8595,8601],{},[173,8575,8576,4020,8579,8582,8583,8585,8586,8588],{},[2055,8577,8578],{},"No access token cookie at all.",[32,8580,8581],{},"IsTokenExpired(null)"," returns ",[32,8584,1206],{},". If there's no refresh token cookie either, the middleware does nothing, and ",[32,8587,8316],{}," returns 401.",[173,8590,8591,8594],{},[2055,8592,8593],{},"Access token expired, refresh token missing."," Middleware does nothing, request falls through to 401. The user logs in again.",[173,8596,8597,8600],{},[2055,8598,8599],{},"Access token expired, refresh token present but doesn't match the server's record."," Lookup fails, middleware does nothing, request gets 401. This is what protects against a stolen refresh token: if the server's record doesn't match (because the user logged out, or the refresh was rotated by another request), the stolen token is useless.",[173,8602,8603,4020,8606,8609],{},[2055,8604,8605],{},"Refresh token matches but the user no longer exists.",[32,8607,8608],{},"users.FirstOrDefault(...)"," returns null, middleware does nothing, request gets 401.",[16,8611,8612,8613,8615],{},"The shape in all four is \"if anything fails, let the request fall through to ",[32,8614,8316],{}," and get rejected the normal way.\" The middleware never throws on its own. It either succeeds and rewrites the request, or it sits out and lets the existing auth layer say no.",[11,8617,8619],{"id":8618},"what-youd-add-for-production","What you'd add for production",[16,8621,8622,8623,8625],{},"The middleware, the cookie configuration, the ",[32,8624,7021],{}," hook, the auto-refresh flow: all of those are production-shaped as-is. What needs to change is mostly the data plumbing:",[1789,8627,8628,8641,8651,8661,8667,8673,8685],{},[173,8629,8630,8636,8637,8640],{},[2055,8631,8632,8633,92],{},"Refresh tokens in ",[32,8634,8635],{},"Dictionary\u003Cstring, string>"," Replace with a database table (or Redis), keyed by user ID. Store the ",[2055,8638,8639],{},"hash"," of the refresh token, not the token itself; compare hashes on validation. When you horizontally scale, the store needs to be shared, so process-local memory doesn't cut it.",[173,8642,8643,8646,8647,8650],{},[2055,8644,8645],{},"Refresh token rotation with reuse detection."," The demo rotates on every refresh (",[32,8648,8649],{},"userRefreshTokenDict[username!] = newRefreshToken;","), which is correct. Production should also detect the \"reuse of an already-rotated refresh token\" case and revoke everything for that user; reuse means either a bug or a stolen token, and either way you want to invalidate.",[173,8652,8653,8656,8657,8660],{},[2055,8654,8655],{},"Token versioning for revocation."," Add a ",[32,8658,8659],{},"tokenVersion"," claim to the JWT, compare against a value in the user table on every refresh. Lets you force-invalidate all active tokens (logout-from-all-devices, password change, suspected compromise).",[173,8662,8663,8666],{},[2055,8664,8665],{},"Hardcoded signing key."," Move to configuration, ideally a managed secret store, rotated on a schedule. For higher-security setups, switch from HS256 (shared secret) to RS256 (public\u002Fprivate keypair) so verifiers don't need the signing key.",[173,8668,8669,8672],{},[2055,8670,8671],{},"In-memory users with plaintext passwords."," Use the real user table. Passwords hashed with BCrypt or Argon2id, never plaintext or just-SHA256.",[173,8674,8675,8680,8681,8684],{},[2055,8676,8677,8678,92],{},"CSRF beyond ",[32,8679,7074],{}," Strict gets you most of the way. If you ever have to relax to ",[32,8682,8683],{},"Lax"," (cross-subdomain navigation, OAuth callbacks), add an anti-forgery token on state-changing endpoints.",[173,8686,8687,8690,8691,8694],{},[2055,8688,8689],{},"HTTPS only."," Already enforced by ",[32,8692,8693],{},"Secure=true"," on the cookies, but the host itself should refuse HTTP entirely in production.",[11,8696,4106],{"id":4105},[16,8698,8699],{},"Clone the repo, run it, watch the cookies rotate in DevTools, and you'll have most of what you need to build this into a real codebase.",[16,8701,2558,8702],{},[20,8703,7014],{"href":7014,"rel":8704},[52],[2563,8706,8707],{},"html pre.shiki code .s7GmK, html code.shiki .s7GmK{--shiki-default:#383A42;--shiki-dark:#E5C07B}html pre.shiki code .s5ixo, html code.shiki .s5ixo{--shiki-default:#383A42;--shiki-dark:#ABB2BF}html pre.shiki code .sAdtL, html code.shiki .sAdtL{--shiki-default:#4078F2;--shiki-dark:#61AFEF}html pre.shiki code .sDhpE, html code.shiki .sDhpE{--shiki-default:#50A14F;--shiki-dark:#98C379}html pre.shiki code .sz0mV, html code.shiki .sz0mV{--shiki-default:#383A42;--shiki-dark:#E06C75}html pre.shiki code .sC09Y, html code.shiki .sC09Y{--shiki-default:#C18401;--shiki-dark:#E5C07B}html pre.shiki code .sknuh, html code.shiki .sknuh{--shiki-default:#383A42;--shiki-dark:#56B6C2}html pre.shiki code .sAGMh, html code.shiki .sAGMh{--shiki-default:#986801;--shiki-dark:#D19A66}html pre.shiki code .sW2Sy, html code.shiki .sW2Sy{--shiki-default:#A0A1A7;--shiki-default-font-style:italic;--shiki-dark:#7F848E;--shiki-dark-font-style:italic}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sLKXg, html code.shiki .sLKXg{--shiki-default:#A626A4;--shiki-dark:#C678DD}",{"title":59,"searchDepth":115,"depth":115,"links":8709},[8710,8711,8712,8713,8714,8715,8716,8717,8718,8719,8720],{"id":13,"depth":115,"text":14},{"id":7025,"depth":115,"text":7026},{"id":7056,"depth":115,"text":7057},{"id":7336,"depth":115,"text":7337},{"id":7541,"depth":115,"text":7542},{"id":7837,"depth":115,"text":7838},{"id":8375,"depth":115,"text":8376},{"id":8458,"depth":115,"text":8459},{"id":8567,"depth":115,"text":8568},{"id":8618,"depth":115,"text":8619},{"id":4105,"depth":115,"text":4106},"2025-12-05","A walkthrough of a small ASP.NET Core demo that handles access tokens, refresh tokens, and a middleware that refreshes them transparently.",{},"\u002Fblog\u002Fjwt-with-auto-refresh-in-cookies",{"title":6985,"description":8722},"blog\u002Fjwt-with-auto-refresh-in-cookies",[2594],"1i9fJ7H91Wcd4oTrXxDApOAxP3xdVHXK0aK8x33HRCI",{"id":8730,"title":8731,"body":8732,"book":2585,"date":9874,"description":9875,"extension":2588,"meta":9876,"navigation":587,"path":9877,"seo":9878,"stem":9879,"tags":9880,"__hash__":9881},"blog\u002Fblog\u002Fbuilding-ml-inference-part-3.md","Building an ML Inference API, Part III",{"type":8,"value":8733,"toc":9862},[8734,8738,8750,8779,8786,8792,8795,8800,8851,8855,8862,8868,8871,8881,8898,8911,8920,8931,8942,8946,8952,8955,8997,9000,9031,9034,9041,9045,9052,9058,9064,9077,9088,9104,9121,9127,9130,9138,9145,9149,9159,9177,9187,9203,9206,9239,9242,9250,9253,9259,9316,9326,9337,9340,9346,9396,9399,9410,9423,9467,9475,9498,9501,9505,9516,9522,9533,9536,9542,9553,9556,9578,9583,9589,9604,9607,9625,9628,9639,9650,9654,9657,9671,9678,9684,9690,9703,9709,9726,9729,9734,9745,9756,9760,9859],[11,8735,8737],{"id":8736},"template","Template",[16,8739,8740,8741,8743,8744,8749],{},"As a follow-up to ",[20,8742,4228],{"href":4227},": if you are looking for a template to start building an ML Inference API engine, check out ",[20,8745,8748],{"href":8746,"rel":8747},"https:\u002F\u002Fgithub.com\u002Fhaiilong\u002Fml-inference-api-template",[52],"haiilong\u002Fml-inference-api-template"," on GitHub. It covers:",[170,8751,8752,8755,8758,8761,8764,8767,8770,8773,8776],{},[173,8753,8754],{},"Project structure",[173,8756,8757],{},"Sample codes with 2 different endpoints",[173,8759,8760],{},"2 sample models",[173,8762,8763],{},"Middlewares set up",[173,8765,8766],{},"Dockerfile",[173,8768,8769],{},"uv as Python package manager",[173,8771,8772],{},"Unit tests (pytest), load tests (k6), e2e tests (node-fetch)",[173,8774,8775],{},"Deployment (Docker registry, Kubernetes cluster, gitlab-ci)",[173,8777,8778],{},"How to extend and apply to your use case",[16,8780,8781,8782,8785],{},"This structure is ",[2055,8783,8784],{},"production-ready",". Of course you can always add your own stuff like HPA, VPA or other middlewares if needed.",[16,8787,8788,8789,92],{},"PLEASE READ THE README CAREFULLY IF YOU WANT TO USE THIS TEMPLATE ",[8790,8791],"d",{},[16,8793,8794],{},"In this blog, I will be explaining the technical decisions used in this template that we eventually used for all ML Inference API.",[16,8796,8797],{},[2055,8798,8799],{},"Sections:",[170,8801,8802,8808,8814,8820,8826,8839,8845],{},[173,8803,8804],{},[20,8805,8807],{"href":8806},"#_1-why-fastapi-uvicorn-gunicorn-not-flask-gevent","Why FastAPI \u002F Uvicorn \u002F Gunicorn (not Flask + gevent)",[173,8809,8810],{},[20,8811,8813],{"href":8812},"#_2-why-orjson-over-the-default-json-encoder","Why orjson over the default JSON encoder",[173,8815,8816],{},[20,8817,8819],{"href":8818},"#_3-why-uv-over-conda","Why uv over conda",[173,8821,8822],{},[20,8823,8825],{"href":8824},"#_4-anyio-vs-asyncio","anyio vs asyncio",[173,8827,8828],{},[20,8829,8831,8834,8835,8838],{"href":8830},"#_5-async-def-plus-run_in_threadpool-the-importance-of-the-offload",[32,8832,8833],{},"async def"," plus ",[32,8836,8837],{},"run_in_threadpool"," (the importance of the offload)",[173,8840,8841],{},[20,8842,8844],{"href":8843},"#_6-why-an-initcontainer-loads-models-from-a-pvc-in-kubernetes","Why an initContainer loads models from a PVC in Kubernetes",[173,8846,8847],{},[20,8848,8850],{"href":8849},"#_7-hot-reload-models-in-production-or-redeploy","Hot reload models in production, or redeploy?",[11,8852,8854],{"id":8853},"_1-why-fastapi-uvicorn-gunicorn-not-flask-gevent","1. Why FastAPI \u002F Uvicorn \u002F Gunicorn (not Flask + gevent)",[16,8856,8857,8858,8861],{},"A common Python ML serving stack is Flask running under gunicorn with gevent workers. It works, it has been deployed everywhere, and for very simple cases it is fine. The template uses ",[2055,8859,8860],{},"FastAPI on Uvicorn workers under Gunicorn"," instead. Reasons:",[16,8863,8864,8867],{},[2055,8865,8866],{},"Concurrency model."," Gevent achieves concurrency by monkey-patching the standard library to make blocking I\u002FO cooperatively yield. That model is great for I\u002FO-bound web apps but has well-known sharp edges with C extensions. NumPy, scikit-learn, XGBoost, and CatBoost all spend most of their time inside C code that does not yield to gevent's scheduler, so a single slow inference still blocks every request that the worker is multiplexing. You end up tuning around the very thing you wanted concurrency for.",[16,8869,8870],{},"FastAPI on Uvicorn uses an asyncio event loop. CPU-bound inference is offloaded explicitly to a bounded thread pool (see section 5), which means the loop never blocks even when prediction takes 200 ms. It is the same end goal as gevent but with explicit boundaries instead of monkey-patched implicit ones.",[16,8872,8873,8876,8877,8880],{},[2055,8874,8875],{},"Validation and schema for free."," FastAPI builds on Pydantic. Defining a request body as a ",[32,8878,8879],{},"BaseModel"," gives you:",[1789,8882,8883,8886,8891],{},[173,8884,8885],{},"automatic 422 responses with field-level error messages on bad input,",[173,8887,8888,8889,5175],{},"generated OpenAPI \u002F Swagger UI at ",[32,8890,4120],{},[173,8892,8893,8894,8897],{},"typed request handlers, so editor tooling and ",[32,8895,8896],{},"mypy"," actually understand the code.",[16,8899,8900,8901,508,8904,508,8907,8910],{},"In Flask, you write that boilerplate by hand or pull in extensions (",[32,8902,8903],{},"marshmallow",[32,8905,8906],{},"flask-pydantic",[32,8908,8909],{},"flask-smorest","). For an inference API where the request shape is the contract with the caller, the FastAPI default is much closer to \"what you would build anyway.\"",[16,8912,8913,8916,8917,8880],{},[2055,8914,8915],{},"Why Gunicorn at all if Uvicorn can serve directly."," Uvicorn is the ASGI server (one process, one event loop). Gunicorn is a process supervisor that manages multiple worker processes, handles graceful reload on SIGHUP, restarts crashed workers, listens on a single socket and load-balances across workers, and integrates with most ops tooling. The combination ",[32,8918,8919],{},"gunicorn -k uvicorn.workers.UvicornWorker",[1789,8921,8922,8925,8928],{},[173,8923,8924],{},"process-level isolation across CPU cores (one event loop per worker),",[173,8926,8927],{},"production-grade signal handling and graceful shutdown,",[173,8929,8930],{},"the ASGI runtime you actually want.",[16,8932,8933,8934,8937,8938,8941],{},"Tuning: set ",[32,8935,8936],{},"GUNICORN_WORKERS"," to roughly ",[32,8939,8940],{},"min(cpu_count, target_concurrency \u002F threads_per_worker)",". For inference workloads, one worker per CPU core is usually a good starting point.",[11,8943,8945],{"id":8944},"_2-why-orjson-over-the-default-json-encoder","2. Why orjson over the default JSON encoder",[16,8947,8948,8949,8951],{},"Python's stdlib ",[32,8950,3808],{}," module is pure Python. For inference APIs that return predictions as JSON, the encoder can become a measurable share of total request time, especially when responses contain numpy floats or longer batch outputs.",[16,8953,8954],{},"orjson is written in Rust and:",[1789,8956,8957,8963,8977,8994],{},[173,8958,8959,8960,8962],{},"Serializes typically 3 to 10 times faster than stdlib ",[32,8961,3808],{}," and 2 to 4 times faster than ujson.",[173,8964,8965,8966,8969,8970,8972,8973,8976],{},"Returns ",[32,8967,8968],{},"bytes"," directly, which is the native ASGI write type. Stdlib ",[32,8971,3808],{}," produces a ",[32,8974,8975],{},"str"," that has to be encoded to bytes again.",[173,8978,8979,8980,508,8983,508,8986,8989,8990,8993],{},"Handles ",[32,8981,8982],{},"datetime",[32,8984,8985],{},"UUID",[32,8987,8988],{},"dataclasses",", and (via passthrough flags) numpy arrays without a custom ",[32,8991,8992],{},"default="," function.",[173,8995,8996],{},"Is strict about the JSON spec (no NaN by default, deterministic key order if asked).",[16,8998,8999],{},"We wire it in once on the app object:",[54,9001,9003],{"className":5066,"code":9002,"language":5068,"meta":59,"style":59},"app = FastAPI(default_response_class=ORJSONResponse, ...)\n",[32,9004,9005],{"__ignoreMap":59},[63,9006,9007,9010,9012,9015,9017,9020,9022,9025,9029],{"class":65,"line":66},[63,9008,9009],{"class":91},"app ",[63,9011,577],{"class":132},[63,9013,9014],{"class":5101}," FastAPI",[63,9016,142],{"class":91},[63,9018,9019],{"class":5107},"default_response_class",[63,9021,577],{"class":132},[63,9023,9024],{"class":91},"ORJSONResponse, ",[63,9026,9028],{"class":9027},"sYebD","...",[63,9030,474],{"class":91},[16,9032,9033],{},"After that, every endpoint returns orjson-encoded responses with no per-route changes.",[16,9035,9036,9037,9040],{},"When ",[3240,9038,9039],{},"not"," to bother: if your responses are tiny (a single float) and your throughput is low (under 100 RPS), the win is nanoseconds and you should not care. For batch predictions, payload sizes grow linearly with batch size and orjson is the right default.",[11,9042,9044],{"id":9043},"_3-why-uv-over-conda","3. Why uv over conda",[16,9046,9047,9048,9051],{},"ML projects often default to conda because that is what notebook environments use. For a deployable inference service, ",[32,9049,9050],{},"uv"," is dramatically better.",[16,9053,9054,9057],{},[2055,9055,9056],{},"Speed."," uv is written in Rust. A clean dependency install for this template takes a few seconds. The equivalent conda env solve (especially with custom channels and pinned versions) routinely takes minutes. CI and Docker builds inherit that delta.",[16,9059,9060,9063],{},[2055,9061,9062],{},"Simpler container images."," With conda, you typically end up with two Dockerfiles:",[1789,9065,9066,9072],{},[173,9067,9068,9071],{},[32,9069,9070],{},"Dockerfile.base"," that installs micromamba and creates the env.",[173,9073,9074,9076],{},[32,9075,8766],{}," that copies source on top of the base.",[16,9078,9079,9080,9083,9084,9087],{},"Plus a ",[32,9081,9082],{},"Dockerfile.local"," and a ",[32,9085,9086],{},"Dockerfile.test"," to keep things consistent. The base image has to be pre-built and cached in a registry to keep main builds fast.",[16,9089,9090,9091,9094,9095,9098,9099,9101,9102,6359],{},"With uv, the production image is one stage, no base, no separate registry artifact. ",[32,9092,9093],{},"uv sync --frozen --no-dev"," installs into ",[32,9096,9097],{},"\u002Fopt\u002Fvenv"," quickly enough that you do not need a pre-built base. The template uses a single ",[32,9100,8766],{}," (plus a tiny ",[32,9103,9086],{},[16,9105,9106,4020,9109,9112,9113,9116,9117,9120],{},[2055,9107,9108],{},"First-class lockfile.",[32,9110,9111],{},"uv.lock"," is committed and reproducible. ",[32,9114,9115],{},"uv lock --check"," in CI fails fast if anyone forgot to update it. Conda's lockfile story (",[32,9118,9119],{},"conda-lock",") works but is a separate tool with separate pitfalls.",[16,9122,9123,9126],{},[2055,9124,9125],{},"Pure PyPI."," No custom channels. No channel ordering bugs. No \"which channel does this come from\" mysteries.",[16,9128,9129],{},"When conda still wins:",[1789,9131,9132,9135],{},[173,9133,9134],{},"Hard non-Python dependencies that conda packages provide and PyPI does not (some MKL builds, some GIS stacks, some CUDA-pinned tooling).",[173,9136,9137],{},"Notebook \u002F data science workflows where you want one tool managing kernels, Python versions, and packages together.",[16,9139,9140,9141,9144],{},"For an ML inference ",[3240,9142,9143],{},"service"," whose dependencies are joblib, scikit-learn, xgboost, and FastAPI, those gaps do not apply.",[11,9146,9148],{"id":9147},"_4-anyio-vs-asyncio","4. anyio vs asyncio",[16,9150,9151,9154,9155,9158],{},[32,9152,9153],{},"asyncio"," is the stdlib async runtime. ",[32,9156,9157],{},"anyio"," is a higher-level wrapper that runs on top of asyncio (or trio) and adds:",[1789,9160,9161,9164,9171,9174],{},[173,9162,9163],{},"structured concurrency (task groups with proper cancellation propagation),",[173,9165,9166,9167,9170],{},"a global thread limiter so ",[32,9168,9169],{},"anyio.to_thread.run_sync"," does not spawn unbounded threads,",[173,9172,9173],{},"cleaner cancel scopes and timeout primitives,",[173,9175,9176],{},"a portable API: the same code runs on asyncio or trio.",[16,9178,9179,9180,9182,9183,9186],{},"You do not need to import ",[32,9181,9157],{}," directly. ",[2055,9184,9185],{},"Starlette and FastAPI use anyio internally",", which means:",[1789,9188,9189,9197,9200],{},[173,9190,9191,9194,9195,92],{},[32,9192,9193],{},"fastapi.concurrency.run_in_threadpool"," is a thin wrapper over ",[32,9196,9169],{},[173,9198,9199],{},"The thread pool that runs your offloaded inference is anyio's, capped by anyio's global limiter (default 40 threads).",[173,9201,9202],{},"Dependency injection, background tasks, and lifespan all run on the anyio runtime.",[16,9204,9205],{},"Practical implications:",[1789,9207,9208,9221,9228],{},[173,9209,9210,9211,9214,9215,9217,9218,9220],{},"Do not write ",[32,9212,9213],{},"asyncio.to_thread(...)"," in handlers. It bypasses the limiter and creates a separate pool that anyio cannot govern. Use ",[32,9216,8837],{}," (or ",[32,9219,9169],{}," directly).",[173,9222,9223,9224,9227],{},"If you need to raise the thread limit for high-concurrency CPU-bound inference, do it once at startup with ",[32,9225,9226],{},"anyio.to_thread.current_default_thread_limiter().total_tokens = N",". Be deliberate; threads have memory overhead and switching cost.",[173,9229,9230,9231,9234,9235,9238],{},"Mixing ",[32,9232,9233],{},"asyncio.create_task"," is fine, but prefer ",[32,9236,9237],{},"anyio.create_task_group"," for anything where you want clean cancellation on error.",[16,9240,9241],{},"In short: anyio is the runtime; asyncio is the engine underneath. Code against the FastAPI \u002F anyio surface and you stay portable and bounded.",[11,9243,9245,9246,8834,9248,8838],{"id":9244},"_5-async-def-plus-run_in_threadpool-the-importance-of-the-offload","5. ",[32,9247,8833],{},[32,9249,8837],{},[16,9251,9252],{},"The README covers the basic recipe. This section explains the \"why it matters\" part.",[16,9254,9255,9258],{},[2055,9256,9257],{},"The pitfall."," A handler written like this looks innocent:",[54,9260,9262],{"className":5066,"code":9261,"language":5068,"meta":59,"style":59},"@app.post(\"\u002Fpredict\u002Fprice\")\nasync def post_predict_price(request: PricePredictionRequest):\n    return calculate_price(request)  # WRONG: blocking call inside async handler\n",[32,9263,9264,9281,9303],{"__ignoreMap":59},[63,9265,9266,9269,9271,9274,9276,9279],{"class":65,"line":66},[63,9267,9268],{"class":95},"@app",[63,9270,92],{"class":2769},[63,9272,9273],{"class":95},"post",[63,9275,142],{"class":91},[63,9277,9278],{"class":145},"\"\u002Fpredict\u002Fprice\"",[63,9280,474],{"class":91},[63,9282,9283,9285,9288,9291,9293,9295,9297,9301],{"class":65,"line":115},[63,9284,7863],{"class":439},[63,9286,9287],{"class":439}," def",[63,9289,9290],{"class":95}," post_predict_price",[63,9292,142],{"class":91},[63,9294,2022],{"class":5171},[63,9296,429],{"class":91},[63,9298,9300],{"class":9299},"sxymB"," PricePredictionRequest",[63,9302,5185],{"class":91},[63,9304,9305,9307,9310,9313],{"class":65,"line":121},[63,9306,1890],{"class":439},[63,9308,9309],{"class":5101}," calculate_price",[63,9311,9312],{"class":91},"(request)  ",[63,9314,9315],{"class":2731},"# WRONG: blocking call inside async handler\n",[16,9317,9318,9321,9322,9325],{},[32,9319,9320],{},"calculate_price"," calls ",[32,9323,9324],{},"model.predict(...)",", which is a synchronous CPU-bound C extension call. The event loop is blocked for the entire duration of that call. While it runs:",[1789,9327,9328,9331,9334],{},[173,9329,9330],{},"no other request handler on this worker can make progress,",[173,9332,9333],{},"no health check can be answered,",[173,9335,9336],{},"no timeout, cancellation, or middleware can interleave.",[16,9338,9339],{},"If predictions take 50 ms and you receive 100 concurrent requests, the 100th caller waits 5 seconds for what should be a 50 ms operation. The event loop's whole value proposition (cheap concurrency) is wasted.",[16,9341,9342,9345],{},[2055,9343,9344],{},"The fix."," Offload the blocking call to a worker thread:",[54,9347,9349],{"className":5066,"code":9348,"language":5068,"meta":59,"style":59},"@app.post(\"\u002Fpredict\u002Fprice\")\nasync def post_predict_price(request: PricePredictionRequest):\n    return await run_in_threadpool(calculate_price, request)\n",[32,9350,9351,9365,9383],{"__ignoreMap":59},[63,9352,9353,9355,9357,9359,9361,9363],{"class":65,"line":66},[63,9354,9268],{"class":95},[63,9356,92],{"class":2769},[63,9358,9273],{"class":95},[63,9360,142],{"class":91},[63,9362,9278],{"class":145},[63,9364,474],{"class":91},[63,9366,9367,9369,9371,9373,9375,9377,9379,9381],{"class":65,"line":115},[63,9368,7863],{"class":439},[63,9370,9287],{"class":439},[63,9372,9290],{"class":95},[63,9374,142],{"class":91},[63,9376,2022],{"class":5171},[63,9378,429],{"class":91},[63,9380,9300],{"class":9299},[63,9382,5185],{"class":91},[63,9384,9385,9387,9390,9393],{"class":65,"line":121},[63,9386,1890],{"class":439},[63,9388,9389],{"class":439}," await",[63,9391,9392],{"class":5101}," run_in_threadpool",[63,9394,9395],{"class":91},"(calculate_price, request)\n",[16,9397,9398],{},"Now the event loop awaits the thread pool and stays responsive. Other requests are served concurrently up to the size of the thread pool. The Python GIL still serializes pure-Python work between threads, but ML libraries (numpy, sklearn, xgboost) release the GIL inside their C kernels, so multi-threaded inference does scale.",[16,9400,9401],{},[2055,9402,9403,9404,9406,9407,9409],{},"Why not let FastAPI do it implicitly with ",[32,9405,5163],{}," instead of ",[32,9408,8833],{},"?",[16,9411,9412,9413,9416,9417,9419,9420,9422],{},"If you write a ",[3240,9414,9415],{},"sync"," handler (",[32,9418,5163],{},", not ",[32,9421,8833],{},"), FastAPI auto-runs it in the threadpool for you. So you could write:",[54,9424,9426],{"className":5066,"code":9425,"language":5068,"meta":59,"style":59},"@app.post(\"\u002Fpredict\u002Fprice\")\ndef post_predict_price(request: PricePredictionRequest):\n    return calculate_price(request)\n",[32,9427,9428,9442,9458],{"__ignoreMap":59},[63,9429,9430,9432,9434,9436,9438,9440],{"class":65,"line":66},[63,9431,9268],{"class":95},[63,9433,92],{"class":2769},[63,9435,9273],{"class":95},[63,9437,142],{"class":91},[63,9439,9278],{"class":145},[63,9441,474],{"class":91},[63,9443,9444,9446,9448,9450,9452,9454,9456],{"class":65,"line":115},[63,9445,5163],{"class":439},[63,9447,9290],{"class":95},[63,9449,142],{"class":91},[63,9451,2022],{"class":5171},[63,9453,429],{"class":91},[63,9455,9300],{"class":9299},[63,9457,5185],{"class":91},[63,9459,9460,9462,9464],{"class":65,"line":121},[63,9461,1890],{"class":439},[63,9463,9309],{"class":5101},[63,9465,9466],{"class":91},"(request)\n",[16,9468,9469,9470,8834,9472,9474],{},"and it would behave correctly. The reason the template prefers explicit ",[32,9471,8833],{},[32,9473,8837],{}," anyway:",[1789,9476,9477,9483,9492],{},[173,9478,9479,9482],{},[2055,9480,9481],{},"Boundary visibility."," The reader sees exactly which call is blocking and which is async. With auto-offload, every reader has to remember the FastAPI rule.",[173,9484,9485,9488,9489,9491],{},[2055,9486,9487],{},"Mixed work."," The moment you need to await something else (fetch a fresh feature from a cache, log to an async sink, call a remote service) you must convert the handler to ",[32,9490,8833],{},". Starting that way avoids the rewrite.",[173,9493,9494,9497],{},[2055,9495,9496],{},"Middleware composition."," Async middleware (like the access log middleware in this template) interleaves more cleanly when handlers are async and only the inference itself is offloaded.",[16,9499,9500],{},"Both patterns are valid. The template picks the more explicit one.",[11,9502,9504],{"id":9503},"_6-why-an-initcontainer-loads-models-from-a-pvc-in-kubernetes","6. Why an initContainer loads models from a PVC in Kubernetes",[16,9506,9507,9508,9511,9512,9515],{},"The deployment spec uses an ",[32,9509,9510],{},"initContainer"," that copies model files from a PVC into a shared ",[32,9513,9514],{},"emptyDir",", which the app container then mounts read-only. Why not just bake the model into the image?",[16,9517,9518,9521],{},[2055,9519,9520],{},"Image size and rebuild cost."," Model files are often hundreds of MB to several GB. Baking them into the image means:",[1789,9523,9524,9527,9530],{},[173,9525,9526],{},"every deploy pushes those bytes to the registry,",[173,9528,9529],{},"every node pulls them on cold start,",[173,9531,9532],{},"a model update requires a full image rebuild, registry round-trip, and rollout.",[16,9534,9535],{},"For frequently retrained models, this turns deploys into slow, expensive operations even when the code did not change.",[16,9537,9538,9541],{},[2055,9539,9540],{},"Decoupled lifecycle."," Models are produced by a training pipeline. Code is produced by an application repo. Tying them together in a single image means you cannot:",[1789,9543,9544,9547,9550],{},[173,9545,9546],{},"update a model without code review for the API repo,",[173,9548,9549],{},"run two pods with different model versions for A\u002FB testing without two images,",[173,9551,9552],{},"roll back the API independently of the model.",[16,9554,9555],{},"The PVC + initContainer pattern lets the training pipeline write to a known location (object storage backed by a PVC, or a model registry mount) and the API simply consumes whatever is there at pod start.",[16,9557,9558,9561,9562,9565,9566,9569,9570,9573,9574,9577],{},[2055,9559,9560],{},"Failure-loud at startup."," The ",[32,9563,9564],{},"cp ... || echo \"not found\""," pattern in the initContainer is intentionally permissive about ",[3240,9567,9568],{},"which"," files are present, but the ",[32,9571,9572],{},"lifespan"," handler in ",[32,9575,9576],{},"app.py"," is strict: a missing model crashes startup. That gives you the right behavior:",[1789,9579,9580],{},[173,9581,9582],{},"if the PVC mount is broken, the pod fails its readiness probe and the rolling deploy stops, so you never serve traffic with a partially loaded model.",[16,9584,9585,9588],{},[2055,9586,9587],{},"Trade-offs."," This pattern has costs:",[1789,9590,9591,9598,9601],{},[173,9592,9593,9594,9597],{},"a deployment is no longer fully described by ",[32,9595,9596],{},"image:tag"," alone; you also need to know what was in the PVC at the moment of pod start. For audit \u002F compliance, capture the model checksum at startup and log it.",[173,9599,9600],{},"the cluster needs shared storage. In simple single-node setups, baking the model into the image is fine.",[173,9602,9603],{},"cold start is slower (initContainer pulls model files into the emptyDir).",[16,9605,9606],{},"When to bake the model into the image instead:",[1789,9608,9609,9612,9615],{},[173,9610,9611],{},"the model is small (under ~100 MB) and changes only when code changes,",[173,9613,9614],{},"you do not have shared storage in the target cluster,",[173,9616,9617,9618,9621,9622,9624],{},"you specifically ",[3240,9619,9620],{},"want"," the deployment to be reproducible from ",[32,9623,9596],{}," alone (regulated environments).",[16,9626,9627],{},"When to use a model registry (MLflow, Vertex AI, S3 prefixes) instead of a PVC:",[1789,9629,9630,9633,9636],{},[173,9631,9632],{},"multiple clusters need the same model files,",[173,9634,9635],{},"you want versioned model URIs in the config rather than \"whatever is on the PVC,\"",[173,9637,9638],{},"you want rollback to a specific historical version.",[16,9640,9641,9642,9645,9646,9649],{},"The PVC pattern in the template is the smallest k8s-native version of \"model lives outside the image.\" Swap the PVC for a ",[32,9643,9644],{},"gcsfuse"," or ",[32,9647,9648],{},"s3fs"," mount, or replace the initContainer with an in-process download from a registry, when you outgrow it.",[11,9651,9653],{"id":9652},"_7-hot-reload-models-in-production-or-redeploy","7. Hot reload models in production, or redeploy?",[16,9655,9656],{},"Two designs:",[1789,9658,9659,9665],{},[173,9660,9661,9664],{},[2055,9662,9663],{},"(A) Hot reload."," A background task watches a model registry (or a file path, or polls an HTTP endpoint), and atomically swaps the in-memory model reference when a new version arrives. Pods stay up. Optionally, an admin endpoint triggers a manual reload.",[173,9666,9667,9670],{},[2055,9668,9669],{},"(B) Redeploy."," Model paths or versions are pinned in source \u002F config. Updating the model means rolling out a new pod (with a new initContainer fetch, or a new image).",[16,9672,9673,9674,9677],{},"This template uses ",[2055,9675,9676],{},"(B)",". The trade-offs:",[16,9679,9680,9683],{},[2055,9681,9682],{},"Reproducibility."," With (B), a git SHA plus an image tag (plus, for this template, the model file checksum logged at startup) fully describes runtime behavior. When something goes wrong in production at 02:00, you can reconstruct exactly what was running. With (A), you also need \"which model version was loaded in this pod at the moment of the request,\" which means more logging discipline and more places for drift.",[16,9685,9686,9689],{},[2055,9687,9688],{},"Atomic rollout."," Kubernetes already gives you a great rollout primitive: rolling deploys with health checks, automatic rollback on failed readiness, traffic shifting. With (B), updating a model uses that machinery for free. With (A), you reinvent it: you need a per-pod swap protocol, a way to drain in-flight requests off the old reference, and a rollback mechanism that is not just \"swap back.\"",[16,9691,9692,9695,9696,508,9699,9702],{},[2055,9693,9694],{},"Multi-pod consistency."," During a hot reload, different pods will briefly serve different model versions. Usually fine, occasionally surprising (especially if the model output range changed between versions and downstream consumers care). Rolling redeploys still cause this transiently, but the kubelet's rollout strategy gives you knobs (",[32,9697,9698],{},"maxSurge",[32,9700,9701],{},"maxUnavailable",") to bound the window. Hot reload across N pods does not.",[16,9704,9705,9708],{},[2055,9706,9707],{},"Failure surface."," Hot reload adds:",[1789,9710,9711,9714,9717,9720,9723],{},[173,9712,9713],{},"a background polling thread or scheduler,",[173,9715,9716],{},"a model registry client,",[173,9718,9719],{},"an admin endpoint or signal handler (which needs auth),",[173,9721,9722],{},"atomic swap logic that is correct under concurrent reads from N request handlers,",[173,9724,9725],{},"monitoring for \"did the swap actually happen on every pod?\"",[16,9727,9728],{},"Each of those is a place a bug can live. Redeploys reuse Kubernetes machinery you already trust.",[16,9730,9731],{},[2055,9732,9733],{},"When you would still want hot reload:",[1789,9735,9736,9739,9742],{},[173,9737,9738],{},"Models retrain hourly or faster and redeploys are expensive (multi-GB images, slow startup with large models, many pods).",[173,9740,9741],{},"You need A\u002FB testing where the routing changes at runtime, not at deploy time.",[173,9743,9744],{},"You operate at a fleet scale where triggering N redeploys is itself a problem.",[16,9746,9747,9748,9751,9752,9755],{},"For a template aimed at \"first ML inference service,\" (B) is correct: it is simpler, more reproducible, and gives you Kubernetes-native rollout for free. If you outgrow it, the structure of the template (a ",[32,9749,9750],{},"MLModels"," singleton with explicit accessor methods) makes adding a ",[32,9753,9754],{},"reload()"," method localized and safe.",[11,9757,9759],{"id":9758},"summary-table","Summary table",[1121,9761,9762,9775],{},[1124,9763,9764],{},[1127,9765,9766,9769,9772],{},[1130,9767,9768],{},"Decision",[1130,9770,9771],{},"Choice",[1130,9773,9774],{},"Main reason",[1140,9776,9777,9788,9802,9812,9823,9837,9848],{},[1127,9778,9779,9782,9785],{},[1145,9780,9781],{},"Web framework",[1145,9783,9784],{},"FastAPI on Uvicorn under Gunicorn",[1145,9786,9787],{},"Native async, Pydantic validation, OpenAPI for free, no monkey-patching",[1127,9789,9790,9793,9799],{},[1145,9791,9792],{},"JSON encoder",[1145,9794,9795,9796],{},"orjson via ",[32,9797,9798],{},"ORJSONResponse",[1145,9800,9801],{},"3 to 10x faster, returns bytes, handles numpy and datetime cleanly",[1127,9803,9804,9807,9809],{},[1145,9805,9806],{},"Package \u002F env manager",[1145,9808,9050],{},[1145,9810,9811],{},"Fast, single Dockerfile, first-class lockfile, no custom channels",[1127,9813,9814,9817,9820],{},[1145,9815,9816],{},"Async runtime",[1145,9818,9819],{},"anyio (via FastAPI)",[1145,9821,9822],{},"Bounded thread pool, structured concurrency, what FastAPI uses anyway",[1127,9824,9825,9828,9834],{},[1145,9826,9827],{},"Inference call style",[1145,9829,9830,8834,9832],{},[32,9831,8833],{},[32,9833,8837],{},[1145,9835,9836],{},"Explicit offload boundary, composes with async middleware",[1127,9838,9839,9842,9845],{},[1145,9840,9841],{},"Model loading in k8s",[1145,9843,9844],{},"initContainer plus PVC plus emptyDir",[1145,9846,9847],{},"Decouples model lifecycle from image lifecycle, smaller images",[1127,9849,9850,9853,9856],{},[1145,9851,9852],{},"Model update strategy",[1145,9854,9855],{},"Redeploy, not hot reload",[1145,9857,9858],{},"Reproducibility, atomic rollout, fewer failure modes",[2563,9860,9861],{},"html pre.shiki code .s5ixo, html code.shiki .s5ixo{--shiki-default:#383A42;--shiki-dark:#ABB2BF}html pre.shiki code .sknuh, html code.shiki .sknuh{--shiki-default:#383A42;--shiki-dark:#56B6C2}html pre.shiki code .slOjB, html code.shiki .slOjB{--shiki-default:#383A42;--shiki-dark:#61AFEF}html pre.shiki code .sp7wS, html code.shiki .sp7wS{--shiki-default:#986801;--shiki-default-font-style:inherit;--shiki-dark:#E06C75;--shiki-dark-font-style:italic}html pre.shiki code .sYebD, html code.shiki .sYebD{--shiki-default:#383A42;--shiki-dark:#D19A66}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sAdtL, html code.shiki .sAdtL{--shiki-default:#4078F2;--shiki-dark:#61AFEF}html pre.shiki code .siaei, html code.shiki .siaei{--shiki-default:#4078F2;--shiki-dark:#ABB2BF}html pre.shiki code .sDhpE, html code.shiki .sDhpE{--shiki-default:#50A14F;--shiki-dark:#98C379}html pre.shiki code .sLKXg, html code.shiki .sLKXg{--shiki-default:#A626A4;--shiki-dark:#C678DD}html pre.shiki code .so_Uh, html code.shiki .so_Uh{--shiki-default:#986801;--shiki-default-font-style:inherit;--shiki-dark:#D19A66;--shiki-dark-font-style:italic}html pre.shiki code .sxymB, html code.shiki .sxymB{--shiki-default:#986801;--shiki-dark:#ABB2BF}html pre.shiki code .sW2Sy, html code.shiki .sW2Sy{--shiki-default:#A0A1A7;--shiki-default-font-style:italic;--shiki-dark:#7F848E;--shiki-dark-font-style:italic}",{"title":59,"searchDepth":115,"depth":115,"links":9863},[9864,9865,9866,9867,9868,9869,9871,9872,9873],{"id":8736,"depth":115,"text":8737},{"id":8853,"depth":115,"text":8854},{"id":8944,"depth":115,"text":8945},{"id":9043,"depth":115,"text":9044},{"id":9147,"depth":115,"text":9148},{"id":9244,"depth":115,"text":9870},"5. async def plus run_in_threadpool (the importance of the offload)",{"id":9503,"depth":115,"text":9504},{"id":9652,"depth":115,"text":9653},{"id":9758,"depth":115,"text":9759},"2025-09-05","Technical design choices for ML Inference API",{},"\u002Fblog\u002Fbuilding-ml-inference-part-3",{"title":8731,"description":9875},"blog\u002Fbuilding-ml-inference-part-3",[2594],"2ze6JSE-YMpOJ1SLCozMf_nHeDWV4Xy8MX0cik5K3ZM",{"id":9883,"title":9884,"body":9885,"book":2585,"date":11347,"description":11348,"extension":2588,"meta":11349,"navigation":587,"path":4227,"seo":11350,"stem":11351,"tags":11352,"__hash__":11353},"blog\u002Fblog\u002Fbuilding-ml-inference-part-2.md","Building an ML Inference API, Part II",{"type":8,"value":9886,"toc":11326},[9887,9889,9897,9904,9908,9911,9915,9940,9964,9967,10101,10104,10227,10230,10234,10240,10573,10576,10580,10583,10589,10602,10609,10634,10644,10647,10651,10660,10667,10683,10847,10853,10859,10873,10956,10980,10988,10991,11086,11089,11098,11101,11196,11204,11210,11221,11232,11236,11239,11290,11293,11297,11308,11312,11323],[11,9888,14],{"id":13},[16,9890,9891,9892,9896],{},"Picking up where ",[20,9893,9895],{"href":9894},"\u002Fblog\u002Fbuilding-ml-inference-part-1","Part I"," left off: by 2023, all our services had been Dockerized and were running on Kubernetes. The VM era, and the constraint that we couldn't run Python web services, was over. The question became: now that we can run Python directly, what should an inference API look like?",[16,9898,9899,9900,9903],{},"This post covers the two phases I went through. The first was Flask + gunicorn + gevent + conda. The second was a rewrite to FastAPI + Uvicorn, which is what we run today. ",[20,9901,9902],{"href":9877},"Part III"," covers the technical decisions in detail; this post is how I got there.",[11,9905,9907],{"id":9906},"i-flask-gunicorn-gevent-conda","I. Flask + gunicorn + gevent + conda",[16,9909,9910],{},"The first version used the most familiar tools. Flask is what every Python developer knows. I honestly don't remember why I decided to use Gunicorn + Gevent, so most likely from Googling and reading other people's blogs. Conda was what the data science team used, so the pickled model files dropped in without numpy version conflicts.",[720,9912,9914],{"id":9913},"the-two-dockerfile-dance","The two-Dockerfile dance",[16,9916,9917,9918,508,9921,508,9924,508,9927,508,9930,508,9933,508,9936,9939],{},"The conda environment solve for our dependencies (",[32,9919,9920],{},"numpy",[32,9922,9923],{},"pandas",[32,9925,9926],{},"xgboost",[32,9928,9929],{},"catboost",[32,9931,9932],{},"flask",[32,9934,9935],{},"gunicorn",[32,9937,9938],{},"gevent",") routinely took five to seven minutes on every CI build. To keep build times down, the image was split in two:",[1789,9941,9942,9951],{},[173,9943,9944,9946,9947,9950],{},[32,9945,9070],{},": installs the conda environment from ",[32,9948,9949],{},"environment.yml",". Built once, pushed to the internal registry, rebuilt only when dependencies changed.",[173,9952,9953,9955,9956,9959,9960,9963],{},[32,9954,8766],{},": starts ",[32,9957,9958],{},"FROM \u003Cregistry>\u002Finference-env",", copies ",[32,9961,9962],{},"src\u002F",", sets the entrypoint. Rebuilt every CI run.",[16,9965,9966],{},"The base image:",[54,9968,9972],{"className":9969,"code":9970,"language":9971,"meta":59,"style":59},"language-dockerfile shiki shiki-themes one-light one-dark-pro","FROM python:3.11-slim AS builder\nARG MAMBA_VERSION=1.5.8\n\nRUN apt-get update && apt-get install -y --no-install-recommends curl bzip2 && \\\n    curl -Ls \"https:\u002F\u002Fmicro.mamba.pm\u002Fapi\u002Fmicromamba\u002Flinux-64\u002F${MAMBA_VERSION}\" \\\n        | tar -xvj -C \u002Ftmp bin\u002Fmicromamba && \\\n    apt-get purge -y --auto-remove curl bzip2 && \\\n    rm -rf \u002Fvar\u002Flib\u002Fapt\u002Flists\u002F*\n\nFROM python:3.11-slim\nENV MAMBA_ROOT_PREFIX=\"\u002Fopt\u002Fconda\"\nENV PATH=\"${MAMBA_ROOT_PREFIX}\u002Fbin:\u002Fusr\u002Flocal\u002Fbin:${PATH}\"\n\nCOPY --from=builder \u002Ftmp\u002Fbin\u002Fmicromamba \u002Fusr\u002Flocal\u002Fbin\u002Fmicromamba\nCOPY environment.yml .\n\nRUN micromamba create -n inference-env -f environment.yml -y && \\\n    micromamba clean -a -y\n","dockerfile",[32,9973,9974,9988,9996,10000,10008,10019,10024,10029,10034,10038,10045,10056,10066,10070,10078,10085,10089,10096],{"__ignoreMap":59},[63,9975,9976,9979,9982,9985],{"class":65,"line":66},[63,9977,9978],{"class":95},"FROM",[63,9980,9981],{"class":91}," python:3.11-slim ",[63,9983,9984],{"class":95},"AS",[63,9986,9987],{"class":91}," builder\n",[63,9989,9990,9993],{"class":65,"line":115},[63,9991,9992],{"class":95},"ARG",[63,9994,9995],{"class":91}," MAMBA_VERSION=1.5.8\n",[63,9997,9998],{"class":65,"line":121},[63,9999,588],{"emptyLinePlaceholder":587},[63,10001,10002,10005],{"class":65,"line":152},[63,10003,10004],{"class":95},"RUN",[63,10006,10007],{"class":91}," apt-get update && apt-get install -y --no-install-recommends curl bzip2 && \\\n",[63,10009,10010,10013,10016],{"class":65,"line":253},[63,10011,10012],{"class":91},"    curl -Ls ",[63,10014,10015],{"class":145},"\"https:\u002F\u002Fmicro.mamba.pm\u002Fapi\u002Fmicromamba\u002Flinux-64\u002F${MAMBA_VERSION}\"",[63,10017,10018],{"class":91}," \\\n",[63,10020,10021],{"class":65,"line":277},[63,10022,10023],{"class":91},"        | tar -xvj -C \u002Ftmp bin\u002Fmicromamba && \\\n",[63,10025,10026],{"class":65,"line":295},[63,10027,10028],{"class":91},"    apt-get purge -y --auto-remove curl bzip2 && \\\n",[63,10030,10031],{"class":65,"line":301},[63,10032,10033],{"class":91},"    rm -rf \u002Fvar\u002Flib\u002Fapt\u002Flists\u002F*\n",[63,10035,10036],{"class":65,"line":313},[63,10037,588],{"emptyLinePlaceholder":587},[63,10039,10040,10042],{"class":65,"line":318},[63,10041,9978],{"class":95},[63,10043,10044],{"class":91}," python:3.11-slim\n",[63,10046,10047,10050,10053],{"class":65,"line":340},[63,10048,10049],{"class":95},"ENV",[63,10051,10052],{"class":91}," MAMBA_ROOT_PREFIX=",[63,10054,10055],{"class":145},"\"\u002Fopt\u002Fconda\"\n",[63,10057,10058,10060,10063],{"class":65,"line":369},[63,10059,10049],{"class":95},[63,10061,10062],{"class":91}," PATH=",[63,10064,10065],{"class":145},"\"${MAMBA_ROOT_PREFIX}\u002Fbin:\u002Fusr\u002Flocal\u002Fbin:${PATH}\"\n",[63,10067,10068],{"class":65,"line":374},[63,10069,588],{"emptyLinePlaceholder":587},[63,10071,10072,10075],{"class":65,"line":387},[63,10073,10074],{"class":95},"COPY",[63,10076,10077],{"class":91}," --from=builder \u002Ftmp\u002Fbin\u002Fmicromamba \u002Fusr\u002Flocal\u002Fbin\u002Fmicromamba\n",[63,10079,10080,10082],{"class":65,"line":392},[63,10081,10074],{"class":95},[63,10083,10084],{"class":91}," environment.yml .\n",[63,10086,10087],{"class":65,"line":406},[63,10088,588],{"emptyLinePlaceholder":587},[63,10090,10091,10093],{"class":65,"line":2931},[63,10092,10004],{"class":95},[63,10094,10095],{"class":91}," micromamba create -n inference-env -f environment.yml -y && \\\n",[63,10097,10098],{"class":65,"line":2937},[63,10099,10100],{"class":91},"    micromamba clean -a -y\n",[16,10102,10103],{},"And the app image, the one CI rebuilt every commit:",[54,10105,10107],{"className":9969,"code":10106,"language":9971,"meta":59,"style":59},"FROM \u003Cregistry>\u002Finference-env\n\nWORKDIR \u002Fapp\nCOPY src\u002F* .\n\nENTRYPOINT [\"conda\", \"run\", \"--no-capture-output\", \"-n\", \"inference-env\", \\\n            \"gunicorn\", \"--worker-class\", \"gevent\", \"--log-level\", \"INFO\", \\\n            \"-w\", \"2\", \"-b\", \"0.0.0.0:80\", \"app:app\"]\n",[32,10108,10109,10116,10120,10128,10135,10139,10173,10200],{"__ignoreMap":59},[63,10110,10111,10113],{"class":65,"line":66},[63,10112,9978],{"class":95},[63,10114,10115],{"class":91}," \u003Cregistry>\u002Finference-env\n",[63,10117,10118],{"class":65,"line":115},[63,10119,588],{"emptyLinePlaceholder":587},[63,10121,10122,10125],{"class":65,"line":121},[63,10123,10124],{"class":95},"WORKDIR",[63,10126,10127],{"class":91}," \u002Fapp\n",[63,10129,10130,10132],{"class":65,"line":152},[63,10131,10074],{"class":95},[63,10133,10134],{"class":91}," src\u002F* .\n",[63,10136,10137],{"class":65,"line":253},[63,10138,588],{"emptyLinePlaceholder":587},[63,10140,10141,10144,10147,10150,10152,10155,10157,10160,10162,10165,10167,10170],{"class":65,"line":277},[63,10142,10143],{"class":95},"ENTRYPOINT",[63,10145,10146],{"class":91}," [",[63,10148,10149],{"class":145},"\"conda\"",[63,10151,508],{"class":91},[63,10153,10154],{"class":145},"\"run\"",[63,10156,508],{"class":91},[63,10158,10159],{"class":145},"\"--no-capture-output\"",[63,10161,508],{"class":91},[63,10163,10164],{"class":145},"\"-n\"",[63,10166,508],{"class":91},[63,10168,10169],{"class":145},"\"inference-env\"",[63,10171,10172],{"class":91},", \\\n",[63,10174,10175,10178,10180,10183,10185,10188,10190,10193,10195,10198],{"class":65,"line":295},[63,10176,10177],{"class":145},"            \"gunicorn\"",[63,10179,508],{"class":91},[63,10181,10182],{"class":145},"\"--worker-class\"",[63,10184,508],{"class":91},[63,10186,10187],{"class":145},"\"gevent\"",[63,10189,508],{"class":91},[63,10191,10192],{"class":145},"\"--log-level\"",[63,10194,508],{"class":91},[63,10196,10197],{"class":145},"\"INFO\"",[63,10199,10172],{"class":91},[63,10201,10202,10205,10207,10210,10212,10215,10217,10220,10222,10225],{"class":65,"line":301},[63,10203,10204],{"class":145},"            \"-w\"",[63,10206,508],{"class":91},[63,10208,10209],{"class":145},"\"2\"",[63,10211,508],{"class":91},[63,10213,10214],{"class":145},"\"-b\"",[63,10216,508],{"class":91},[63,10218,10219],{"class":145},"\"0.0.0.0:80\"",[63,10221,508],{"class":91},[63,10223,10224],{"class":145},"\"app:app\"",[63,10226,5150],{"class":91},[16,10228,10229],{},"Two gevent workers per pod. One environment, two images, one extra registry artifact to maintain.",[720,10231,10233],{"id":10232},"the-handler","The handler",[16,10235,10236,10237,10239],{},"A single ",[32,10238,9576],{},", one route per model:",[54,10241,10243],{"className":5066,"code":10242,"language":5068,"meta":59,"style":59},"import pickle\nimport pandas as pd\nfrom flask import Flask, request\n\napp = Flask(__name__)\n\n# Models loaded at import time as module globals\nscore_model = pickle.load(open(\"score_model.pkl\", \"rb\"))\nrisk_model = pickle.load(open(\"risk_model.pkl\", \"rb\"))\n\n\n@app.route(\"\u002Fhealth\")\ndef healthcheck():\n    return \"1\"\n\n\n@app.route(\"\u002Fpredict\u002Fscore\", methods=[\"POST\"])\ndef predict_score():\n    try:\n        body = request.json\n        features = pd.json_normalize(body[\"features\"])\n        result = score_model.predict(features)\n        return {\"score\": float(result[0])}\n    except Exception as e:\n        app.logger.error(repr(e))\n        return {\"score\": 0.0}, 400\n",[32,10244,10245,10252,10264,10277,10281,10297,10301,10306,10336,10362,10366,10370,10386,10396,10403,10407,10411,10438,10447,10454,10464,10485,10501,10524,10539,10555],{"__ignoreMap":59},[63,10246,10247,10249],{"class":65,"line":66},[63,10248,5075],{"class":439},[63,10250,10251],{"class":91}," pickle\n",[63,10253,10254,10256,10259,10261],{"class":65,"line":115},[63,10255,5075],{"class":439},[63,10257,10258],{"class":91}," pandas ",[63,10260,5081],{"class":439},[63,10262,10263],{"class":91}," pd\n",[63,10265,10266,10269,10272,10274],{"class":65,"line":121},[63,10267,10268],{"class":439},"from",[63,10270,10271],{"class":91}," flask ",[63,10273,5075],{"class":439},[63,10275,10276],{"class":91}," Flask, request\n",[63,10278,10279],{"class":65,"line":152},[63,10280,588],{"emptyLinePlaceholder":587},[63,10282,10283,10285,10287,10290,10292,10295],{"class":65,"line":253},[63,10284,9009],{"class":91},[63,10286,577],{"class":132},[63,10288,10289],{"class":5101}," Flask",[63,10291,142],{"class":91},[63,10293,10294],{"class":2976},"__name__",[63,10296,474],{"class":91},[63,10298,10299],{"class":65,"line":277},[63,10300,588],{"emptyLinePlaceholder":587},[63,10302,10303],{"class":65,"line":295},[63,10304,10305],{"class":2731},"# Models loaded at import time as module globals\n",[63,10307,10308,10311,10313,10316,10319,10321,10324,10326,10329,10331,10334],{"class":65,"line":301},[63,10309,10310],{"class":91},"score_model ",[63,10312,577],{"class":132},[63,10314,10315],{"class":91}," pickle.",[63,10317,10318],{"class":5101},"load",[63,10320,142],{"class":91},[63,10322,10323],{"class":5259},"open",[63,10325,142],{"class":91},[63,10327,10328],{"class":145},"\"score_model.pkl\"",[63,10330,508],{"class":91},[63,10332,10333],{"class":145},"\"rb\"",[63,10335,1106],{"class":91},[63,10337,10338,10341,10343,10345,10347,10349,10351,10353,10356,10358,10360],{"class":65,"line":313},[63,10339,10340],{"class":91},"risk_model ",[63,10342,577],{"class":132},[63,10344,10315],{"class":91},[63,10346,10318],{"class":5101},[63,10348,142],{"class":91},[63,10350,10323],{"class":5259},[63,10352,142],{"class":91},[63,10354,10355],{"class":145},"\"risk_model.pkl\"",[63,10357,508],{"class":91},[63,10359,10333],{"class":145},[63,10361,1106],{"class":91},[63,10363,10364],{"class":65,"line":318},[63,10365,588],{"emptyLinePlaceholder":587},[63,10367,10368],{"class":65,"line":340},[63,10369,588],{"emptyLinePlaceholder":587},[63,10371,10372,10374,10376,10379,10381,10384],{"class":65,"line":369},[63,10373,9268],{"class":95},[63,10375,92],{"class":2769},[63,10377,10378],{"class":95},"route",[63,10380,142],{"class":91},[63,10382,10383],{"class":145},"\"\u002Fhealth\"",[63,10385,474],{"class":91},[63,10387,10388,10390,10393],{"class":65,"line":374},[63,10389,5163],{"class":439},[63,10391,10392],{"class":95}," healthcheck",[63,10394,10395],{"class":91},"():\n",[63,10397,10398,10400],{"class":65,"line":387},[63,10399,1890],{"class":439},[63,10401,10402],{"class":145}," \"1\"\n",[63,10404,10405],{"class":65,"line":392},[63,10406,588],{"emptyLinePlaceholder":587},[63,10408,10409],{"class":65,"line":406},[63,10410,588],{"emptyLinePlaceholder":587},[63,10412,10413,10415,10417,10419,10421,10424,10426,10429,10431,10433,10436],{"class":65,"line":2931},[63,10414,9268],{"class":95},[63,10416,92],{"class":2769},[63,10418,10378],{"class":95},[63,10420,142],{"class":91},[63,10422,10423],{"class":145},"\"\u002Fpredict\u002Fscore\"",[63,10425,5175],{"class":91},[63,10427,10428],{"class":5107}," methods",[63,10430,577],{"class":132},[63,10432,4353],{"class":91},[63,10434,10435],{"class":145},"\"POST\"",[63,10437,5853],{"class":91},[63,10439,10440,10442,10445],{"class":65,"line":2937},[63,10441,5163],{"class":439},[63,10443,10444],{"class":95}," predict_score",[63,10446,10395],{"class":91},[63,10448,10449,10451],{"class":65,"line":2956},[63,10450,3466],{"class":439},[63,10452,10453],{"class":91},":\n",[63,10455,10456,10459,10461],{"class":65,"line":2961},[63,10457,10458],{"class":91},"        body ",[63,10460,577],{"class":132},[63,10462,10463],{"class":91}," request.json\n",[63,10465,10466,10469,10471,10474,10477,10480,10483],{"class":65,"line":3000},[63,10467,10468],{"class":91},"        features ",[63,10470,577],{"class":132},[63,10472,10473],{"class":91}," pd.",[63,10475,10476],{"class":5101},"json_normalize",[63,10478,10479],{"class":91},"(body[",[63,10481,10482],{"class":145},"\"features\"",[63,10484,5853],{"class":91},[63,10486,10487,10490,10492,10495,10498],{"class":65,"line":3018},[63,10488,10489],{"class":91},"        result ",[63,10491,577],{"class":132},[63,10493,10494],{"class":91}," score_model.",[63,10496,10497],{"class":5101},"predict",[63,10499,10500],{"class":91},"(features)\n",[63,10502,10503,10505,10508,10511,10513,10516,10519,10521],{"class":65,"line":3037},[63,10504,593],{"class":439},[63,10506,10507],{"class":91}," {",[63,10509,10510],{"class":145},"\"score\"",[63,10512,227],{"class":91},[63,10514,10515],{"class":5259},"float",[63,10517,10518],{"class":91},"(result[",[63,10520,867],{"class":289},[63,10522,10523],{"class":91},"])}\n",[63,10525,10526,10529,10533,10536],{"class":65,"line":3056},[63,10527,10528],{"class":439},"    except",[63,10530,10532],{"class":10531},"st7oF"," Exception",[63,10534,10535],{"class":439}," as",[63,10537,10538],{"class":91}," e:\n",[63,10540,10541,10544,10547,10549,10552],{"class":65,"line":5491},[63,10542,10543],{"class":91},"        app.logger.",[63,10545,10546],{"class":5101},"error",[63,10548,142],{"class":91},[63,10550,10551],{"class":5259},"repr",[63,10553,10554],{"class":91},"(e))\n",[63,10556,10557,10559,10561,10563,10565,10567,10570],{"class":65,"line":5515},[63,10558,593],{"class":439},[63,10560,10507],{"class":91},[63,10562,10510],{"class":145},[63,10564,227],{"class":91},[63,10566,4956],{"class":289},[63,10568,10569],{"class":91},"}, ",[63,10571,10572],{"class":289},"400\n",[16,10574,10575],{},"This worked. Models served traffic.",[720,10577,10579],{"id":10578},"where-it-started-to-fall-apart","Where it started to fall apart",[16,10581,10582],{},"Three things became obvious over time.",[16,10584,10585,10588],{},[2055,10586,10587],{},"Requests felt synchronous."," Under load, latencies spiked in patterns that didn't match the underlying inference cost. A handler that took 80 ms in isolation would take 600 ms when twenty other callers were hitting different endpoints on the same pod. p99 was much worse than the averages suggested.",[16,10590,10591,10592,10596,10597,3246,10599,10601],{},"The reason, which I only fully understood later, is in ",[20,10593,10595],{"href":10594},"\u002Fblog\u002Fbuilding-ml-inference-part-3#1-why-fastapi--uvicorn--gunicorn-not-flask--gevent","Part III, section 1",": gevent gets concurrency by monkey-patching the standard library's blocking I\u002FO calls so they cooperatively yield. That works for I\u002FO-bound workloads. It does not help when 95% of the time is spent inside ",[32,10598,9920],{},[32,10600,9926],{}," C extensions that don't yield to gevent's scheduler. Each worker still ran one inference at a time, with everyone else queued behind it.",[16,10603,10604,10605,10608],{},"We tuned the only knob we had, ",[32,10606,10607],{},"-w"," (worker count). Going from 2 to 4 to 8 helped throughput but not tail latencies.",[16,10610,10611,4020,10614,10617,10618,10621,10622,10625,10626,10629,10630,10633],{},[2055,10612,10613],{},"No request validation.",[32,10615,10616],{},"body = request.json",", then ",[32,10619,10620],{},"body[\"features\"]",". If the caller forgot the key or sent the wrong shape, we got a ",[32,10623,10624],{},"KeyError"," in production logs and returned a 400 with no useful message. We added more ",[32,10627,10628],{},"try\u002Fexcept",". We sometimes bolted on ",[32,10631,10632],{},"pydantic"," as a separate step. Each model endpoint ended up with its own subtly different schema handling.",[16,10635,10636,10639,10640,10643],{},[2055,10637,10638],{},"No model binding."," Models were loaded at module import as global variables. If a ",[32,10641,10642],{},".pkl"," file was missing on disk, the import crashed and gunicorn restarted in a loop until alerts caught it. There was no catalog, no central record of what the service served, no clean way to add a new model without grepping the source. Ten models in, this was painful.",[16,10645,10646],{},"We shipped, but we knew we were working around the framework rather than with it.",[11,10648,10650],{"id":10649},"ii-fastapi-uvicorn-under-gunicorn","II. FastAPI + Uvicorn (under gunicorn)",[16,10652,10653,10654,10656,10657,10659],{},"By the time we were considering yet another round of ",[32,10655,10628],{}," improvements, FastAPI was the obvious next direction. Async-native, Pydantic for validation, OpenAPI auto-generated, no monkey-patching of the standard library. The \"why\" for each is in ",[20,10658,9902],{"href":9877},"; this section is how I got the implementation right, which took a few tries.",[720,10661,10663,10664,10666],{"id":10662},"first-cut-sync-def-handlers","First cut: sync ",[32,10665,5163],{}," handlers",[16,10668,10669,10670,8503,10673,10676,10677,10680,10681,429],{},"The initial port was almost mechanical. Replace ",[32,10671,10672],{},"Flask",[32,10674,10675],{},"FastAPI",", replace ",[32,10678,10679],{},"request.json"," with a Pydantic body model, keep handlers as plain ",[32,10682,5163],{},[54,10684,10686],{"className":5066,"code":10685,"language":5068,"meta":59,"style":59},"from fastapi import FastAPI\nfrom pydantic import BaseModel\n\napp = FastAPI()\n\n\nclass PredictRequest(BaseModel):\n    features: list[dict]\n\n\n@app.post(\"\u002Fpredict\u002Fscore\")\ndef predict_score(req: PredictRequest):\n    df = pd.DataFrame(req.features)\n    result = score_model.predict(df)\n    return {\"score\": float(result[0])}\n",[32,10687,10688,10700,10712,10716,10726,10730,10734,10751,10761,10765,10769,10783,10800,10815,10829],{"__ignoreMap":59},[63,10689,10690,10692,10695,10697],{"class":65,"line":66},[63,10691,10268],{"class":439},[63,10693,10694],{"class":91}," fastapi ",[63,10696,5075],{"class":439},[63,10698,10699],{"class":91}," FastAPI\n",[63,10701,10702,10704,10707,10709],{"class":65,"line":115},[63,10703,10268],{"class":439},[63,10705,10706],{"class":91}," pydantic ",[63,10708,5075],{"class":439},[63,10710,10711],{"class":91}," BaseModel\n",[63,10713,10714],{"class":65,"line":121},[63,10715,588],{"emptyLinePlaceholder":587},[63,10717,10718,10720,10722,10724],{"class":65,"line":152},[63,10719,9009],{"class":91},[63,10721,577],{"class":132},[63,10723,9014],{"class":5101},[63,10725,5131],{"class":91},[63,10727,10728],{"class":65,"line":253},[63,10729,588],{"emptyLinePlaceholder":587},[63,10731,10732],{"class":65,"line":277},[63,10733,588],{"emptyLinePlaceholder":587},[63,10735,10736,10739,10742,10745,10747,10749],{"class":65,"line":295},[63,10737,10738],{"class":439},"class",[63,10740,10741],{"class":102}," PredictRequest",[63,10743,142],{"class":10744},"sBTfl",[63,10746,8879],{"class":102},[63,10748,5359],{"class":10744},[63,10750,10453],{"class":91},[63,10752,10753,10756,10759],{"class":65,"line":301},[63,10754,10755],{"class":91},"    features: list[",[63,10757,10758],{"class":5259},"dict",[63,10760,5150],{"class":91},[63,10762,10763],{"class":65,"line":313},[63,10764,588],{"emptyLinePlaceholder":587},[63,10766,10767],{"class":65,"line":318},[63,10768,588],{"emptyLinePlaceholder":587},[63,10770,10771,10773,10775,10777,10779,10781],{"class":65,"line":340},[63,10772,9268],{"class":95},[63,10774,92],{"class":2769},[63,10776,9273],{"class":95},[63,10778,142],{"class":91},[63,10780,10423],{"class":145},[63,10782,474],{"class":91},[63,10784,10785,10787,10789,10791,10794,10796,10798],{"class":65,"line":369},[63,10786,5163],{"class":439},[63,10788,10444],{"class":95},[63,10790,142],{"class":91},[63,10792,10793],{"class":5171},"req",[63,10795,429],{"class":91},[63,10797,10741],{"class":9299},[63,10799,5185],{"class":91},[63,10801,10802,10805,10807,10809,10812],{"class":65,"line":374},[63,10803,10804],{"class":91},"    df ",[63,10806,577],{"class":132},[63,10808,10473],{"class":91},[63,10810,10811],{"class":5101},"DataFrame",[63,10813,10814],{"class":91},"(req.features)\n",[63,10816,10817,10820,10822,10824,10826],{"class":65,"line":387},[63,10818,10819],{"class":91},"    result ",[63,10821,577],{"class":132},[63,10823,10494],{"class":91},[63,10825,10497],{"class":5101},[63,10827,10828],{"class":91},"(df)\n",[63,10830,10831,10833,10835,10837,10839,10841,10843,10845],{"class":65,"line":392},[63,10832,1890],{"class":439},[63,10834,10507],{"class":91},[63,10836,10510],{"class":145},[63,10838,227],{"class":91},[63,10840,10515],{"class":5259},[63,10842,10518],{"class":91},[63,10844,867],{"class":289},[63,10846,10523],{"class":91},[16,10848,10849,10850,10852],{},"This worked. FastAPI auto-runs sync handlers in a threadpool. The Pydantic class alone was a big win: callers sending the wrong shape now got a 422 with a per-field error message before my code ran. The 2 AM ",[32,10851,10624],{},"s went away in the first deploy.",[720,10854,10856,10857],{"id":10855},"switching-to-async-def","Switching to ",[32,10858,8833],{},[16,10860,10861,10862,10864,10865,10868,10869,10872],{},"I converted handlers to ",[32,10863,8833],{}," shortly after. I don't remember the exact trigger. It was probably an ",[32,10866,10867],{},"@app.middleware(\"http\")"," for request timing, which composes more cleanly when handlers are async, or a point where we needed ",[32,10870,10871],{},"await request.body()"," for a custom payload. The change:",[54,10874,10876],{"className":5066,"code":10875,"language":5068,"meta":59,"style":59},"@app.post(\"\u002Fpredict\u002Fscore\")\nasync def predict_score(req: PredictRequest):\n    df = pd.DataFrame(req.features)\n    result = score_model.predict(df)   # blocking call, inside async handler\n    return {\"score\": float(result[0])}\n",[32,10877,10878,10892,10910,10922,10938],{"__ignoreMap":59},[63,10879,10880,10882,10884,10886,10888,10890],{"class":65,"line":66},[63,10881,9268],{"class":95},[63,10883,92],{"class":2769},[63,10885,9273],{"class":95},[63,10887,142],{"class":91},[63,10889,10423],{"class":145},[63,10891,474],{"class":91},[63,10893,10894,10896,10898,10900,10902,10904,10906,10908],{"class":65,"line":115},[63,10895,7863],{"class":439},[63,10897,9287],{"class":439},[63,10899,10444],{"class":95},[63,10901,142],{"class":91},[63,10903,10793],{"class":5171},[63,10905,429],{"class":91},[63,10907,10741],{"class":9299},[63,10909,5185],{"class":91},[63,10911,10912,10914,10916,10918,10920],{"class":65,"line":121},[63,10913,10804],{"class":91},[63,10915,577],{"class":132},[63,10917,10473],{"class":91},[63,10919,10811],{"class":5101},[63,10921,10814],{"class":91},[63,10923,10924,10926,10928,10930,10932,10935],{"class":65,"line":152},[63,10925,10819],{"class":91},[63,10927,577],{"class":132},[63,10929,10494],{"class":91},[63,10931,10497],{"class":5101},[63,10933,10934],{"class":91},"(df)   ",[63,10936,10937],{"class":2731},"# blocking call, inside async handler\n",[63,10939,10940,10942,10944,10946,10948,10950,10952,10954],{"class":65,"line":253},[63,10941,1890],{"class":439},[63,10943,10507],{"class":91},[63,10945,10510],{"class":145},[63,10947,227],{"class":91},[63,10949,10515],{"class":5259},[63,10951,10518],{"class":91},[63,10953,867],{"class":289},[63,10955,10523],{"class":91},[16,10957,10958,10959,10961,10962,10964,10965,10967,10968,10970,10971,10974,10975,10979],{},"Latencies got worse. Switching from ",[32,10960,5163],{}," to ",[32,10963,8833],{}," silently changes FastAPI's behaviour: with ",[32,10966,5163],{},", FastAPI runs the handler in a threadpool; with ",[32,10969,8833],{},", it expects you to be cooperative and runs the code directly on the event loop. My CPU-bound ",[32,10972,10973],{},"model.predict"," was now sitting on the loop, blocking it for the full inference duration. ",[20,10976,10978],{"href":10977},"\u002Fblog\u002Fbuilding-ml-inference-part-3#5-async-def-plus-run_in_threadpool-the-importance-of-the-offload","Part III, section 5"," has the full pitfall.",[720,10981,10983,10617,10986],{"id":10982},"asyncioto_thread-then-run_in_threadpool",[32,10984,10985],{},"asyncio.to_thread",[32,10987,8837],{},[16,10989,10990],{},"The call needed to be off the event loop. The first thing I reached for was the standard library:",[54,10992,10994],{"className":5066,"code":10993,"language":5068,"meta":59,"style":59},"import asyncio\n\n@app.post(\"\u002Fpredict\u002Fscore\")\nasync def predict_score(req: PredictRequest):\n    df = pd.DataFrame(req.features)\n    result = await asyncio.to_thread(score_model.predict, df)\n    return {\"score\": float(result[0])}\n",[32,10995,10996,11003,11007,11021,11039,11051,11068],{"__ignoreMap":59},[63,10997,10998,11000],{"class":65,"line":66},[63,10999,5075],{"class":439},[63,11001,11002],{"class":91}," asyncio\n",[63,11004,11005],{"class":65,"line":115},[63,11006,588],{"emptyLinePlaceholder":587},[63,11008,11009,11011,11013,11015,11017,11019],{"class":65,"line":121},[63,11010,9268],{"class":95},[63,11012,92],{"class":2769},[63,11014,9273],{"class":95},[63,11016,142],{"class":91},[63,11018,10423],{"class":145},[63,11020,474],{"class":91},[63,11022,11023,11025,11027,11029,11031,11033,11035,11037],{"class":65,"line":152},[63,11024,7863],{"class":439},[63,11026,9287],{"class":439},[63,11028,10444],{"class":95},[63,11030,142],{"class":91},[63,11032,10793],{"class":5171},[63,11034,429],{"class":91},[63,11036,10741],{"class":9299},[63,11038,5185],{"class":91},[63,11040,11041,11043,11045,11047,11049],{"class":65,"line":253},[63,11042,10804],{"class":91},[63,11044,577],{"class":132},[63,11046,10473],{"class":91},[63,11048,10811],{"class":5101},[63,11050,10814],{"class":91},[63,11052,11053,11055,11057,11059,11062,11065],{"class":65,"line":277},[63,11054,10819],{"class":91},[63,11056,577],{"class":132},[63,11058,9389],{"class":439},[63,11060,11061],{"class":91}," asyncio.",[63,11063,11064],{"class":5101},"to_thread",[63,11066,11067],{"class":91},"(score_model.predict, df)\n",[63,11069,11070,11072,11074,11076,11078,11080,11082,11084],{"class":65,"line":295},[63,11071,1890],{"class":439},[63,11073,10507],{"class":91},[63,11075,10510],{"class":145},[63,11077,227],{"class":91},[63,11079,10515],{"class":5259},[63,11081,10518],{"class":91},[63,11083,867],{"class":289},[63,11085,10523],{"class":91},[16,11087,11088],{},"Latencies recovered. I shipped it.",[16,11090,11091,11092,11094,11095,11097],{},"A few weeks later, while reading FastAPI's source for an unrelated reason, I noticed something I had missed: FastAPI doesn't use asyncio's threadpool. It uses anyio's, via ",[32,11093,9193],{},". They are different pools. Anyio's has a global concurrency limiter (default 40 threads); asyncio's ",[32,11096,11064],{}," does not, every call goes into the default executor with no cooperative cap.",[16,11099,11100],{},"For our load, that mostly didn't matter. But our inference workload was bypassing the bound the rest of the framework respected, with no single place to tune it. The fix was a one-line change:",[54,11102,11104],{"className":5066,"code":11103,"language":5068,"meta":59,"style":59},"from fastapi.concurrency import run_in_threadpool\n\n@app.post(\"\u002Fpredict\u002Fscore\")\nasync def predict_score(req: PredictRequest):\n    df = pd.DataFrame(req.features)\n    result = await run_in_threadpool(score_model.predict, df)\n    return {\"score\": float(result[0])}\n",[32,11105,11106,11118,11122,11136,11154,11166,11178],{"__ignoreMap":59},[63,11107,11108,11110,11113,11115],{"class":65,"line":66},[63,11109,10268],{"class":439},[63,11111,11112],{"class":91}," fastapi.concurrency ",[63,11114,5075],{"class":439},[63,11116,11117],{"class":91}," run_in_threadpool\n",[63,11119,11120],{"class":65,"line":115},[63,11121,588],{"emptyLinePlaceholder":587},[63,11123,11124,11126,11128,11130,11132,11134],{"class":65,"line":121},[63,11125,9268],{"class":95},[63,11127,92],{"class":2769},[63,11129,9273],{"class":95},[63,11131,142],{"class":91},[63,11133,10423],{"class":145},[63,11135,474],{"class":91},[63,11137,11138,11140,11142,11144,11146,11148,11150,11152],{"class":65,"line":152},[63,11139,7863],{"class":439},[63,11141,9287],{"class":439},[63,11143,10444],{"class":95},[63,11145,142],{"class":91},[63,11147,10793],{"class":5171},[63,11149,429],{"class":91},[63,11151,10741],{"class":9299},[63,11153,5185],{"class":91},[63,11155,11156,11158,11160,11162,11164],{"class":65,"line":253},[63,11157,10804],{"class":91},[63,11159,577],{"class":132},[63,11161,10473],{"class":91},[63,11163,10811],{"class":5101},[63,11165,10814],{"class":91},[63,11167,11168,11170,11172,11174,11176],{"class":65,"line":277},[63,11169,10819],{"class":91},[63,11171,577],{"class":132},[63,11173,9389],{"class":439},[63,11175,9392],{"class":5101},[63,11177,11067],{"class":91},[63,11179,11180,11182,11184,11186,11188,11190,11192,11194],{"class":65,"line":295},[63,11181,1890],{"class":439},[63,11183,10507],{"class":91},[63,11185,10510],{"class":145},[63,11187,227],{"class":91},[63,11189,10515],{"class":5259},[63,11191,10518],{"class":91},[63,11193,867],{"class":289},[63,11195,10523],{"class":91},[16,11197,11198,11199,11203],{},"That is the version still running in production. ",[20,11200,11202],{"href":11201},"\u002Fblog\u002Fbuilding-ml-inference-part-3#4-anyio-vs-asyncio","Part III, section 4"," covers the anyio vs asyncio rationale.",[720,11205,11207,11208],{"id":11206},"why-we-kept-async-def","Why we kept ",[32,11209,8833],{},[16,11211,11212,11213,8834,11215,11217,11218,11220],{},"With ",[32,11214,8833],{},[32,11216,8837],{},", the handler does roughly the same thing FastAPI would have done automatically with a sync ",[32,11219,5163],{},". Two reasons we kept the explicit version, both expanded in Part III:",[1789,11222,11223,11226],{},[173,11224,11225],{},"The offload boundary is visible in the code. A reader sees which call is blocking and which is async.",[173,11227,11228,11229,11231],{},"The access-log middleware is async. The ",[32,11230,9572],{}," startup hook is async. Any future cache lookup or remote call would also be async. Starting that way avoids the conversion later.",[720,11233,11235],{"id":11234},"process-model","Process model",[16,11237,11238],{},"Uvicorn alone is fine in dev. In production we run it under gunicorn:",[54,11240,11242],{"className":9969,"code":11241,"language":9971,"meta":59,"style":59},"ENTRYPOINT [\"gunicorn\", \"-w\", \"1\", \"-k\", \"uvicorn.workers.UvicornWorker\", \\\n            \"-b\", \"0.0.0.0:80\", \"app:app\"]\n",[32,11243,11244,11275],{"__ignoreMap":59},[63,11245,11246,11248,11250,11253,11255,11258,11260,11263,11265,11268,11270,11273],{"class":65,"line":66},[63,11247,10143],{"class":95},[63,11249,10146],{"class":91},[63,11251,11252],{"class":145},"\"gunicorn\"",[63,11254,508],{"class":91},[63,11256,11257],{"class":145},"\"-w\"",[63,11259,508],{"class":91},[63,11261,11262],{"class":145},"\"1\"",[63,11264,508],{"class":91},[63,11266,11267],{"class":145},"\"-k\"",[63,11269,508],{"class":91},[63,11271,11272],{"class":145},"\"uvicorn.workers.UvicornWorker\"",[63,11274,10172],{"class":91},[63,11276,11277,11280,11282,11284,11286,11288],{"class":65,"line":115},[63,11278,11279],{"class":145},"            \"-b\"",[63,11281,508],{"class":91},[63,11283,10219],{"class":145},[63,11285,508],{"class":91},[63,11287,10224],{"class":145},[63,11289,5150],{"class":91},[16,11291,11292],{},"One Uvicorn worker per pod, scaled horizontally by Kubernetes. Gunicorn handles process supervision, graceful shutdown, and the listening socket; Uvicorn is the ASGI runtime inside the worker. We prefer more pods over more workers per pod, since it keeps the unit of failure smaller and concentrates scheduling decisions in the kubelet.",[720,11294,11296],{"id":11295},"conda-to-uv","Conda to uv",[16,11298,11299,11300,11302,11303,11307],{},"The last change is not strictly part of the FastAPI rewrite, but it lands in the same era. The two-Dockerfile dance, plus the registry artifact to maintain, was always a smell. uv installs the dependency set in seconds rather than minutes. The production image is back to a single ",[32,11301,8766],{},". CI is faster. ",[20,11304,11306],{"href":11305},"\u002Fblog\u002Fbuilding-ml-inference-part-3#3-why-uv-over-conda","Part III, section 3"," covers the why.",[11,11309,11311],{"id":11310},"closing-thoughts","Closing thoughts",[16,11313,11314,11315,8834,11317,11319,11320,11322],{},"The shape we landed on (",[32,11316,8833],{},[32,11318,8837],{},", Pydantic for the request boundary, FastAPI under gunicorn) is the shape the template in ",[20,11321,9902],{"href":9877}," ships with. Part III explains why each choice is the default. This post is how I got there. If you are starting an inference service today, skip the journey and use the template.",[2563,11324,11325],{},"html pre.shiki code .sAdtL, html code.shiki .sAdtL{--shiki-default:#4078F2;--shiki-dark:#61AFEF}html pre.shiki code .s5ixo, html code.shiki .s5ixo{--shiki-default:#383A42;--shiki-dark:#ABB2BF}html pre.shiki code .sDhpE, html code.shiki .sDhpE{--shiki-default:#50A14F;--shiki-dark:#98C379}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sLKXg, html code.shiki .sLKXg{--shiki-default:#A626A4;--shiki-dark:#C678DD}html pre.shiki code .sknuh, html code.shiki .sknuh{--shiki-default:#383A42;--shiki-dark:#56B6C2}html pre.shiki code .slOjB, html code.shiki .slOjB{--shiki-default:#383A42;--shiki-dark:#61AFEF}html pre.shiki code .sJa8x, html code.shiki .sJa8x{--shiki-default:#E45649;--shiki-dark:#E06C75}html pre.shiki code .sW2Sy, html code.shiki .sW2Sy{--shiki-default:#A0A1A7;--shiki-default-font-style:italic;--shiki-dark:#7F848E;--shiki-dark-font-style:italic}html pre.shiki code .s_Sar, html code.shiki .s_Sar{--shiki-default:#0184BC;--shiki-dark:#56B6C2}html pre.shiki code .siaei, html code.shiki .siaei{--shiki-default:#4078F2;--shiki-dark:#ABB2BF}html pre.shiki code .sp7wS, html code.shiki .sp7wS{--shiki-default:#986801;--shiki-default-font-style:inherit;--shiki-dark:#E06C75;--shiki-dark-font-style:italic}html pre.shiki code .sAGMh, html code.shiki .sAGMh{--shiki-default:#986801;--shiki-dark:#D19A66}html pre.shiki code .st7oF, html code.shiki .st7oF{--shiki-default:#0184BC;--shiki-dark:#ABB2BF}html pre.shiki code .sC09Y, html code.shiki .sC09Y{--shiki-default:#C18401;--shiki-dark:#E5C07B}html pre.shiki code .sBTfl, html code.shiki .sBTfl{--shiki-default:#C18401;--shiki-dark:#ABB2BF}html pre.shiki code .so_Uh, html code.shiki .so_Uh{--shiki-default:#986801;--shiki-default-font-style:inherit;--shiki-dark:#D19A66;--shiki-dark-font-style:italic}html pre.shiki code .sxymB, html code.shiki .sxymB{--shiki-default:#986801;--shiki-dark:#ABB2BF}",{"title":59,"searchDepth":115,"depth":115,"links":11327},[11328,11329,11334,11346],{"id":13,"depth":115,"text":14},{"id":9906,"depth":115,"text":9907,"children":11330},[11331,11332,11333],{"id":9913,"depth":121,"text":9914},{"id":10232,"depth":121,"text":10233},{"id":10578,"depth":121,"text":10579},{"id":10649,"depth":115,"text":10650,"children":11335},[11336,11338,11340,11342,11344,11345],{"id":10662,"depth":121,"text":11337},"First cut: sync def handlers",{"id":10855,"depth":121,"text":11339},"Switching to async def",{"id":10982,"depth":121,"text":11341},"asyncio.to_thread, then run_in_threadpool",{"id":11206,"depth":121,"text":11343},"Why we kept async def",{"id":11234,"depth":121,"text":11235},{"id":11295,"depth":121,"text":11296},{"id":11310,"depth":115,"text":11311},"2025-09-04","Building Scalable Python Web API for ML Inference",{},{"title":9884,"description":11348},"blog\u002Fbuilding-ml-inference-part-2",[2594],"QjFs8sQmcjpTZnpc6uD-LMBlIfSm_yQ7lG_GtUGtAN8",{"id":11355,"title":11356,"body":11357,"book":2585,"date":13140,"description":13141,"extension":2588,"meta":13142,"navigation":587,"path":9894,"seo":13143,"stem":13144,"tags":13145,"__hash__":13146},"blog\u002Fblog\u002Fbuilding-ml-inference-part-1.md","Building an ML Inference API, Part I",{"type":8,"value":11358,"toc":13129},[11359,11361,11364,11378,11381,11384,11387,11391,11394,11397,11408,11412,11419,11693,11697,11704,11751,11758,11900,11903,12217,12224,12228,12231,12234,12237,12240,12243,12246,12272,12275,12281,12284,12287,12568,12571,12574,13093,13102,13105,13108,13121,13123,13126],[11,11360,14],{"id":13},[16,11362,11363],{},"Back in early 2020, I was working on a Software Engineering (SWE) team whose stack was .NET (.NET Core 2.2, to be precise) and SQL Server, with all projects deployed to Windows Server VMs. I worked closely with a Data Science team that operated strictly in Python. Even though the two teams had independent workstreams, my responsibility often involved integrating Machine Learning (ML) models into production. The workflow typically looked like this:",[170,11365,11366,11369,11372,11375],{},[173,11367,11368],{},"A .NET project reads from a database or data source for input.",[173,11370,11371],{},"It formats the input and passes it to the Python-trained ML model.",[173,11373,11374],{},"The model runs a prediction on the input.",[173,11376,11377],{},"The result is passed back to the .NET project to be used downstream.",[16,11379,11380],{},"One of the main challenges was making this process seamless, fast, and scalable (up to a few hundred calls per second), all while ensuring Python-trained models worked effectively within a .NET environment.",[16,11382,11383],{},"This problem stuck with me. It remained relevant even in early 2023, when most projects in the company were fully Dockerized and managed by Kubernetes (k8s), no longer running on Windows Server VMs.",[16,11385,11386],{},"I want to document the problems and the solutions I devised, given the unique constraints I had to work with.",[11,11388,11390],{"id":11389},"i-when-net-was-running-in-windows-vms","I. When .NET was running in Windows VMs",[16,11392,11393],{},"At the time, for \"security\" and \"we-don't-have-enough-non-C#-experts\" reasons, building Python projects on these VMs was not permitted.",[16,11395,11396],{},"The task was to run an XGBoost model based on specific conditions. The call frequency was low (a few calls every 15 minutes), so performance wasn't the top priority; getting it to run at all was. The simplest solution we came up with was to drop a Python script onto the same VM and invoke it as a subprocess from our .NET application.",[16,11398,11399,11400,11403,11404,11407],{},"While building Python ",[3240,11401,11402],{},"projects"," was strictly forbidden, apparently no one thought to ban installing the Python runtime and ",[32,11405,11406],{},"pip"," packages. We took the win.",[720,11409,11411],{"id":11410},"the-python-script","The Python script",[16,11413,11414,11415,11418],{},"Save this as ",[32,11416,11417],{},"predict.py",". It loads the model, accepts a JSON string as an argument, and prints the result to standard output.",[54,11420,11422],{"className":5066,"code":11421,"language":5068,"meta":59,"style":59},"import sys\nimport json\nimport xgboost as xgb\nimport numpy as np\n\n# 1. Load the model (done once when the script starts)\nmodel = xgb.XGBClassifier()\nmodel.load_model(\"xgboost_model.json\")\n\ndef run_inference(input_features):\n    # Convert input list to numpy array and reshape for single prediction\n    data = np.array(input_features).reshape(1, -1)\n    prediction = model.predict(data)\n    return float(prediction[0])\n\nif __name__ == \"__main__\":\n    try:\n        # 2. Read the argument passed from C#\n        input_json = sys.argv[1]\n        features = json.loads(input_json)\n\n        # 3. Run prediction and print result\n        result = run_inference(features)\n        print(result)\n    except Exception as e:\n        sys.stderr.write(str(e))\n",[32,11423,11424,11431,11438,11450,11462,11466,11471,11486,11501,11505,11519,11524,11555,11570,11584,11588,11602,11608,11613,11627,11642,11646,11651,11661,11669,11679],{"__ignoreMap":59},[63,11425,11426,11428],{"class":65,"line":66},[63,11427,5075],{"class":439},[63,11429,11430],{"class":91}," sys\n",[63,11432,11433,11435],{"class":65,"line":115},[63,11434,5075],{"class":439},[63,11436,11437],{"class":91}," json\n",[63,11439,11440,11442,11445,11447],{"class":65,"line":121},[63,11441,5075],{"class":439},[63,11443,11444],{"class":91}," xgboost ",[63,11446,5081],{"class":439},[63,11448,11449],{"class":91}," xgb\n",[63,11451,11452,11454,11457,11459],{"class":65,"line":152},[63,11453,5075],{"class":439},[63,11455,11456],{"class":91}," numpy ",[63,11458,5081],{"class":439},[63,11460,11461],{"class":91}," np\n",[63,11463,11464],{"class":65,"line":253},[63,11465,588],{"emptyLinePlaceholder":587},[63,11467,11468],{"class":65,"line":277},[63,11469,11470],{"class":2731},"# 1. Load the model (done once when the script starts)\n",[63,11472,11473,11476,11478,11481,11484],{"class":65,"line":295},[63,11474,11475],{"class":91},"model ",[63,11477,577],{"class":132},[63,11479,11480],{"class":91}," xgb.",[63,11482,11483],{"class":5101},"XGBClassifier",[63,11485,5131],{"class":91},[63,11487,11488,11491,11494,11496,11499],{"class":65,"line":301},[63,11489,11490],{"class":91},"model.",[63,11492,11493],{"class":5101},"load_model",[63,11495,142],{"class":91},[63,11497,11498],{"class":145},"\"xgboost_model.json\"",[63,11500,474],{"class":91},[63,11502,11503],{"class":65,"line":313},[63,11504,588],{"emptyLinePlaceholder":587},[63,11506,11507,11509,11512,11514,11517],{"class":65,"line":318},[63,11508,5163],{"class":439},[63,11510,11511],{"class":95}," run_inference",[63,11513,142],{"class":91},[63,11515,11516],{"class":5171},"input_features",[63,11518,5185],{"class":91},[63,11520,11521],{"class":65,"line":340},[63,11522,11523],{"class":2731},"    # Convert input list to numpy array and reshape for single prediction\n",[63,11525,11526,11529,11531,11534,11537,11540,11543,11545,11547,11549,11551,11553],{"class":65,"line":369},[63,11527,11528],{"class":91},"    data ",[63,11530,577],{"class":132},[63,11532,11533],{"class":91}," np.",[63,11535,11536],{"class":5101},"array",[63,11538,11539],{"class":91},"(input_features).",[63,11541,11542],{"class":5101},"reshape",[63,11544,142],{"class":91},[63,11546,880],{"class":289},[63,11548,508],{"class":91},[63,11550,4932],{"class":132},[63,11552,880],{"class":289},[63,11554,474],{"class":91},[63,11556,11557,11560,11562,11565,11567],{"class":65,"line":374},[63,11558,11559],{"class":91},"    prediction ",[63,11561,577],{"class":132},[63,11563,11564],{"class":91}," model.",[63,11566,10497],{"class":5101},[63,11568,11569],{"class":91},"(data)\n",[63,11571,11572,11574,11577,11580,11582],{"class":65,"line":387},[63,11573,1890],{"class":439},[63,11575,11576],{"class":5259}," float",[63,11578,11579],{"class":91},"(prediction[",[63,11581,867],{"class":289},[63,11583,5853],{"class":91},[63,11585,11586],{"class":65,"line":392},[63,11587,588],{"emptyLinePlaceholder":587},[63,11589,11590,11592,11595,11597,11600],{"class":65,"line":406},[63,11591,4345],{"class":439},[63,11593,11594],{"class":2976}," __name__",[63,11596,8032],{"class":132},[63,11598,11599],{"class":145}," \"__main__\"",[63,11601,10453],{"class":91},[63,11603,11604,11606],{"class":65,"line":2931},[63,11605,3466],{"class":439},[63,11607,10453],{"class":91},[63,11609,11610],{"class":65,"line":2937},[63,11611,11612],{"class":2731},"        # 2. Read the argument passed from C#\n",[63,11614,11615,11618,11620,11623,11625],{"class":65,"line":2956},[63,11616,11617],{"class":91},"        input_json ",[63,11619,577],{"class":132},[63,11621,11622],{"class":91}," sys.argv[",[63,11624,880],{"class":289},[63,11626,5150],{"class":91},[63,11628,11629,11631,11633,11636,11639],{"class":65,"line":2961},[63,11630,10468],{"class":91},[63,11632,577],{"class":132},[63,11634,11635],{"class":91}," json.",[63,11637,11638],{"class":5101},"loads",[63,11640,11641],{"class":91},"(input_json)\n",[63,11643,11644],{"class":65,"line":3000},[63,11645,588],{"emptyLinePlaceholder":587},[63,11647,11648],{"class":65,"line":3018},[63,11649,11650],{"class":2731},"        # 3. Run prediction and print result\n",[63,11652,11653,11655,11657,11659],{"class":65,"line":3037},[63,11654,10489],{"class":91},[63,11656,577],{"class":132},[63,11658,11511],{"class":5101},[63,11660,10500],{"class":91},[63,11662,11663,11666],{"class":65,"line":3056},[63,11664,11665],{"class":5259},"        print",[63,11667,11668],{"class":91},"(result)\n",[63,11670,11671,11673,11675,11677],{"class":65,"line":5491},[63,11672,10528],{"class":439},[63,11674,10532],{"class":10531},[63,11676,10535],{"class":439},[63,11678,10538],{"class":91},[63,11680,11681,11684,11687,11689,11691],{"class":65,"line":5515},[63,11682,11683],{"class":91},"        sys.stderr.",[63,11685,11686],{"class":5101},"write",[63,11688,142],{"class":91},[63,11690,8975],{"class":5259},[63,11692,10554],{"class":91},[720,11694,11696],{"id":11695},"the-c-implementation","The C# implementation",[16,11698,11699,11700,11703],{},"The ",[32,11701,11702],{},"jsonString"," here is the feature array. Pull it from a DTO or wherever the input lives, then form the string.",[54,11705,11707],{"className":78,"code":11706,"language":80,"meta":59,"style":59},"var jsonString = \"[0.52,0.1,0.8,1.5,2.2,1.4]\";\nvar pythonPath = @\"python\";\nvar scriptPath = @\"predict.py\";\n",[32,11708,11709,11723,11737],{"__ignoreMap":59},[63,11710,11711,11713,11716,11718,11721],{"class":65,"line":66},[63,11712,2067],{"class":439},[63,11714,11715],{"class":528}," jsonString",[63,11717,133],{"class":132},[63,11719,11720],{"class":145}," \"[0.52,0.1,0.8,1.5,2.2,1.4]\"",[63,11722,274],{"class":91},[63,11724,11725,11727,11730,11732,11735],{"class":65,"line":115},[63,11726,2067],{"class":439},[63,11728,11729],{"class":528}," pythonPath",[63,11731,133],{"class":132},[63,11733,11734],{"class":145}," @\"python\"",[63,11736,274],{"class":91},[63,11738,11739,11741,11744,11746,11749],{"class":65,"line":121},[63,11740,2067],{"class":439},[63,11742,11743],{"class":528}," scriptPath",[63,11745,133],{"class":132},[63,11747,11748],{"class":145}," @\"predict.py\"",[63,11750,274],{"class":91},[16,11752,11753,11754,11757],{},"Next, we configure ",[32,11755,11756],{},"ProcessStartInfo",". This is the critical part: we tell Windows to suppress the black command prompt window and redirect the output so we can read it in C#.",[54,11759,11761],{"className":78,"code":11760,"language":80,"meta":59,"style":59},"var processStartInfo = new ProcessStartInfo\n{\n    FileName = pythonPath,\n    \u002F\u002F Wrap the JSON in quotes to handle spaces safely\n    Arguments = $\"{scriptPath} \\\"{jsonString}\\\"\",\n\n    \u002F\u002F Crucial settings:\n    RedirectStandardOutput = true, \u002F\u002F Capture the print() result\n    RedirectStandardError = true,  \u002F\u002F Capture errors\n    UseShellExecute = false,       \u002F\u002F Required to redirect streams\n    CreateNoWindow = true          \u002F\u002F Don't pop up a black CMD window\n};\n",[32,11762,11763,11777,11781,11792,11797,11830,11834,11839,11853,11868,11883,11895],{"__ignoreMap":59},[63,11764,11765,11767,11770,11772,11774],{"class":65,"line":66},[63,11766,2067],{"class":439},[63,11768,11769],{"class":528}," processStartInfo",[63,11771,133],{"class":132},[63,11773,136],{"class":91},[63,11775,11776],{"class":102},"ProcessStartInfo\n",[63,11778,11779],{"class":65,"line":115},[63,11780,118],{"class":91},[63,11782,11783,11786,11788,11790],{"class":65,"line":121},[63,11784,11785],{"class":528},"    FileName",[63,11787,133],{"class":132},[63,11789,11729],{"class":528},[63,11791,233],{"class":91},[63,11793,11794],{"class":65,"line":152},[63,11795,11796],{"class":2731},"    \u002F\u002F Wrap the JSON in quotes to handle spaces safely\n",[63,11798,11799,11802,11804,11807,11809,11812,11814,11817,11819,11821,11823,11826,11828],{"class":65,"line":253},[63,11800,11801],{"class":528},"    Arguments",[63,11803,133],{"class":132},[63,11805,11806],{"class":145}," $\"",[63,11808,5231],{"class":6466},[63,11810,11811],{"class":528},"scriptPath",[63,11813,5237],{"class":6466},[63,11815,11816],{"class":5259}," \\\"",[63,11818,5231],{"class":6466},[63,11820,11702],{"class":528},[63,11822,5237],{"class":6466},[63,11824,11825],{"class":5259},"\\\"",[63,11827,5228],{"class":145},[63,11829,233],{"class":91},[63,11831,11832],{"class":65,"line":277},[63,11833,588],{"emptyLinePlaceholder":587},[63,11835,11836],{"class":65,"line":295},[63,11837,11838],{"class":2731},"    \u002F\u002F Crucial settings:\n",[63,11840,11841,11844,11846,11848,11850],{"class":65,"line":301},[63,11842,11843],{"class":528},"    RedirectStandardOutput",[63,11845,133],{"class":132},[63,11847,1515],{"class":289},[63,11849,508],{"class":91},[63,11851,11852],{"class":2731},"\u002F\u002F Capture the print() result\n",[63,11854,11855,11858,11860,11862,11865],{"class":65,"line":313},[63,11856,11857],{"class":528},"    RedirectStandardError",[63,11859,133],{"class":132},[63,11861,1515],{"class":289},[63,11863,11864],{"class":91},",  ",[63,11866,11867],{"class":2731},"\u002F\u002F Capture errors\n",[63,11869,11870,11873,11875,11877,11880],{"class":65,"line":318},[63,11871,11872],{"class":528},"    UseShellExecute",[63,11874,133],{"class":132},[63,11876,3493],{"class":289},[63,11878,11879],{"class":91},",       ",[63,11881,11882],{"class":2731},"\u002F\u002F Required to redirect streams\n",[63,11884,11885,11888,11890,11892],{"class":65,"line":340},[63,11886,11887],{"class":528},"    CreateNoWindow",[63,11889,133],{"class":132},[63,11891,1515],{"class":289},[63,11893,11894],{"class":2731},"          \u002F\u002F Don't pop up a black CMD window\n",[63,11896,11897],{"class":65,"line":369},[63,11898,11899],{"class":91},"};\n",[16,11901,11902],{},"Finally, we execute the process. This is a synchronous operation, meaning the C# app will wait for Python to finish before continuing.",[54,11904,11906],{"className":78,"code":11905,"language":80,"meta":59,"style":59},"Process process = null;\n\ntry\n{\n    process = new Process { StartInfo = processStartInfo };\n    process.Start();\n\n    \u002F\u002F Read the output synchronously (wait for Python to finish)\n    var stringResult = process.StandardOutput.ReadToEnd();\n    var errors = process.StandardError.ReadToEnd();\n\n    process.WaitForExit();\n\n    if (process.ExitCode == 0)\n    {\n        var prediction = double.Parse(stringResult);\n        \u002F\u002F Use prediction...\n    }\n    else\n    {\n        _logger.LogError($\"Python script failed: {errors}\");\n    }\n}\ncatch (Exception ex)\n{\n    _logger.LogError($\"C# execution failed: {ex.Message}\");\n}\nfinally\n{\n    process?.Dispose();\n}\n",[32,11907,11908,11922,11926,11931,11935,11958,11969,11973,11978,12001,12023,12027,12038,12042,12062,12066,12088,12093,12097,12101,12105,12131,12135,12139,12154,12158,12188,12192,12197,12201,12213],{"__ignoreMap":59},[63,11909,11910,11913,11916,11918,11920],{"class":65,"line":66},[63,11911,11912],{"class":102},"Process",[63,11914,11915],{"class":528}," process",[63,11917,133],{"class":132},[63,11919,3607],{"class":289},[63,11921,274],{"class":91},[63,11923,11924],{"class":65,"line":115},[63,11925,588],{"emptyLinePlaceholder":587},[63,11927,11928],{"class":65,"line":121},[63,11929,11930],{"class":439},"try\n",[63,11932,11933],{"class":65,"line":152},[63,11934,118],{"class":91},[63,11936,11937,11940,11942,11944,11946,11948,11951,11953,11955],{"class":65,"line":253},[63,11938,11939],{"class":528},"    process",[63,11941,133],{"class":132},[63,11943,136],{"class":91},[63,11945,11912],{"class":102},[63,11947,3486],{"class":91},[63,11949,11950],{"class":528},"StartInfo",[63,11952,133],{"class":132},[63,11954,11769],{"class":528},[63,11956,11957],{"class":91}," };\n",[63,11959,11960,11962,11964,11967],{"class":65,"line":277},[63,11961,11939],{"class":87},[63,11963,92],{"class":91},[63,11965,11966],{"class":95},"Start",[63,11968,403],{"class":91},[63,11970,11971],{"class":65,"line":295},[63,11972,588],{"emptyLinePlaceholder":587},[63,11974,11975],{"class":65,"line":301},[63,11976,11977],{"class":2731},"    \u002F\u002F Read the output synchronously (wait for Python to finish)\n",[63,11979,11980,11982,11985,11987,11989,11991,11994,11996,11999],{"class":65,"line":313},[63,11981,6335],{"class":439},[63,11983,11984],{"class":528}," stringResult",[63,11986,133],{"class":132},[63,11988,11915],{"class":87},[63,11990,92],{"class":91},[63,11992,11993],{"class":87},"StandardOutput",[63,11995,92],{"class":91},[63,11997,11998],{"class":95},"ReadToEnd",[63,12000,403],{"class":91},[63,12002,12003,12005,12008,12010,12012,12014,12017,12019,12021],{"class":65,"line":318},[63,12004,6335],{"class":439},[63,12006,12007],{"class":528}," errors",[63,12009,133],{"class":132},[63,12011,11915],{"class":87},[63,12013,92],{"class":91},[63,12015,12016],{"class":87},"StandardError",[63,12018,92],{"class":91},[63,12020,11998],{"class":95},[63,12022,403],{"class":91},[63,12024,12025],{"class":65,"line":340},[63,12026,588],{"emptyLinePlaceholder":587},[63,12028,12029,12031,12033,12036],{"class":65,"line":369},[63,12030,11939],{"class":87},[63,12032,92],{"class":91},[63,12034,12035],{"class":95},"WaitForExit",[63,12037,403],{"class":91},[63,12039,12040],{"class":65,"line":374},[63,12041,588],{"emptyLinePlaceholder":587},[63,12043,12044,12046,12048,12051,12053,12056,12058,12060],{"class":65,"line":387},[63,12045,3410],{"class":439},[63,12047,3366],{"class":91},[63,12049,12050],{"class":87},"process",[63,12052,92],{"class":91},[63,12054,12055],{"class":87},"ExitCode",[63,12057,8032],{"class":132},[63,12059,4464],{"class":289},[63,12061,474],{"class":91},[63,12063,12064],{"class":65,"line":392},[63,12065,250],{"class":91},[63,12067,12068,12070,12073,12075,12077,12079,12081,12083,12086],{"class":65,"line":406},[63,12069,525],{"class":439},[63,12071,12072],{"class":528}," prediction",[63,12074,133],{"class":132},[63,12076,4432],{"class":439},[63,12078,92],{"class":91},[63,12080,2986],{"class":95},[63,12082,142],{"class":91},[63,12084,12085],{"class":528},"stringResult",[63,12087,149],{"class":91},[63,12089,12090],{"class":65,"line":2931},[63,12091,12092],{"class":2731},"        \u002F\u002F Use prediction...\n",[63,12094,12095],{"class":65,"line":2937},[63,12096,621],{"class":91},[63,12098,12099],{"class":65,"line":2956},[63,12100,4618],{"class":439},[63,12102,12103],{"class":65,"line":2961},[63,12104,250],{"class":91},[63,12106,12107,12110,12112,12115,12117,12120,12122,12125,12127,12129],{"class":65,"line":3000},[63,12108,12109],{"class":87},"        _logger",[63,12111,92],{"class":91},[63,12113,12114],{"class":95},"LogError",[63,12116,142],{"class":91},[63,12118,12119],{"class":145},"$\"Python script failed: ",[63,12121,5231],{"class":6466},[63,12123,12124],{"class":528},"errors",[63,12126,5237],{"class":6466},[63,12128,5228],{"class":145},[63,12130,149],{"class":91},[63,12132,12133],{"class":65,"line":3018},[63,12134,621],{"class":91},[63,12136,12137],{"class":65,"line":3037},[63,12138,626],{"class":91},[63,12140,12141,12144,12146,12149,12152],{"class":65,"line":3056},[63,12142,12143],{"class":439},"catch",[63,12145,3366],{"class":91},[63,12147,12148],{"class":102},"Exception",[63,12150,12151],{"class":528}," ex",[63,12153,474],{"class":91},[63,12155,12156],{"class":65,"line":5491},[63,12157,118],{"class":91},[63,12159,12160,12163,12165,12167,12169,12172,12174,12177,12179,12182,12184,12186],{"class":65,"line":5515},[63,12161,12162],{"class":87},"    _logger",[63,12164,92],{"class":91},[63,12166,12114],{"class":95},[63,12168,142],{"class":91},[63,12170,12171],{"class":145},"$\"C# execution failed: ",[63,12173,5231],{"class":6466},[63,12175,12176],{"class":87},"ex",[63,12178,92],{"class":6466},[63,12180,12181],{"class":87},"Message",[63,12183,5237],{"class":6466},[63,12185,5228],{"class":145},[63,12187,149],{"class":91},[63,12189,12190],{"class":65,"line":5520},[63,12191,626],{"class":91},[63,12193,12194],{"class":65,"line":5545},[63,12195,12196],{"class":439},"finally\n",[63,12198,12199],{"class":65,"line":5550},[63,12200,118],{"class":91},[63,12202,12203,12205,12208,12211],{"class":65,"line":8275},[63,12204,11939],{"class":87},[63,12206,12207],{"class":91},"?.",[63,12209,12210],{"class":95},"Dispose",[63,12212,403],{"class":91},[63,12214,12215],{"class":65,"line":8280},[63,12216,626],{"class":91},[16,12218,12219,12220,12223],{},"For our use case, this hacked-together solution worked surprisingly well. A nice bonus: updating the model was as simple as deploying a new ",[32,12221,12222],{},"xgboost_model.json"," to all the servers (i.e., remoting in and swapping the file).",[11,12225,12227],{"id":12226},"ii-when-net-was-running-in-windows-vms-but-now-we-had-to-care-about-scale","II. When .NET was running in Windows VMs (but now we had to care about scale)",[16,12229,12230],{},"Within half a year, a similar task came up, but this time the model would be called very frequently and asynchronously. We're talking an average of 400,000 times per day, with distinct peak and off-peak periods.",[16,12232,12233],{},"The task was to integrate this model into a .NET Web API deployed across 10 VMs. The model would be invoked in one of the API's endpoints.",[16,12235,12236],{},"The previous approach doesn't scale at all. We were spinning up a new OS process for every single prediction. For 400k requests a day, that means the server has to start the Python runtime, load the libraries, load the model into memory, run the prediction, and tear everything down, 400,000 times. The overhead and CPU jitter alone would be crippling.",[16,12238,12239],{},"The ideal solution would have been to deploy a Python Web API to those VMs and let it serve as a dedicated inference service, keeping the model loaded in memory and handling requests over HTTP. But given the earlier restrictions, that wasn't on the table.",[16,12241,12242],{},"So the question became: can we run XGBoost directly in C#? And, naturally, has anyone actually done it?",[16,12244,12245],{},"I found two options:",[170,12247,12248,12264],{},[173,12249,12250,12255,12256,12259,12260,12263],{},[20,12251,12254],{"href":12252,"rel":12253},"https:\u002F\u002Fgithub.com\u002FPicNet\u002FXGBoost.Net",[52],"PicNet\u002FXGBoost.Net",", a community library built on top of the native ",[32,12257,12258],{},"xgboost.dll"," (Windows) or ",[32,12261,12262],{},"libxgboost.so"," (Linux).",[173,12265,12266,12271],{},[20,12267,12270],{"href":12268,"rel":12269},"https:\u002F\u002Fgithub.com\u002Fdotnet\u002Fmachinelearning",[52],"ML.NET",", Microsoft's own ML framework, where the approach is to convert the XGBoost model to ONNX (Open Neural Network Exchange) and run inference through ML.NET.",[16,12273,12274],{},"Option 2 felt like overkill for a pure inference problem. ONNX interoperability is a powerful concept, but it's bringing a cannon to a knife fight.",[16,12276,12277,12278,12280],{},"Option 1, on the other hand, was direct: a thin wrapper around the native C++ XGBoost library. It just worked. I did run into issues ensuring the native binaries (",[32,12279,12258],{},") and model files were copied to the right output directories and referenced correctly, but that was more a setup fumble on my part than a problem with the library itself.",[720,12282,11696],{"id":12283},"the-c-implementation-1",[16,12285,12286],{},"Using the PicNet library, the code looks surprisingly close to the Python version.",[54,12288,12290],{"className":78,"code":12289,"language":80,"meta":59,"style":59},"using XGBoost.Lib;\n\n\u002F\u002F 1. Load the model\n\u002F\u002F You'll likely need to resolve the path via the executing or calling assembly. Fun times.\nvar modelPath = @\"xgboost_model.json\";\nusing var booster = XGBoost.Booster.BoosterLoad(modelPath);\n\n\u002F\u002F 2. Prepare the data\n\u002F\u002F Pass in an array of floats, which gets converted to a DMatrix internally\nvar features = new float[] { 0.52f, 0.1f, 0.8f, 1.5f, 2.2f, 1.4f };\nvar numRows = 1;\nvar numCols = features.Length;\nusing var matrix = XGBoost.DMatrix.FromMat(features, numRows, numCols, 0.0f);\n\n\u002F\u002F 3. Run inference\n\u002F\u002F Predict returns a 2D array [rows, output_classes]\nvar prediction = booster.Predict(matrix);\n\n\u002F\u002F 4. Parse the result\n\u002F\u002F For simple classification, grab the first value of the first row\nvar result = prediction[0][0];\n",[32,12291,12292,12306,12310,12315,12320,12334,12363,12367,12372,12377,12423,12436,12454,12498,12502,12507,12512,12533,12537,12542,12547],{"__ignoreMap":59},[63,12293,12294,12296,12299,12301,12304],{"class":65,"line":66},[63,12295,1687],{"class":439},[63,12297,12298],{"class":102}," XGBoost",[63,12300,92],{"class":91},[63,12302,12303],{"class":102},"Lib",[63,12305,274],{"class":91},[63,12307,12308],{"class":65,"line":115},[63,12309,588],{"emptyLinePlaceholder":587},[63,12311,12312],{"class":65,"line":121},[63,12313,12314],{"class":2731},"\u002F\u002F 1. Load the model\n",[63,12316,12317],{"class":65,"line":152},[63,12318,12319],{"class":2731},"\u002F\u002F You'll likely need to resolve the path via the executing or calling assembly. Fun times.\n",[63,12321,12322,12324,12327,12329,12332],{"class":65,"line":253},[63,12323,2067],{"class":439},[63,12325,12326],{"class":528}," modelPath",[63,12328,133],{"class":132},[63,12330,12331],{"class":145}," @\"xgboost_model.json\"",[63,12333,274],{"class":91},[63,12335,12336,12338,12340,12343,12345,12347,12349,12351,12353,12356,12358,12361],{"class":65,"line":277},[63,12337,1687],{"class":439},[63,12339,1690],{"class":439},[63,12341,12342],{"class":528}," booster",[63,12344,133],{"class":132},[63,12346,12298],{"class":87},[63,12348,92],{"class":91},[63,12350,5102],{"class":87},[63,12352,92],{"class":91},[63,12354,12355],{"class":95},"BoosterLoad",[63,12357,142],{"class":91},[63,12359,12360],{"class":528},"modelPath",[63,12362,149],{"class":91},[63,12364,12365],{"class":65,"line":295},[63,12366,588],{"emptyLinePlaceholder":587},[63,12368,12369],{"class":65,"line":301},[63,12370,12371],{"class":2731},"\u002F\u002F 2. Prepare the data\n",[63,12373,12374],{"class":65,"line":313},[63,12375,12376],{"class":2731},"\u002F\u002F Pass in an array of floats, which gets converted to a DMatrix internally\n",[63,12378,12379,12381,12384,12386,12388,12390,12393,12396,12398,12401,12403,12406,12408,12411,12413,12416,12418,12421],{"class":65,"line":318},[63,12380,2067],{"class":439},[63,12382,12383],{"class":528}," features",[63,12385,133],{"class":132},[63,12387,136],{"class":91},[63,12389,10515],{"class":439},[63,12391,12392],{"class":91},"[] { ",[63,12394,12395],{"class":289},"0.52f",[63,12397,508],{"class":91},[63,12399,12400],{"class":289},"0.1f",[63,12402,508],{"class":91},[63,12404,12405],{"class":289},"0.8f",[63,12407,508],{"class":91},[63,12409,12410],{"class":289},"1.5f",[63,12412,508],{"class":91},[63,12414,12415],{"class":289},"2.2f",[63,12417,508],{"class":91},[63,12419,12420],{"class":289},"1.4f",[63,12422,11957],{"class":91},[63,12424,12425,12427,12430,12432,12434],{"class":65,"line":340},[63,12426,2067],{"class":439},[63,12428,12429],{"class":528}," numRows",[63,12431,133],{"class":132},[63,12433,887],{"class":289},[63,12435,274],{"class":91},[63,12437,12438,12440,12443,12445,12447,12449,12452],{"class":65,"line":369},[63,12439,2067],{"class":439},[63,12441,12442],{"class":528}," numCols",[63,12444,133],{"class":132},[63,12446,12383],{"class":87},[63,12448,92],{"class":91},[63,12450,12451],{"class":87},"Length",[63,12453,274],{"class":91},[63,12455,12456,12458,12460,12463,12465,12467,12469,12472,12474,12477,12479,12481,12483,12486,12488,12491,12493,12496],{"class":65,"line":374},[63,12457,1687],{"class":439},[63,12459,1690],{"class":439},[63,12461,12462],{"class":528}," matrix",[63,12464,133],{"class":132},[63,12466,12298],{"class":87},[63,12468,92],{"class":91},[63,12470,12471],{"class":87},"DMatrix",[63,12473,92],{"class":91},[63,12475,12476],{"class":95},"FromMat",[63,12478,142],{"class":91},[63,12480,4350],{"class":528},[63,12482,508],{"class":91},[63,12484,12485],{"class":528},"numRows",[63,12487,508],{"class":91},[63,12489,12490],{"class":528},"numCols",[63,12492,508],{"class":91},[63,12494,12495],{"class":289},"0.0f",[63,12497,149],{"class":91},[63,12499,12500],{"class":65,"line":387},[63,12501,588],{"emptyLinePlaceholder":587},[63,12503,12504],{"class":65,"line":392},[63,12505,12506],{"class":2731},"\u002F\u002F 3. Run inference\n",[63,12508,12509],{"class":65,"line":406},[63,12510,12511],{"class":2731},"\u002F\u002F Predict returns a 2D array [rows, output_classes]\n",[63,12513,12514,12516,12518,12520,12522,12524,12526,12528,12531],{"class":65,"line":2931},[63,12515,2067],{"class":439},[63,12517,12072],{"class":528},[63,12519,133],{"class":132},[63,12521,12342],{"class":87},[63,12523,92],{"class":91},[63,12525,4889],{"class":95},[63,12527,142],{"class":91},[63,12529,12530],{"class":528},"matrix",[63,12532,149],{"class":91},[63,12534,12535],{"class":65,"line":2937},[63,12536,588],{"emptyLinePlaceholder":587},[63,12538,12539],{"class":65,"line":2956},[63,12540,12541],{"class":2731},"\u002F\u002F 4. Parse the result\n",[63,12543,12544],{"class":65,"line":2961},[63,12545,12546],{"class":2731},"\u002F\u002F For simple classification, grab the first value of the first row\n",[63,12548,12549,12551,12553,12555,12557,12559,12561,12564,12566],{"class":65,"line":3000},[63,12550,2067],{"class":439},[63,12552,4879],{"class":528},[63,12554,133],{"class":132},[63,12556,12072],{"class":87},[63,12558,4353],{"class":91},[63,12560,867],{"class":289},[63,12562,12563],{"class":91},"][",[63,12565,867],{"class":289},[63,12567,5867],{"class":91},[16,12569,12570],{},"This eliminated all the process-spawning overhead. The model is loaded once at startup, and predictions happen entirely in memory. It handled 400k requests per day across 10 VMs without breaking a sweat, and arguably faster than a Python Web API would have been, since there's no network hop and no need to think about how to scale the Python side.",[16,12572,12573],{},"There was one significant caveat, though. Because the model is loaded once at startup, updating it requires restarting the application. Since we wanted to avoid forcing a restart every time the data science team retrained the model, we added two admin endpoints: one to upload a new model file, and one to trigger an in-memory reload.",[54,12575,12577],{"className":78,"code":12576,"language":80,"meta":59,"style":59},"[ApiController]\n[Route(\"api\u002F[controller]\")]\npublic class ModelAdminController : ControllerBase\n{\n    private static XGBoost.Booster _currentBooster;\n    private static readonly object _lock = new object();\n\n    \u002F\u002F Endpoint 1: Upload the new model file to the server\n    [HttpPost(\"upload\")]\n    public IActionResult UploadModel(IFormFile file)\n    {\n        if (file == null || file.Length == 0)\n            return BadRequest(\"File is empty\");\n\n        var filePath = Path.Combine(Directory.GetCurrentDirectory(), \"xgboost_model.json\");\n\n        using (var stream = new FileStream(filePath, FileMode.Create))\n        {\n            file.CopyTo(stream);\n        }\n\n        return Ok(\"Model file uploaded successfully. Call \u002Freload to apply.\");\n    }\n\n    \u002F\u002F Endpoint 2: Reload the model into memory\n    [HttpPost(\"reload\")]\n    public IActionResult ReloadModel()\n    {\n        try\n        {\n            lock (_lock)\n            {\n                _currentBooster?.Dispose();\n\n                var modelPath = Path.Combine(Directory.GetCurrentDirectory(), \"xgboost_model.json\");\n                _currentBooster = XGBoost.Booster.BoosterLoad(modelPath);\n            }\n\n            return Ok(\"Model reloaded successfully.\");\n        }\n        catch (Exception ex)\n        {\n            return StatusCode(500, $\"Failed to reload model: {ex.Message}\");\n        }\n    }\n}\n",[32,12578,12579,12588,12603,12618,12622,12639,12661,12665,12670,12684,12704,12708,12736,12750,12754,12788,12792,12826,12830,12847,12851,12855,12869,12873,12877,12882,12895,12906,12910,12915,12919,12931,12935,12946,12950,12978,13000,13004,13009,13023,13028,13042,13047,13078,13083,13088],{"__ignoreMap":59},[63,12580,12581,12583,12586],{"class":65,"line":66},[63,12582,4353],{"class":91},[63,12584,12585],{"class":102},"ApiController",[63,12587,5150],{"class":91},[63,12589,12590,12592,12595,12597,12600],{"class":65,"line":115},[63,12591,4353],{"class":91},[63,12593,12594],{"class":102},"Route",[63,12596,142],{"class":91},[63,12598,12599],{"class":145},"\"api\u002F[controller]\"",[63,12601,12602],{"class":91},")]\n",[63,12604,12605,12607,12609,12612,12615],{"class":65,"line":121},[63,12606,440],{"class":439},[63,12608,446],{"class":439},[63,12610,12611],{"class":102}," ModelAdminController",[63,12613,12614],{"class":91}," : ",[63,12616,12617],{"class":102},"ControllerBase\n",[63,12619,12620],{"class":65,"line":152},[63,12621,118],{"class":91},[63,12623,12624,12626,12628,12630,12632,12634,12637],{"class":65,"line":253},[63,12625,2964],{"class":439},[63,12627,2967],{"class":439},[63,12629,12298],{"class":102},[63,12631,92],{"class":91},[63,12633,5102],{"class":102},[63,12635,12636],{"class":2976}," _currentBooster",[63,12638,274],{"class":91},[63,12640,12641,12643,12645,12647,12650,12653,12655,12657,12659],{"class":65,"line":277},[63,12642,2964],{"class":439},[63,12644,2967],{"class":439},[63,12646,2970],{"class":439},[63,12648,12649],{"class":439}," object",[63,12651,12652],{"class":2976}," _lock",[63,12654,133],{"class":132},[63,12656,136],{"class":91},[63,12658,565],{"class":439},[63,12660,403],{"class":91},[63,12662,12663],{"class":65,"line":295},[63,12664,588],{"emptyLinePlaceholder":587},[63,12666,12667],{"class":65,"line":301},[63,12668,12669],{"class":2731},"    \u002F\u002F Endpoint 1: Upload the new model file to the server\n",[63,12671,12672,12674,12677,12679,12682],{"class":65,"line":313},[63,12673,456],{"class":91},[63,12675,12676],{"class":102},"HttpPost",[63,12678,142],{"class":91},[63,12680,12681],{"class":145},"\"upload\"",[63,12683,12602],{"class":91},[63,12685,12686,12688,12691,12694,12696,12699,12702],{"class":65,"line":318},[63,12687,483],{"class":439},[63,12689,12690],{"class":102}," IActionResult",[63,12692,12693],{"class":95}," UploadModel",[63,12695,142],{"class":91},[63,12697,12698],{"class":102},"IFormFile",[63,12700,12701],{"class":87}," file",[63,12703,474],{"class":91},[63,12705,12706],{"class":65,"line":340},[63,12707,250],{"class":91},[63,12709,12710,12712,12714,12717,12719,12721,12724,12726,12728,12730,12732,12734],{"class":65,"line":369},[63,12711,4795],{"class":439},[63,12713,3366],{"class":91},[63,12715,12716],{"class":528},"file",[63,12718,8032],{"class":132},[63,12720,3607],{"class":289},[63,12722,12723],{"class":132}," ||",[63,12725,12701],{"class":87},[63,12727,92],{"class":91},[63,12729,12451],{"class":87},[63,12731,8032],{"class":132},[63,12733,4464],{"class":289},[63,12735,474],{"class":91},[63,12737,12738,12740,12743,12745,12748],{"class":65,"line":374},[63,12739,4817],{"class":439},[63,12741,12742],{"class":95}," BadRequest",[63,12744,142],{"class":91},[63,12746,12747],{"class":145},"\"File is empty\"",[63,12749,149],{"class":91},[63,12751,12752],{"class":65,"line":387},[63,12753,588],{"emptyLinePlaceholder":587},[63,12755,12756,12758,12761,12763,12766,12768,12771,12773,12776,12778,12781,12784,12786],{"class":65,"line":392},[63,12757,525],{"class":439},[63,12759,12760],{"class":528}," filePath",[63,12762,133],{"class":132},[63,12764,12765],{"class":87}," Path",[63,12767,92],{"class":91},[63,12769,12770],{"class":95},"Combine",[63,12772,142],{"class":91},[63,12774,12775],{"class":87},"Directory",[63,12777,92],{"class":91},[63,12779,12780],{"class":95},"GetCurrentDirectory",[63,12782,12783],{"class":91},"(), ",[63,12785,11498],{"class":145},[63,12787,149],{"class":91},[63,12789,12790],{"class":65,"line":406},[63,12791,588],{"emptyLinePlaceholder":587},[63,12793,12794,12796,12798,12800,12803,12805,12807,12810,12812,12815,12817,12820,12822,12824],{"class":65,"line":2931},[63,12795,1919],{"class":439},[63,12797,3366],{"class":91},[63,12799,2067],{"class":439},[63,12801,12802],{"class":528}," stream",[63,12804,133],{"class":132},[63,12806,136],{"class":91},[63,12808,12809],{"class":102},"FileStream",[63,12811,142],{"class":91},[63,12813,12814],{"class":528},"filePath",[63,12816,508],{"class":91},[63,12818,12819],{"class":87},"FileMode",[63,12821,92],{"class":91},[63,12823,1968],{"class":87},[63,12825,1106],{"class":91},[63,12827,12828],{"class":65,"line":2937},[63,12829,1953],{"class":91},[63,12831,12832,12835,12837,12840,12842,12845],{"class":65,"line":2956},[63,12833,12834],{"class":87},"            file",[63,12836,92],{"class":91},[63,12838,12839],{"class":95},"CopyTo",[63,12841,142],{"class":91},[63,12843,12844],{"class":528},"stream",[63,12846,149],{"class":91},[63,12848,12849],{"class":65,"line":2961},[63,12850,7517],{"class":91},[63,12852,12853],{"class":65,"line":3000},[63,12854,588],{"emptyLinePlaceholder":587},[63,12856,12857,12859,12862,12864,12867],{"class":65,"line":3018},[63,12858,593],{"class":439},[63,12860,12861],{"class":95}," Ok",[63,12863,142],{"class":91},[63,12865,12866],{"class":145},"\"Model file uploaded successfully. Call \u002Freload to apply.\"",[63,12868,149],{"class":91},[63,12870,12871],{"class":65,"line":3037},[63,12872,621],{"class":91},[63,12874,12875],{"class":65,"line":3056},[63,12876,588],{"emptyLinePlaceholder":587},[63,12878,12879],{"class":65,"line":5491},[63,12880,12881],{"class":2731},"    \u002F\u002F Endpoint 2: Reload the model into memory\n",[63,12883,12884,12886,12888,12890,12893],{"class":65,"line":5515},[63,12885,456],{"class":91},[63,12887,12676],{"class":102},[63,12889,142],{"class":91},[63,12891,12892],{"class":145},"\"reload\"",[63,12894,12602],{"class":91},[63,12896,12897,12899,12901,12904],{"class":65,"line":5520},[63,12898,483],{"class":439},[63,12900,12690],{"class":102},[63,12902,12903],{"class":95}," ReloadModel",[63,12905,5131],{"class":91},[63,12907,12908],{"class":65,"line":5545},[63,12909,250],{"class":91},[63,12911,12912],{"class":65,"line":5550},[63,12913,12914],{"class":439},"        try\n",[63,12916,12917],{"class":65,"line":8275},[63,12918,1953],{"class":91},[63,12920,12921,12924,12926,12929],{"class":65,"line":8280},[63,12922,12923],{"class":439},"            lock",[63,12925,3366],{"class":91},[63,12927,12928],{"class":528},"_lock",[63,12930,474],{"class":91},[63,12932,12933],{"class":65,"line":8285},[63,12934,7479],{"class":91},[63,12936,12937,12940,12942,12944],{"class":65,"line":8290},[63,12938,12939],{"class":87},"                _currentBooster",[63,12941,12207],{"class":91},[63,12943,12210],{"class":95},[63,12945,403],{"class":91},[63,12947,12948],{"class":65,"line":8299},[63,12949,588],{"emptyLinePlaceholder":587},[63,12951,12952,12954,12956,12958,12960,12962,12964,12966,12968,12970,12972,12974,12976],{"class":65,"line":8304},[63,12953,8105],{"class":439},[63,12955,12326],{"class":528},[63,12957,133],{"class":132},[63,12959,12765],{"class":87},[63,12961,92],{"class":91},[63,12963,12770],{"class":95},[63,12965,142],{"class":91},[63,12967,12775],{"class":87},[63,12969,92],{"class":91},[63,12971,12780],{"class":95},[63,12973,12783],{"class":91},[63,12975,11498],{"class":145},[63,12977,149],{"class":91},[63,12979,12980,12982,12984,12986,12988,12990,12992,12994,12996,12998],{"class":65,"line":8309},[63,12981,12939],{"class":528},[63,12983,133],{"class":132},[63,12985,12298],{"class":87},[63,12987,92],{"class":91},[63,12989,5102],{"class":87},[63,12991,92],{"class":91},[63,12993,12355],{"class":95},[63,12995,142],{"class":91},[63,12997,12360],{"class":528},[63,12999,149],{"class":91},[63,13001,13002],{"class":65,"line":8321},[63,13003,7499],{"class":91},[63,13005,13007],{"class":65,"line":13006},38,[63,13008,588],{"emptyLinePlaceholder":587},[63,13010,13012,13014,13016,13018,13021],{"class":65,"line":13011},39,[63,13013,4817],{"class":439},[63,13015,12861],{"class":95},[63,13017,142],{"class":91},[63,13019,13020],{"class":145},"\"Model reloaded successfully.\"",[63,13022,149],{"class":91},[63,13024,13026],{"class":65,"line":13025},40,[63,13027,7517],{"class":91},[63,13029,13031,13034,13036,13038,13040],{"class":65,"line":13030},41,[63,13032,13033],{"class":439},"        catch",[63,13035,3366],{"class":91},[63,13037,12148],{"class":102},[63,13039,12151],{"class":528},[63,13041,474],{"class":91},[63,13043,13045],{"class":65,"line":13044},42,[63,13046,1953],{"class":91},[63,13048,13050,13052,13055,13057,13059,13061,13064,13066,13068,13070,13072,13074,13076],{"class":65,"line":13049},43,[63,13051,4817],{"class":439},[63,13053,13054],{"class":95}," StatusCode",[63,13056,142],{"class":91},[63,13058,2900],{"class":289},[63,13060,508],{"class":91},[63,13062,13063],{"class":145},"$\"Failed to reload model: ",[63,13065,5231],{"class":6466},[63,13067,12176],{"class":87},[63,13069,92],{"class":6466},[63,13071,12181],{"class":87},[63,13073,5237],{"class":6466},[63,13075,5228],{"class":145},[63,13077,149],{"class":91},[63,13079,13081],{"class":65,"line":13080},44,[63,13082,7517],{"class":91},[63,13084,13086],{"class":65,"line":13085},45,[63,13087,621],{"class":91},[63,13089,13091],{"class":65,"line":13090},46,[63,13092,626],{"class":91},[16,13094,13095,13096,13101],{},"At the time of writing, there is a newer and better-maintained XGBoost library for C# available at ",[20,13097,13100],{"href":13098,"rel":13099},"https:\u002F\u002Fgithub.com\u002Fmdabros\u002FXGBoostSharp",[52],"mdabros\u002FXGBoostSharp",", built on the same foundations. If you need a native implementation today, use that one. For inference-only use cases, you can ignore the training capabilities entirely.",[16,13103,13104],{},"Then, ...",[16,13106,13107],{},"One day, the team switched from XGBoost to LightGBM. Both are gradient boosting frameworks, but while XGBoost is known for its execution speed and model performance, LightGBM is often praised for being faster still and using less memory, which is a meaningful advantage when working with large datasets.",[16,13109,13110,13111,13116,13117,13120],{},"There is a LightGBM wrapper for .NET called ",[20,13112,13115],{"href":13113,"rel":13114},"https:\u002F\u002Fgithub.com\u002Frca22\u002FLightGBM.Net",[52],"LightGBM.Net",", which saved us here (Thank God!). HOWEVER, I couldn't get it to locate ",[32,13118,13119],{},"lib_lightgbm.dll"," correctly no matter where I placed the file on the server. In the end, I forked the repository and extended it to accept an absolute path for the DLL.",[11,13122,11311],{"id":11310},[16,13124,13125],{},"Path issues and dependency management in Windows VM environments became a recurring theme. Running native libraries inside .NET solved the performance problem, but it introduced a new set of maintenance headaches. What if the data science team wanted to integrate a CatBoost model next? We needed a better way to abstract this complexity away. I'll cover that in Part II, which is, if anything, more relevant in 2025.",[2563,13127,13128],{},"html pre.shiki code .sLKXg, html code.shiki .sLKXg{--shiki-default:#A626A4;--shiki-dark:#C678DD}html pre.shiki code .s5ixo, html code.shiki .s5ixo{--shiki-default:#383A42;--shiki-dark:#ABB2BF}html pre.shiki code .sW2Sy, html code.shiki .sW2Sy{--shiki-default:#A0A1A7;--shiki-default-font-style:italic;--shiki-dark:#7F848E;--shiki-dark-font-style:italic}html pre.shiki code .sknuh, html code.shiki .sknuh{--shiki-default:#383A42;--shiki-dark:#56B6C2}html pre.shiki code .slOjB, html code.shiki .slOjB{--shiki-default:#383A42;--shiki-dark:#61AFEF}html pre.shiki code .sDhpE, html code.shiki .sDhpE{--shiki-default:#50A14F;--shiki-dark:#98C379}html pre.shiki code .sAdtL, html code.shiki .sAdtL{--shiki-default:#4078F2;--shiki-dark:#61AFEF}html pre.shiki code .so_Uh, html code.shiki .so_Uh{--shiki-default:#986801;--shiki-default-font-style:inherit;--shiki-dark:#D19A66;--shiki-dark-font-style:italic}html pre.shiki code .sAGMh, html code.shiki .sAGMh{--shiki-default:#986801;--shiki-dark:#D19A66}html pre.shiki code .s_Sar, html code.shiki .s_Sar{--shiki-default:#0184BC;--shiki-dark:#56B6C2}html pre.shiki code .sJa8x, html code.shiki .sJa8x{--shiki-default:#E45649;--shiki-dark:#E06C75}html pre.shiki code .st7oF, html code.shiki .st7oF{--shiki-default:#0184BC;--shiki-dark:#ABB2BF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sz0mV, html code.shiki .sz0mV{--shiki-default:#383A42;--shiki-dark:#E06C75}html pre.shiki code .sC09Y, html code.shiki .sC09Y{--shiki-default:#C18401;--shiki-dark:#E5C07B}html pre.shiki code .sMj0N, html code.shiki .sMj0N{--shiki-default:#50A14F;--shiki-dark:#ABB2BF}html pre.shiki code .s7GmK, html code.shiki .s7GmK{--shiki-default:#383A42;--shiki-dark:#E5C07B}",{"title":59,"searchDepth":115,"depth":115,"links":13130},[13131,13132,13136,13139],{"id":13,"depth":115,"text":14},{"id":11389,"depth":115,"text":11390,"children":13133},[13134,13135],{"id":11410,"depth":121,"text":11411},{"id":11695,"depth":121,"text":11696},{"id":12226,"depth":115,"text":12227,"children":13137},[13138],{"id":12283,"depth":121,"text":11696},{"id":11310,"depth":115,"text":11311},"2025-09-02","Some history. The naive approach given the architecture back then.",{},{"title":11356,"description":13141},"blog\u002Fbuilding-ml-inference-part-1",[2594],"0j3-BAE7xZF4ZVsFSdsfpUjGmI6PWFnjXQr4o1Vg9MA",{"id":13148,"title":1338,"body":13149,"book":2585,"date":13778,"description":13779,"extension":2588,"meta":13780,"navigation":587,"path":1337,"seo":13781,"stem":13782,"tags":13783,"__hash__":13784},"blog\u002Fblog\u002Fhttpclient-connection-lifetime-observed.md",{"type":8,"value":13150,"toc":13766},[13151,13153,13158,13179,13182,13191,13198,13202,13205,13223,13226,13230,13265,13271,13275,13281,13284,13290,13293,13304,13310,13320,13323,13335,13341,13344,13354,13360,13366,13378,13384,13393,13403,13412,13419,13422,13429,13449,13458,13469,13475,13478,13482,13485,13572,13578,13585,13597,13610,13622,13660,13684,13699,13720,13723,13746,13748,13757,13763],[11,13152,14],{"id":13},[16,13154,13155,13157],{},[32,13156,34],{}," in .NET has two configuration knobs around connection lifetime. They look similar on paper, but they're solving different problems and the difference shows up the moment you watch a real connection:",[1789,13159,13160,13166],{},[173,13161,13162,13165],{},[32,13163,13164],{},"SocketsHttpHandler.PooledConnectionLifetime",": how long an individual pooled connection stays alive before it gets closed and replaced.",[173,13167,13168,13171,13172,13175,13176,13178],{},[32,13169,13170],{},"IHttpClientBuilder.SetHandlerLifetime",": how long the entire ",[32,13173,13174],{},"HttpMessageHandler"," stays alive before ",[32,13177,161],{}," rotates it.",[16,13180,13181],{},"Both have been part of my setup for years, often together, without me ever sitting down to watch what they do on the wire. The mental model came from blog posts and the Microsoft docs, but never the receipts.",[16,13183,13184,13185,13187,13188,13190],{},"This is a small follow-up to ",[20,13186,23],{"href":22}," from 2024, where I leaned on ",[32,13189,1149],{}," as one of the recommended fixes without ever showing what it actually does to a connection.",[16,13192,13193,13194],{},"Repo with the code: ",[20,13195,13196],{"href":13196,"rel":13197},"https:\u002F\u002Fgithub.com\u002Fhaiilong\u002Fdotnet-http-client-connection-test",[52],[11,13199,13201],{"id":13200},"how-i-set-it-up","How I set it up",[16,13203,13204],{},"Two projects in the solution:",[1789,13206,13207,13217],{},[173,13208,13209,13212,13213,13216],{},[2055,13210,13211],{},"TestServer",": a small ASP.NET Core API. On every incoming request, it grabs ",[32,13214,13215],{},"HttpContext.Connection.Id"," (Kestrel assigns a unique ID to each TCP connection it accepts) and echoes it back. Same TCP connection means same ID. New TCP connection means new ID.",[173,13218,13219,13222],{},[2055,13220,13221],{},"TestClient",": a console app that hits the server in five different ways and logs the connection IDs it sees come back.",[16,13224,13225],{},"So the test is just: does each client setup reuse one connection, or does it open new ones, and if it rotates, when?",[11,13227,13229],{"id":13228},"the-five-setups","The five setups",[170,13231,13232,13238,13244,13249,13255],{},[173,13233,13234,13235,13237],{},"One ",[32,13236,34],{}," instance, used for every request.",[173,13239,13240,13241,13243],{},"A brand new ",[32,13242,34],{}," instance for each request (the famous anti-pattern).",[173,13245,13246,13247,92],{},"A typed client registered with ",[32,13248,161],{},[173,13250,13251,13252,92],{},"A typed client with ",[32,13253,13254],{},"SocketsHttpHandler.PooledConnectionLifetime = 3s",[173,13256,13257,13258,13261,13262,13264],{},"A typed client with a short ",[32,13259,13260],{},"SetHandlerLifetime"," and a long ",[32,13263,1149],{},". This is the one I actually wanted to look at.",[16,13266,13267,13268,13270],{},"The first three are sanity checks against the mental model. Number four shows what ",[32,13269,1149],{}," does on its own. Number five is the question I had: if you set the pool lifetime to \"long\" but the handler lifetime to \"short\", which one wins?",[11,13272,13274],{"id":13273},"what-actually-happens","What actually happens",[16,13276,13277,13280],{},[2055,13278,13279],{},"Scenarios 1 to 3"," lined up with what I expected.",[16,13282,13283],{},"One client reused for every request: same connection ID, every time. The TCP connection is held open by HTTP keep-alive and reused for the life of the process.",[54,13285,13288],{"className":13286,"code":13287,"language":2529},[2527],"Request 1:  Connection OJN, 1st Request (took 3183ms)\nRequest 2:  Connection OJN, 2nd Request (took 5ms)\nRequest 3:  Connection OJN, 3rd Request (took 1ms)\nRequest 4:  Connection OJN, 4th Request (took 1ms)\nRequest 5:  Connection OJN, 5th Request (took 0ms)\n",[32,13289,13287],{"__ignoreMap":59},[16,13291,13292],{},"The first request paid the cold-start cost (TLS handshake, JIT warmup, the usual first-time overhead). Every subsequent request hit the warm connection and finished in single-digit milliseconds.",[16,13294,13295,13296,13299,13300,13303],{},"A new client per request: fresh connection ID every time, and (if you watch ",[32,13297,13298],{},"netstat",") a slow accumulation of sockets stuck in ",[32,13301,13302],{},"TIME_WAIT"," for the OS-default duration before the kernel cleans them up. The docs have called this out for years, but watching the count tick up locally is more concrete than reading about it.",[54,13305,13308],{"className":13306,"code":13307,"language":2529},[2527],"Request 1:  Connection OJO, 1st Request (took 2027ms)\nRequest 2:  Connection OJP, 1st Request (took 2073ms)\nRequest 3:  Connection OJQ, 1st Request (took 2023ms)\nRequest 4:  Connection OJR, 1st Request (took 2031ms)\nRequest 5:  Connection OJS, 1st Request (took 2026ms)\n",[32,13309,13307],{"__ignoreMap":59},[16,13311,13312,13313,13315,13316,13319],{},"Five different connection IDs (OJO through OJS) for five requests. There's no ",[32,13314,3508],{}," in this scenario; the ~2s per request is the actual cost of opening each new connection on this Windows setup, almost certainly dominated by the TLS handshake against the localhost dev cert (Windows does certificate chain validation, including possible revocation checks, surprisingly slowly). The contrast with scenario 1 is the lesson: a warm reused connection completed in under 5ms, while every cold connection here costs two full seconds. The \"don't ",[32,13317,13318],{},"new"," HttpClient per request\" rule isn't theoretical.",[16,13321,13322],{},"For context: in a Linux production setup talking to a known host in the same cloud region, a new TLS connection is usually well under 50ms, often less with TLS 1.3 and a warm DNS cache. The 2 second figure here is a Windows-localhost-dev-cert artifact. But cold-versus-warm is always going to be meaningfully slower in any environment; the magnitude just shifts with where you're running.",[16,13324,13325,13326,13328,13329,13331,13332,13334],{},"Typed client via ",[32,13327,161],{},": same connection ID across requests. The factory keeps one ",[32,13330,13174],{}," alive in its internal cache and hands out lightweight ",[32,13333,34],{}," wrappers around it. From the connection's point of view, every request through this client looks the same.",[54,13336,13339],{"className":13337,"code":13338,"language":2529},[2527],"Request 1:  Connection OJT, 1st Request (took 2051ms)\nRequest 2:  Connection OJT, 2nd Request (took 1ms)\nRequest 3:  Connection OJT, 3rd Request (took 1ms)\nRequest 4:  Connection OJT, 4th Request (took 1ms)\nRequest 5:  Connection OJT, 5th Request (took 1ms)\n",[32,13340,13338],{"__ignoreMap":59},[16,13342,13343],{},"Same shape as scenario 1: one connection (OJT), warm-up on the first hit, sub-millisecond for the rest.",[16,13345,13346,13349,13350,13353],{},[2055,13347,13348],{},"Scenario 4"," is the interesting one. With ",[32,13351,13352],{},"PooledConnectionLifetime = 3s",", the connection ID stayed the same for around three seconds, then flipped. Then stayed the same for another three seconds, then flipped again. One connection rotated at a time, the handler itself stayed put, no hiccup in the request stream.",[54,13355,13358],{"className":13356,"code":13357,"language":2529},[2527],"Request 1:  Connection OJU, 1st Request at 12:26:01 (took 2016ms)\nRequest 2:  Connection OJU, 2nd Request at 12:26:03 (took 3ms)\nRequest 3:  Connection OJV, 1st Request at 12:26:07 (took 2030ms)\nRequest 4:  Connection OJV, 2nd Request at 12:26:09 (took 3ms)\nRequest 5:  Connection OK0, 1st Request at 12:26:13 (took 2040ms)\n",[32,13359,13357],{"__ignoreMap":59},[16,13361,13362,13363,13365],{},"The pattern is exactly what ",[32,13364,1149],{}," is supposed to produce. Requests 1 and 2 share OJU because they happen within the 3 second window. Request 3, four seconds after request 2, arrives after OJU's pooled lifetime has expired, so the pool rotates and produces OJV. Requests 3 and 4 share OJV because they too land within a 3 second window. Request 5 hits after OJV has expired and OK0 takes over. Throughout all five requests the handler itself is the same; only the connections inside its pool cycle.",[16,13367,13368,13371,13372,13374,13375,13377],{},[2055,13369,13370],{},"Scenario 5"," is where I learned something. Before running it, I had quietly assumed that a long ",[32,13373,1149],{}," would protect existing connections even when the handler rotated underneath. It does not. As soon as ",[32,13376,13260],{}," expired and the factory swapped in a fresh handler, every subsequent request landed on a brand new connection ID, regardless of how much time those pooled connections had left on the clock.",[54,13379,13382],{"className":13380,"code":13381,"language":2529},[2527],"Request 1:  Connection OK1, 1st Request at 12:26:15 (took 2050ms)\nRequest 2:  Connection OK2, 1st Request at 12:26:19 (took 2043ms)\nRequest 3:  Connection OK3, 1st Request at 12:26:23 (took 2050ms)\nRequest 4:  Connection OK4, 1st Request at 12:26:27 (took 2042ms)\nRequest 5:  Connection OK5, 1st Request at 12:26:31 (took 2036ms)\n",[32,13383,13381],{"__ignoreMap":59},[16,13385,13386,13387,13389,13390,13392],{},"Five requests, five different connections (OK1, OK2, OK3, OK4, OK5). Requests are four seconds apart, ",[32,13388,13260],{}," is one second, so by the time each request lands, the previous handler has already aged out of the factory's active slot. A fresh handler means a fresh pool means a fresh connection. The long ",[32,13391,1149],{}," setting doesn't get a chance to matter because the pool it lives in has already been discarded.",[16,13394,13395,13396,13398,13399,13402],{},"Which makes sense once you think about it. ",[32,13397,1149],{}," is a property ",[3240,13400,13401],{},"on the handler's connection pool",". The pool lives inside the handler. Once the handler is disposed, the pool goes with it, and so do the connections. There is no shared connection state across handlers in the factory's cache.",[16,13404,13405,13406,13408,13409,13411],{},"So the two settings really are not interchangeable. ",[32,13407,1149],{}," rotates connections gracefully under a stable handler. ",[32,13410,13260],{}," resets the whole pool. If both fire, the handler one wins, because the pool only exists inside the handler.",[11,13413,13415,13416,13418],{"id":13414},"a-short-detour-into-how-ihttpclientfactory-actually-works","A short detour into how ",[32,13417,161],{}," actually works",[16,13420,13421],{},"Mapping this out is what made scenario 5 stop feeling surprising, so it's worth a paragraph.",[16,13423,13424,13425,13428],{},"When you call ",[32,13426,13427],{},"AddHttpClient(...)",", the factory keeps an internal cache mapping the client name to an \"active handler entry\". Each entry holds:",[1789,13430,13431,13439,13442],{},[173,13432,13433,13434,13436,13437,6359],{},"The actual ",[32,13435,13174],{}," (which has its own connection pool, in the case of ",[32,13438,38],{},[173,13440,13441],{},"A timestamp for when it was created.",[173,13443,13444,13445,13448],{},"The configured ",[32,13446,13447],{},"HandlerLifetime"," (default 2 minutes).",[16,13450,13451,13452,13454,13455,13457],{},"When you ask for an ",[32,13453,34],{},", the factory checks the cache. If the active entry is still within its lifetime, you get a fresh ",[32,13456,34],{}," wrapping the same handler, and therefore the same pool. If it's expired, the factory:",[170,13459,13460,13463,13466],{},[173,13461,13462],{},"Moves the expired entry to an \"expired handlers\" list.",[173,13464,13465],{},"Creates a new active entry with a fresh handler.",[173,13467,13468],{},"Starts a cleanup timer.",[16,13470,13471,13472,13474],{},"The expired handler is not disposed right away. The factory holds onto it for a grace period (4 minutes, hardcoded last I checked) so any in-flight ",[32,13473,34],{}," instances that already hold a reference can finish their requests. Once that grace period passes and no references remain, the expired handler is disposed, which closes every connection in its pool.",[16,13476,13477],{},"That grace period is why in scenario 5 you don't see existing requests get interrupted. New requests start landing on a new handler's pool, which means a new connection. The old handler is still alive in the factory's expired list, waiting to be cleaned up.",[11,13479,13481],{"id":13480},"what-i-actually-want-in-production","What I actually want in production",[16,13483,13484],{},"For long-running .NET services that talk to other services over HTTP (which, in a Kubernetes world, is most of them), the configuration I keep coming back to is:",[54,13486,13488],{"className":78,"code":13487,"language":80,"meta":59,"style":59},"services\n    .AddHttpClient\u003CMyClient>()\n    .ConfigurePrimaryHttpMessageHandler(_ => new SocketsHttpHandler\n    {\n        PooledConnectionLifetime = TimeSpan.FromMinutes(2),\n    })\n    .SetHandlerLifetime(Timeout.InfiniteTimeSpan);\n",[32,13489,13490,13495,13509,13527,13531,13550,13555],{"__ignoreMap":59},[63,13491,13492],{"class":65,"line":66},[63,13493,13494],{"class":528},"services\n",[63,13496,13497,13499,13501,13503,13506],{"class":65,"line":115},[63,13498,1111],{"class":91},[63,13500,96],{"class":95},[63,13502,99],{"class":91},[63,13504,13505],{"class":102},"MyClient",[63,13507,13508],{"class":91},">()\n",[63,13510,13511,13513,13516,13518,13521,13524],{"class":65,"line":121},[63,13512,1111],{"class":91},[63,13514,13515],{"class":95},"ConfigurePrimaryHttpMessageHandler",[63,13517,142],{"class":91},[63,13519,13520],{"class":87},"_",[63,13522,13523],{"class":91}," => new ",[63,13525,13526],{"class":102},"SocketsHttpHandler\n",[63,13528,13529],{"class":65,"line":152},[63,13530,250],{"class":91},[63,13532,13533,13536,13538,13540,13542,13544,13546,13548],{"class":65,"line":253},[63,13534,13535],{"class":528},"        PooledConnectionLifetime",[63,13537,133],{"class":132},[63,13539,682],{"class":87},[63,13541,92],{"class":91},[63,13543,3939],{"class":95},[63,13545,142],{"class":91},[63,13547,6285],{"class":289},[63,13549,991],{"class":91},[63,13551,13552],{"class":65,"line":277},[63,13553,13554],{"class":91},"    })\n",[63,13556,13557,13559,13561,13563,13565,13567,13570],{"class":65,"line":295},[63,13558,1111],{"class":91},[63,13560,13260],{"class":95},[63,13562,142],{"class":91},[63,13564,1583],{"class":87},[63,13566,92],{"class":91},[63,13568,13569],{"class":87},"InfiniteTimeSpan",[63,13571,149],{"class":91},[16,13573,13574,13575,13577],{},"One handler that stays alive for the whole app, with ",[32,13576,1149],{}," quietly rotating connections underneath it. If you set both to short values, you stack the handler-disposal pain on top of pool rotation, paying twice for what one of them already does.",[11,13579,13581,13582,13584],{"id":13580},"why-timeoutinfinitetimespan-and-what-the-default-is","Why ",[32,13583,1316],{},", and what the default is",[16,13586,13587,13588,13590,13591,13593,13594,13596],{},"The default for ",[32,13589,13260],{}," is ",[2055,13592,1152],{},". If you call ",[32,13595,13427],{}," and never touch the lifetime, the factory will rotate the handler every two minutes for the life of your app.",[16,13598,13599,13600,13602,13603,13605,13606,13609],{},"That default exists for historical reasons. ",[32,13601,161],{}," shipped in .NET Core 2.1 (2018) to solve two ",[32,13604,34],{}," problems people kept hitting in production: socket exhaustion from ",[32,13607,13608],{},"new HttpClient()"," per request, and stale DNS on long-lived clients that never re-resolve. Its answer to the DNS problem was the heavy hammer: rotate the entire handler on a fixed schedule, dispose the old one after a grace period, force a fresh DNS lookup on the next request. Two minutes was a reasonable balance between DNS freshness and the cost of throwing the pool away.",[16,13611,13612,13613,13615,13616,13618,13619,13621],{},"The same release also introduced ",[32,13614,38],{},", the fully managed HTTP handler that's been the underlying implementation under ",[32,13617,34],{}," ever since. It exposes ",[32,13620,1149],{},", which does the DNS-refresh job at a finer grain than handler rotation: each pooled connection has its own age, and they cycle individually without disposing the handler or losing its TLS session tickets. For DNS refresh specifically, that's better in basically every measurable way than rotating the whole handler.",[16,13623,13624,13625,13642,13643,13646,13647,13649,13650,13652,13653,13655,13656,13659],{},"Worth being precise about which release did what, because it took me a while to untangle: ",[2055,13626,13627,13629,13630,13633,13634,13636,13637],{},[32,13628,38],{}," only became the default ",[3240,13631,13632],{},"primary"," handler for ",[32,13635,161],{}," in ",[20,13638,13641],{"href":13639,"rel":13640},"https:\u002F\u002Flearn.microsoft.com\u002Fen-us\u002Fdotnet\u002Fcore\u002Fcompatibility\u002Fnetworking\u002F9.0\u002Fdefault-handler",[52],".NET 9 Preview 6",". Before that, the factory's default primary handler was ",[32,13644,13645],{},"HttpClientHandler",", which is a thin wrapper around ",[32,13648,38],{}," that does not expose ",[32,13651,1149],{},". So on .NET 8 and earlier, the only way to get ",[32,13654,1149],{}," in your factory setup was to explicitly opt in with ",[32,13657,13658],{},"ConfigurePrimaryHttpMessageHandler(_ => new SocketsHttpHandler { ... })",", which is exactly what the config block above does.",[16,13661,13662,13663,13665,13666,13668,13669,13671,13672,13677,13678,13680,13681,13683],{},".NET 9 also added a nice touch: when the default primary handler is ",[32,13664,38],{},", the factory now auto-sets ",[32,13667,1149],{}," to match ",[32,13670,13447],{}," if you don't configure either. The motivation, ",[20,13673,13676],{"href":13674,"rel":13675},"https:\u002F\u002Flearn.microsoft.com\u002Fen-us\u002Fdotnet\u002Fcore\u002Fcompatibility\u002Fnetworking\u002F9.0\u002Fdefault-handler#reason-for-change",[52],"per the docs",", is the singleton-capture footgun: if someone injects a typed client into a singleton service, the factory can no longer rotate that handler, and pre-.NET 9 the connections inside it would keep their stale DNS forever. With ",[32,13679,1149],{}," linked to ",[32,13682,13447],{}," by default, the underlying connections still rotate even when the handler doesn't.",[16,13685,13686,13687,13689,13690,13692,13693,13695,13696,13698],{},"The 2 minute default for ",[32,13688,13260],{}," itself never went away. Partly back-compat, partly because not every primary handler is ",[32,13691,38],{},". People still pick ",[32,13694,13645],{}," explicitly for cookie or proxy property access, or run on .NET Framework where ",[32,13697,38],{}," isn't supported at all. The factory can't assume the modern primitive is available.",[16,13700,13701,13702,13704,13705,13707,13708,13711,13712,13714,13715,92],{},"If you're on a recent .NET and using ",[32,13703,38],{}," (default since .NET 9, opt-in via ",[32,13706,13515],{}," before that), the recommendation is still to set ",[32,13709,13710],{},"SetHandlerLifetime(Timeout.InfiniteTimeSpan)"," and let ",[32,13713,1149],{}," do the rotation. Microsoft says as much in ",[20,13716,13719],{"href":13717,"rel":13718},"https:\u002F\u002Flearn.microsoft.com\u002Fen-us\u002Fdotnet\u002Ffundamentals\u002Fnetworking\u002Fhttp\u002Fhttpclient-guidelines",[52],"the current HttpClient guidelines",[16,13721,13722],{},"Practical rules of thumb:",[1789,13724,13725,13734,13737],{},[173,13726,13727,13728,13730,13731,13733],{},"If you control your handler and you're on a recent .NET, set ",[32,13729,13710],{}," and configure ",[32,13732,1149],{}," to something sensible like 1 to 5 minutes.",[173,13735,13736],{},"If you don't configure anything, the 2 minute default still gives you DNS refresh, just less efficiently. It's not broken, it's doing things the old way.",[173,13738,13739,13740,13742,13743,13745],{},"If you're stuck with ",[32,13741,13645],{}," (legacy bind, custom handler chain), keep the default ",[32,13744,13260],{},". It's the only mechanism you have for DNS refresh.",[11,13747,4106],{"id":4105},[16,13749,13750,13751,13753,13754,13756],{},"The thing I keep noticing when I write these small experiments up is how much more sticks after watching the thing run. I'd read about ",[32,13752,1149],{}," versus ",[32,13755,13260],{}," more times than I can count, but the difference only really clicked once the connection IDs started flipping on the screen.",[16,13758,13759,13760],{},"Repo, again, if you want to clone and poke at it yourself: ",[20,13761,13196],{"href":13196,"rel":13762},[52],[2563,13764,13765],{},"html pre.shiki code .sz0mV, html code.shiki .sz0mV{--shiki-default:#383A42;--shiki-dark:#E06C75}html pre.shiki code .s5ixo, html code.shiki .s5ixo{--shiki-default:#383A42;--shiki-dark:#ABB2BF}html pre.shiki code .sAdtL, html code.shiki .sAdtL{--shiki-default:#4078F2;--shiki-dark:#61AFEF}html pre.shiki code .sC09Y, html code.shiki .sC09Y{--shiki-default:#C18401;--shiki-dark:#E5C07B}html pre.shiki code .s7GmK, html code.shiki .s7GmK{--shiki-default:#383A42;--shiki-dark:#E5C07B}html pre.shiki code .sknuh, html code.shiki .sknuh{--shiki-default:#383A42;--shiki-dark:#56B6C2}html pre.shiki code .sAGMh, html code.shiki .sAGMh{--shiki-default:#986801;--shiki-dark:#D19A66}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":59,"searchDepth":115,"depth":115,"links":13767},[13768,13769,13770,13771,13772,13774,13775,13777],{"id":13,"depth":115,"text":14},{"id":13200,"depth":115,"text":13201},{"id":13228,"depth":115,"text":13229},{"id":13273,"depth":115,"text":13274},{"id":13414,"depth":115,"text":13773},"A short detour into how IHttpClientFactory actually works",{"id":13480,"depth":115,"text":13481},{"id":13580,"depth":115,"text":13776},"Why Timeout.InfiniteTimeSpan, and what the default is",{"id":4105,"depth":115,"text":4106},"2025-07-02","A small experiment to see what SetHandlerLifetime and PooledConnectionLifetime actually do to connection reuse.",{},{"title":1338,"description":13779},"blog\u002Fhttpclient-connection-lifetime-observed",[2594],"yuRscd9elsy9xy5u2VwI3JCcBUQJKcggU35sEvNEkDI",{"id":13786,"title":13787,"body":13788,"book":2585,"date":18522,"description":18523,"extension":2588,"meta":18524,"navigation":587,"path":18525,"seo":18526,"stem":18527,"tags":18528,"__hash__":18529},"blog\u002Fblog\u002Freduce-memory-footprint-and-garbage-collection-pressure.md","Reduce Memory Footprint and Garbage Collection Pressure",{"type":8,"value":13789,"toc":18506},[13790,13793,13796,13803,13814,13817,13830,13834,13837,13861,13864,13952,13955,14093,14096,14102,14106,14108,14131,14133,14208,14210,14458,14460,14466,14470,14472,14622,14624,14675,14677,14775,14777,14783,14787,14790,16025,16027,16033,16037,16040,16077,16079,16113,16115,16121,16125,16131,16256,16411,16420,17287,17289,17295,17298,17303,17344,17348,17358,17393,17397,17400,17417,17420,17426,17429,17437,17441,17448,17452,17466,17470,17484,17488,17508,17512,17529,17532,17884,17887,17893,17896,17899,17902,17908,18503],[16,13791,13792],{},"Performance optimization along hot paths (frequent APIs) often comes down to memory allocation often comes down to managing memory allocations and reducing garbage collection pressure.",[16,13794,13795],{},"We explore a few .NET \"Data Structures\" that boosts this.",[11,13797,13799,13800],{"id":13798},"i-readonlyspan","I. ReadOnlySpan",[13801,13802],"t",{},[16,13804,13805,13808,13809,13811,13812],{},[32,13806,13807],{},"ReadOnlySpan\u003CT>"," is a lightweight, stack only type that represents a read only view over a contiguous region of memory. Unlike traditional string operations or array slicing that create new copies of data, ",[32,13810,13807],{}," provides a zero allocation way to work with memory segments. Many modern APIs support ",[32,13813,13807],{},[16,13815,13816],{},"Think of it as pointer's APIs (not really but you can think like that).",[16,13818,13819,13822,13823,3246,13826,13829],{},[32,13820,13821],{},"Span\u003CT>"," will allow you to modify the underlying data and ",[32,13824,13825],{},"Memory\u003CT>",[32,13827,13828],{},"ReadOnly\u003CMemory\u003CT>>"," stay in the heap so you can use them with things like fields.",[11,13831,13833],{"id":13832},"substring-and-concat","Substring and Concat",[16,13835,13836],{},"Given",[54,13838,13840],{"className":78,"code":13839,"language":80,"meta":59,"style":59},"private const string LongText = \"The quick brown fox jumps over the lazy dog\";\n",[32,13841,13842],{"__ignoreMap":59},[63,13843,13844,13846,13849,13851,13854,13856,13859],{"class":65,"line":66},[63,13845,5789],{"class":439},[63,13847,13848],{"class":439}," const",[63,13850,2766],{"class":439},[63,13852,13853],{"class":528}," LongText",[63,13855,133],{"class":132},[63,13857,13858],{"class":145}," \"The quick brown fox jumps over the lazy dog\"",[63,13860,274],{"class":91},[16,13862,13863],{},"Compare",[54,13865,13867],{"className":78,"code":13866,"language":80,"meta":59,"style":59},"public string Traditional()\n{\n    string first = LongText.Substring(4, 5);\n    string second = LongText.Substring(10, 5);\n    return first + second;\n}\n",[32,13868,13869,13880,13884,13910,13936,13948],{"__ignoreMap":59},[63,13870,13871,13873,13875,13878],{"class":65,"line":66},[63,13872,440],{"class":439},[63,13874,2766],{"class":439},[63,13876,13877],{"class":95}," Traditional",[63,13879,5131],{"class":91},[63,13881,13882],{"class":65,"line":115},[63,13883,118],{"class":91},[63,13885,13886,13888,13891,13893,13895,13897,13900,13902,13904,13906,13908],{"class":65,"line":121},[63,13887,1867],{"class":439},[63,13889,13890],{"class":528}," first",[63,13892,133],{"class":132},[63,13894,13853],{"class":87},[63,13896,92],{"class":91},[63,13898,13899],{"class":95},"Substring",[63,13901,142],{"class":91},[63,13903,771],{"class":289},[63,13905,508],{"class":91},[63,13907,1596],{"class":289},[63,13909,149],{"class":91},[63,13911,13912,13914,13917,13919,13921,13923,13925,13927,13930,13932,13934],{"class":65,"line":152},[63,13913,1867],{"class":439},[63,13915,13916],{"class":528}," second",[63,13918,133],{"class":132},[63,13920,13853],{"class":87},[63,13922,92],{"class":91},[63,13924,13899],{"class":95},[63,13926,142],{"class":91},[63,13928,13929],{"class":289},"10",[63,13931,508],{"class":91},[63,13933,1596],{"class":289},[63,13935,149],{"class":91},[63,13937,13938,13940,13942,13944,13946],{"class":65,"line":253},[63,13939,1890],{"class":439},[63,13941,13890],{"class":528},[63,13943,6165],{"class":132},[63,13945,13916],{"class":528},[63,13947,274],{"class":91},[63,13949,13950],{"class":65,"line":277},[63,13951,626],{"class":91},[16,13953,13954],{},"vs",[54,13956,13958],{"className":78,"code":13957,"language":80,"meta":59,"style":59},"public string Span()\n{\n    ReadOnlySpan\u003Cchar> span = LongText.AsSpan();\n    ReadOnlySpan\u003Cchar> first = span.Slice(4, 5);\n    ReadOnlySpan\u003Cchar> second = span.Slice(10, 5);\n\n    return string.Concat(first, second);\n}\n",[32,13959,13960,13971,13975,14000,14033,14064,14068,14089],{"__ignoreMap":59},[63,13961,13962,13964,13966,13969],{"class":65,"line":66},[63,13963,440],{"class":439},[63,13965,2766],{"class":439},[63,13967,13968],{"class":95}," Span",[63,13970,5131],{"class":91},[63,13972,13973],{"class":65,"line":115},[63,13974,118],{"class":91},[63,13976,13977,13980,13982,13985,13987,13989,13991,13993,13995,13998],{"class":65,"line":121},[63,13978,13979],{"class":102},"    ReadOnlySpan",[63,13981,99],{"class":91},[63,13983,13984],{"class":439},"char",[63,13986,1847],{"class":91},[63,13988,63],{"class":528},[63,13990,133],{"class":132},[63,13992,13853],{"class":87},[63,13994,92],{"class":91},[63,13996,13997],{"class":95},"AsSpan",[63,13999,403],{"class":91},[63,14001,14002,14004,14006,14008,14010,14013,14015,14018,14020,14023,14025,14027,14029,14031],{"class":65,"line":152},[63,14003,13979],{"class":102},[63,14005,99],{"class":91},[63,14007,13984],{"class":439},[63,14009,1847],{"class":91},[63,14011,14012],{"class":528},"first",[63,14014,133],{"class":132},[63,14016,14017],{"class":87}," span",[63,14019,92],{"class":91},[63,14021,14022],{"class":95},"Slice",[63,14024,142],{"class":91},[63,14026,771],{"class":289},[63,14028,508],{"class":91},[63,14030,1596],{"class":289},[63,14032,149],{"class":91},[63,14034,14035,14037,14039,14041,14043,14046,14048,14050,14052,14054,14056,14058,14060,14062],{"class":65,"line":253},[63,14036,13979],{"class":102},[63,14038,99],{"class":91},[63,14040,13984],{"class":439},[63,14042,1847],{"class":91},[63,14044,14045],{"class":528},"second",[63,14047,133],{"class":132},[63,14049,14017],{"class":87},[63,14051,92],{"class":91},[63,14053,14022],{"class":95},[63,14055,142],{"class":91},[63,14057,13929],{"class":289},[63,14059,508],{"class":91},[63,14061,1596],{"class":289},[63,14063,149],{"class":91},[63,14065,14066],{"class":65,"line":277},[63,14067,588],{"emptyLinePlaceholder":587},[63,14069,14070,14072,14074,14076,14079,14081,14083,14085,14087],{"class":65,"line":295},[63,14071,1890],{"class":439},[63,14073,2766],{"class":439},[63,14075,92],{"class":91},[63,14077,14078],{"class":95},"Concat",[63,14080,142],{"class":91},[63,14082,14012],{"class":528},[63,14084,508],{"class":91},[63,14086,14045],{"class":528},[63,14088,149],{"class":91},[63,14090,14091],{"class":65,"line":301},[63,14092,626],{"class":91},[16,14094,14095],{},"Result:",[16,14097,14098],{},[7330,14099],{"alt":14100,"src":14101},"image-20250218-020246.png","\u002Fblog\u002Freduce-memory-footprint-and-garbage-collection-pressure\u002Fimage_20250218_020246.png",[11,14103,14105],{"id":14104},"number-parsing","Number Parsing",[16,14107,13836],{},[54,14109,14111],{"className":78,"code":14110,"language":80,"meta":59,"style":59},"private const string Numbers = \"123,456,789\";\n",[32,14112,14113],{"__ignoreMap":59},[63,14114,14115,14117,14119,14121,14124,14126,14129],{"class":65,"line":66},[63,14116,5789],{"class":439},[63,14118,13848],{"class":439},[63,14120,2766],{"class":439},[63,14122,14123],{"class":528}," Numbers",[63,14125,133],{"class":132},[63,14127,14128],{"class":145}," \"123,456,789\"",[63,14130,274],{"class":91},[16,14132,13863],{},[54,14134,14136],{"className":78,"code":14135,"language":80,"meta":59,"style":59},"public List\u003Cint> Traditional()\n{\n    return Numbers.Split(',')\n        .Select(int.Parse)\n        .ToList();\n}\n",[32,14137,14138,14156,14160,14178,14195,14204],{"__ignoreMap":59},[63,14139,14140,14142,14145,14147,14149,14151,14154],{"class":65,"line":66},[63,14141,440],{"class":439},[63,14143,14144],{"class":102}," List",[63,14146,99],{"class":91},[63,14148,856],{"class":439},[63,14150,1847],{"class":91},[63,14152,14153],{"class":95},"Traditional",[63,14155,5131],{"class":91},[63,14157,14158],{"class":65,"line":115},[63,14159,118],{"class":91},[63,14161,14162,14164,14166,14168,14171,14173,14176],{"class":65,"line":121},[63,14163,1890],{"class":439},[63,14165,14123],{"class":87},[63,14167,92],{"class":91},[63,14169,14170],{"class":95},"Split",[63,14172,142],{"class":91},[63,14174,14175],{"class":145},"','",[63,14177,474],{"class":91},[63,14179,14180,14183,14185,14187,14189,14191,14193],{"class":65,"line":152},[63,14181,14182],{"class":91},"        .",[63,14184,6347],{"class":95},[63,14186,142],{"class":91},[63,14188,856],{"class":439},[63,14190,92],{"class":91},[63,14192,2986],{"class":87},[63,14194,474],{"class":91},[63,14196,14197,14199,14202],{"class":65,"line":253},[63,14198,14182],{"class":91},[63,14200,14201],{"class":95},"ToList",[63,14203,403],{"class":91},[63,14205,14206],{"class":65,"line":277},[63,14207,626],{"class":91},[16,14209,13954],{},[54,14211,14213],{"className":78,"code":14212,"language":80,"meta":59,"style":59},"public List\u003Cint> Span()\n{\n    ReadOnlySpan\u003Cchar> span = Numbers.AsSpan();\n    var result = new List\u003Cint>();\n    \n    while (span.Length > 0)\n    {\n        var commaPos = span.IndexOf(',');\n        if (commaPos == -1)\n        {\n            result.Add(int.Parse(span));\n            break;\n        }\n        \n        result.Add(int.Parse(span[..commaPos]));\n        span = span[(commaPos + 1)..];\n    }\n    \n    return result;\n}\n",[32,14214,14215,14232,14236,14258,14278,14283,14302,14306,14328,14345,14349,14372,14379,14383,14388,14417,14438,14442,14446,14454],{"__ignoreMap":59},[63,14216,14217,14219,14221,14223,14225,14227,14230],{"class":65,"line":66},[63,14218,440],{"class":439},[63,14220,14144],{"class":102},[63,14222,99],{"class":91},[63,14224,856],{"class":439},[63,14226,1847],{"class":91},[63,14228,14229],{"class":95},"Span",[63,14231,5131],{"class":91},[63,14233,14234],{"class":65,"line":115},[63,14235,118],{"class":91},[63,14237,14238,14240,14242,14244,14246,14248,14250,14252,14254,14256],{"class":65,"line":121},[63,14239,13979],{"class":102},[63,14241,99],{"class":91},[63,14243,13984],{"class":439},[63,14245,1847],{"class":91},[63,14247,63],{"class":528},[63,14249,133],{"class":132},[63,14251,14123],{"class":87},[63,14253,92],{"class":91},[63,14255,13997],{"class":95},[63,14257,403],{"class":91},[63,14259,14260,14262,14264,14266,14268,14271,14273,14275],{"class":65,"line":152},[63,14261,6335],{"class":439},[63,14263,4879],{"class":528},[63,14265,133],{"class":132},[63,14267,136],{"class":91},[63,14269,14270],{"class":102},"List",[63,14272,99],{"class":91},[63,14274,856],{"class":439},[63,14276,14277],{"class":91},">();\n",[63,14279,14280],{"class":65,"line":253},[63,14281,14282],{"class":91},"    \n",[63,14284,14285,14287,14289,14291,14293,14295,14298,14300],{"class":65,"line":277},[63,14286,5827],{"class":439},[63,14288,3366],{"class":91},[63,14290,63],{"class":87},[63,14292,92],{"class":91},[63,14294,12451],{"class":87},[63,14296,14297],{"class":132}," >",[63,14299,4464],{"class":289},[63,14301,474],{"class":91},[63,14303,14304],{"class":65,"line":295},[63,14305,250],{"class":91},[63,14307,14308,14310,14313,14315,14317,14319,14322,14324,14326],{"class":65,"line":301},[63,14309,525],{"class":439},[63,14311,14312],{"class":528}," commaPos",[63,14314,133],{"class":132},[63,14316,14017],{"class":87},[63,14318,92],{"class":91},[63,14320,14321],{"class":95},"IndexOf",[63,14323,142],{"class":91},[63,14325,14175],{"class":145},[63,14327,149],{"class":91},[63,14329,14330,14332,14334,14337,14339,14341,14343],{"class":65,"line":313},[63,14331,4795],{"class":439},[63,14333,3366],{"class":91},[63,14335,14336],{"class":528},"commaPos",[63,14338,8032],{"class":132},[63,14340,4363],{"class":132},[63,14342,880],{"class":289},[63,14344,474],{"class":91},[63,14346,14347],{"class":65,"line":318},[63,14348,1953],{"class":91},[63,14350,14351,14354,14356,14358,14360,14362,14364,14366,14368,14370],{"class":65,"line":340},[63,14352,14353],{"class":87},"            result",[63,14355,92],{"class":91},[63,14357,357],{"class":95},[63,14359,142],{"class":91},[63,14361,856],{"class":439},[63,14363,92],{"class":91},[63,14365,2986],{"class":95},[63,14367,142],{"class":91},[63,14369,63],{"class":528},[63,14371,366],{"class":91},[63,14373,14374,14377],{"class":65,"line":369},[63,14375,14376],{"class":439},"            break",[63,14378,274],{"class":91},[63,14380,14381],{"class":65,"line":374},[63,14382,7517],{"class":91},[63,14384,14385],{"class":65,"line":387},[63,14386,14387],{"class":91},"        \n",[63,14389,14390,14393,14395,14397,14399,14401,14403,14405,14407,14409,14412,14414],{"class":65,"line":392},[63,14391,14392],{"class":87},"        result",[63,14394,92],{"class":91},[63,14396,357],{"class":95},[63,14398,142],{"class":91},[63,14400,856],{"class":439},[63,14402,92],{"class":91},[63,14404,2986],{"class":95},[63,14406,142],{"class":91},[63,14408,63],{"class":87},[63,14410,14411],{"class":91},"[..",[63,14413,14336],{"class":528},[63,14415,14416],{"class":91},"]));\n",[63,14418,14419,14422,14424,14426,14429,14431,14433,14435],{"class":65,"line":406},[63,14420,14421],{"class":528},"        span",[63,14423,133],{"class":132},[63,14425,14017],{"class":87},[63,14427,14428],{"class":91},"[(",[63,14430,14336],{"class":528},[63,14432,6165],{"class":132},[63,14434,887],{"class":289},[63,14436,14437],{"class":91},")..];\n",[63,14439,14440],{"class":65,"line":2931},[63,14441,621],{"class":91},[63,14443,14444],{"class":65,"line":2937},[63,14445,14282],{"class":91},[63,14447,14448,14450,14452],{"class":65,"line":2956},[63,14449,1890],{"class":439},[63,14451,4879],{"class":528},[63,14453,274],{"class":91},[63,14455,14456],{"class":65,"line":2961},[63,14457,626],{"class":91},[16,14459,14095],{},[16,14461,14462],{},[7330,14463],{"alt":14464,"src":14465},"image-20250218-020623.png","\u002Fblog\u002Freduce-memory-footprint-and-garbage-collection-pressure\u002Fimage_20250218_020623.png",[11,14467,14469],{"id":14468},"string-parsing","String Parsing",[16,14471,13836],{},[54,14473,14475],{"className":78,"code":14474,"language":80,"meta":59,"style":59},"private const string searchString = \"Unique\";\n    \nprivate readonly string sourceString = Random.Shared.Next(10, 51)\n    .ToString()\n    .PadLeft(50, Random.Shared.Next(9, 12).ToString()[0])\n    .Insert(Random.Shared.Next(35), searchString);\n",[32,14476,14477,14495,14499,14536,14544,14590],{"__ignoreMap":59},[63,14478,14479,14481,14483,14485,14488,14490,14493],{"class":65,"line":66},[63,14480,5789],{"class":439},[63,14482,13848],{"class":439},[63,14484,2766],{"class":439},[63,14486,14487],{"class":528}," searchString",[63,14489,133],{"class":132},[63,14491,14492],{"class":145}," \"Unique\"",[63,14494,274],{"class":91},[63,14496,14497],{"class":65,"line":115},[63,14498,14282],{"class":91},[63,14500,14501,14503,14505,14507,14510,14512,14515,14517,14520,14522,14525,14527,14529,14531,14534],{"class":65,"line":121},[63,14502,5789],{"class":439},[63,14504,2970],{"class":439},[63,14506,2766],{"class":439},[63,14508,14509],{"class":528}," sourceString",[63,14511,133],{"class":132},[63,14513,14514],{"class":87}," Random",[63,14516,92],{"class":91},[63,14518,14519],{"class":87},"Shared",[63,14521,92],{"class":91},[63,14523,14524],{"class":95},"Next",[63,14526,142],{"class":91},[63,14528,13929],{"class":289},[63,14530,508],{"class":91},[63,14532,14533],{"class":289},"51",[63,14535,474],{"class":91},[63,14537,14538,14540,14542],{"class":65,"line":152},[63,14539,1111],{"class":91},[63,14541,2232],{"class":95},[63,14543,5131],{"class":91},[63,14545,14546,14548,14551,14553,14556,14558,14561,14563,14565,14567,14569,14571,14574,14576,14579,14581,14583,14586,14588],{"class":65,"line":253},[63,14547,1111],{"class":91},[63,14549,14550],{"class":95},"PadLeft",[63,14552,142],{"class":91},[63,14554,14555],{"class":289},"50",[63,14557,508],{"class":91},[63,14559,14560],{"class":87},"Random",[63,14562,92],{"class":91},[63,14564,14519],{"class":87},[63,14566,92],{"class":91},[63,14568,14524],{"class":95},[63,14570,142],{"class":91},[63,14572,14573],{"class":289},"9",[63,14575,508],{"class":91},[63,14577,14578],{"class":289},"12",[63,14580,6359],{"class":91},[63,14582,2232],{"class":95},[63,14584,14585],{"class":91},"()[",[63,14587,867],{"class":289},[63,14589,5853],{"class":91},[63,14591,14592,14594,14597,14599,14601,14603,14605,14607,14609,14611,14614,14617,14620],{"class":65,"line":277},[63,14593,1111],{"class":91},[63,14595,14596],{"class":95},"Insert",[63,14598,142],{"class":91},[63,14600,14560],{"class":87},[63,14602,92],{"class":91},[63,14604,14519],{"class":87},[63,14606,92],{"class":91},[63,14608,14524],{"class":95},[63,14610,142],{"class":91},[63,14612,14613],{"class":289},"35",[63,14615,14616],{"class":91},"), ",[63,14618,14619],{"class":528},"searchString",[63,14621,149],{"class":91},[16,14623,13863],{},[54,14625,14627],{"className":78,"code":14626,"language":80,"meta":59,"style":59},"public bool Traditional()\n{\n    return sourceString.Contains(searchString, StringComparison.OrdinalIgnoreCase);\n}\n",[32,14628,14629,14640,14644,14671],{"__ignoreMap":59},[63,14630,14631,14633,14636,14638],{"class":65,"line":66},[63,14632,440],{"class":439},[63,14634,14635],{"class":439}," bool",[63,14637,13877],{"class":95},[63,14639,5131],{"class":91},[63,14641,14642],{"class":65,"line":115},[63,14643,118],{"class":91},[63,14645,14646,14648,14650,14652,14655,14657,14659,14661,14664,14666,14669],{"class":65,"line":121},[63,14647,1890],{"class":439},[63,14649,14509],{"class":87},[63,14651,92],{"class":91},[63,14653,14654],{"class":95},"Contains",[63,14656,142],{"class":91},[63,14658,14619],{"class":528},[63,14660,508],{"class":91},[63,14662,14663],{"class":87},"StringComparison",[63,14665,92],{"class":91},[63,14667,14668],{"class":87},"OrdinalIgnoreCase",[63,14670,149],{"class":91},[63,14672,14673],{"class":65,"line":152},[63,14674,626],{"class":91},[16,14676,13954],{},[54,14678,14680],{"className":78,"code":14679,"language":80,"meta":59,"style":59},"public bool Span()\n{\n    ReadOnlySpan\u003Cchar> sourceSpan = sourceString.AsSpan();\n    ReadOnlySpan\u003Cchar> searchSpan = searchString.AsSpan();\n    \n    return sourceSpan.Contains(searchSpan, StringComparison.OrdinalIgnoreCase);\n}\n",[32,14681,14682,14692,14696,14719,14742,14746,14771],{"__ignoreMap":59},[63,14683,14684,14686,14688,14690],{"class":65,"line":66},[63,14685,440],{"class":439},[63,14687,14635],{"class":439},[63,14689,13968],{"class":95},[63,14691,5131],{"class":91},[63,14693,14694],{"class":65,"line":115},[63,14695,118],{"class":91},[63,14697,14698,14700,14702,14704,14706,14709,14711,14713,14715,14717],{"class":65,"line":121},[63,14699,13979],{"class":102},[63,14701,99],{"class":91},[63,14703,13984],{"class":439},[63,14705,1847],{"class":91},[63,14707,14708],{"class":528},"sourceSpan",[63,14710,133],{"class":132},[63,14712,14509],{"class":87},[63,14714,92],{"class":91},[63,14716,13997],{"class":95},[63,14718,403],{"class":91},[63,14720,14721,14723,14725,14727,14729,14732,14734,14736,14738,14740],{"class":65,"line":152},[63,14722,13979],{"class":102},[63,14724,99],{"class":91},[63,14726,13984],{"class":439},[63,14728,1847],{"class":91},[63,14730,14731],{"class":528},"searchSpan",[63,14733,133],{"class":132},[63,14735,14487],{"class":87},[63,14737,92],{"class":91},[63,14739,13997],{"class":95},[63,14741,403],{"class":91},[63,14743,14744],{"class":65,"line":253},[63,14745,14282],{"class":91},[63,14747,14748,14750,14753,14755,14757,14759,14761,14763,14765,14767,14769],{"class":65,"line":277},[63,14749,1890],{"class":439},[63,14751,14752],{"class":87}," sourceSpan",[63,14754,92],{"class":91},[63,14756,14654],{"class":95},[63,14758,142],{"class":91},[63,14760,14731],{"class":528},[63,14762,508],{"class":91},[63,14764,14663],{"class":87},[63,14766,92],{"class":91},[63,14768,14668],{"class":87},[63,14770,149],{"class":91},[63,14772,14773],{"class":65,"line":295},[63,14774,626],{"class":91},[16,14776,14095],{},[16,14778,14779],{},[7330,14780],{"alt":14781,"src":14782},"image-20250218-020846.png","\u002Fblog\u002Freduce-memory-footprint-and-garbage-collection-pressure\u002Fimage_20250218_020846.png",[11,14784,14786],{"id":14785},"string-create","String Create",[16,14788,14789],{},"We will benchmark concatenation of a bunch of strings of size 3 to 8. This is the code",[54,14791,14793],{"className":78,"code":14792,"language":80,"meta":59,"style":59},"using System.Text;\n\nnamespace Performance.Benchmarks;\n\n[MemoryDiagnoser]\npublic class StringCreateBenchmarks\n{\n    private List\u003Cstring> _strings = [];\n\n    private const string _separator = \",\";\n\n    [Params(5, 25, 100, 500)]\n    public int ListSize { get; set; }\n\n    [IterationSetup]\n    public void Init()\n    {\n        _strings = GenerateRandomWords(ListSize);\n    }\n    \n    [Benchmark]\n    public string StringConcat()\n    {\n        var result = string.Empty;\n        for (var i = 0; i \u003C _strings.Count; i++)\n        {\n            result += _strings[i];\n            if (i \u003C _strings.Count - 1)\n                result += _separator;\n        }\n\n        return result;\n    }\n    \n    [Benchmark]\n    public string StringJoin()\n    {\n        return string.Join(',', _strings);\n    }\n    \n    [Benchmark]\n    public string StringBuilder()\n    {\n        var sb = new StringBuilder();\n        for (var i = 0; i \u003C _strings.Count; i++)\n        {\n            sb.Append(_strings[i]);\n            if (i \u003C _strings.Count - 1)\n            {\n                sb.Append(_separator);\n            }\n        }\n\n        return sb.ToString();\n    }\n    \n    [Benchmark]\n    public string StringCreate()\n    {\n        var totalSize = 0;\n        for (var i = 0; i \u003C _strings.Count; i++)\n        {\n            totalSize += _strings[i].Length;\n        }\n        \n        totalSize += _separator.Length * _strings.Count - 1;\n        \n        return string.Create(totalSize, (_strings, _separator), (chars, state) =>\n        {\n            var offset = 0;\n                \n            var separatorSpan = state._separator.AsSpan();\n            for(var i = 0; i \u003C state._strings.Count; i++)\n            {\n                var currentStr = state._strings[i];                    \n                currentStr.AsSpan().CopyTo(chars[offset..]);\n                offset += currentStr.Length;\n                \n                if (i \u003C state._strings.Count - 1)\n                {\n                    separatorSpan.CopyTo(chars[offset..]);\n                    offset += state._separator.Length;\n                }\n            }\n        });\n    }\n\n    private static readonly Random r = new();\n    private static List\u003Cstring> GenerateRandomWords(int count) =>\n    [\n        ..Enumerable.Range(0, count)\n            .Select(_ => \n                new string(Enumerable.Range(0, r.Next(3, 8))\n                    .Select(_ => (char)r.Next('a', 'z' + 1))\n                    .ToArray()))\n    ];\n}\n",[32,14794,14795,14809,14813,14828,14832,14841,14850,14854,14874,14878,14896,14900,14926,14948,14952,14961,14973,14977,14994,14998,15002,15011,15022,15026,15042,15078,15082,15096,15118,15129,15133,15137,15145,15149,15153,15161,15172,15176,15197,15201,15205,15213,15224,15228,15244,15278,15282,15302,15325,15330,15347,15352,15357,15362,15375,15380,15385,15394,15406,15411,15425,15460,15465,15486,15491,15496,15524,15529,15566,15571,15585,15591,15614,15654,15659,15682,15709,15725,15730,15758,15764,15784,15804,15810,15815,15821,15826,15831,15850,15877,15883,15908,15923,15963,16004,16014,16020],{"__ignoreMap":59},[63,14796,14797,14799,14802,14804,14807],{"class":65,"line":66},[63,14798,1687],{"class":439},[63,14800,14801],{"class":102}," System",[63,14803,92],{"class":91},[63,14805,14806],{"class":102},"Text",[63,14808,274],{"class":91},[63,14810,14811],{"class":65,"line":115},[63,14812,588],{"emptyLinePlaceholder":587},[63,14814,14815,14818,14821,14823,14826],{"class":65,"line":121},[63,14816,14817],{"class":439},"namespace",[63,14819,14820],{"class":102}," Performance",[63,14822,92],{"class":91},[63,14824,14825],{"class":102},"Benchmarks",[63,14827,274],{"class":91},[63,14829,14830],{"class":65,"line":152},[63,14831,588],{"emptyLinePlaceholder":587},[63,14833,14834,14836,14839],{"class":65,"line":253},[63,14835,4353],{"class":91},[63,14837,14838],{"class":102},"MemoryDiagnoser",[63,14840,5150],{"class":91},[63,14842,14843,14845,14847],{"class":65,"line":277},[63,14844,440],{"class":439},[63,14846,446],{"class":439},[63,14848,14849],{"class":102}," StringCreateBenchmarks\n",[63,14851,14852],{"class":65,"line":295},[63,14853,118],{"class":91},[63,14855,14856,14858,14860,14862,14864,14866,14869,14871],{"class":65,"line":301},[63,14857,2964],{"class":439},[63,14859,14144],{"class":102},[63,14861,99],{"class":91},[63,14863,502],{"class":439},[63,14865,1847],{"class":91},[63,14867,14868],{"class":2976},"_strings",[63,14870,133],{"class":132},[63,14872,14873],{"class":91}," [];\n",[63,14875,14876],{"class":65,"line":313},[63,14877,588],{"emptyLinePlaceholder":587},[63,14879,14880,14882,14884,14886,14889,14891,14894],{"class":65,"line":318},[63,14881,2964],{"class":439},[63,14883,13848],{"class":439},[63,14885,2766],{"class":439},[63,14887,14888],{"class":2976}," _separator",[63,14890,133],{"class":132},[63,14892,14893],{"class":145}," \",\"",[63,14895,274],{"class":91},[63,14897,14898],{"class":65,"line":340},[63,14899,588],{"emptyLinePlaceholder":587},[63,14901,14902,14904,14907,14909,14911,14913,14916,14918,14920,14922,14924],{"class":65,"line":369},[63,14903,456],{"class":91},[63,14905,14906],{"class":102},"Params",[63,14908,142],{"class":91},[63,14910,1596],{"class":289},[63,14912,508],{"class":91},[63,14914,14915],{"class":289},"25",[63,14917,508],{"class":91},[63,14919,1176],{"class":289},[63,14921,508],{"class":91},[63,14923,2900],{"class":289},[63,14925,12602],{"class":91},[63,14927,14928,14930,14933,14936,14938,14941,14943,14946],{"class":65,"line":374},[63,14929,483],{"class":439},[63,14931,14932],{"class":439}," int",[63,14934,14935],{"class":2769}," ListSize",[63,14937,3486],{"class":91},[63,14939,14940],{"class":439},"get",[63,14942,6016],{"class":91},[63,14944,14945],{"class":439},"set",[63,14947,3446],{"class":91},[63,14949,14950],{"class":65,"line":387},[63,14951,588],{"emptyLinePlaceholder":587},[63,14953,14954,14956,14959],{"class":65,"line":392},[63,14955,456],{"class":91},[63,14957,14958],{"class":102},"IterationSetup",[63,14960,5150],{"class":91},[63,14962,14963,14965,14968,14971],{"class":65,"line":406},[63,14964,483],{"class":439},[63,14966,14967],{"class":439}," void",[63,14969,14970],{"class":95}," Init",[63,14972,5131],{"class":91},[63,14974,14975],{"class":65,"line":2931},[63,14976,250],{"class":91},[63,14978,14979,14982,14984,14987,14989,14992],{"class":65,"line":2937},[63,14980,14981],{"class":528},"        _strings",[63,14983,133],{"class":132},[63,14985,14986],{"class":95}," GenerateRandomWords",[63,14988,142],{"class":91},[63,14990,14991],{"class":528},"ListSize",[63,14993,149],{"class":91},[63,14995,14996],{"class":65,"line":2956},[63,14997,621],{"class":91},[63,14999,15000],{"class":65,"line":2961},[63,15001,14282],{"class":91},[63,15003,15004,15006,15009],{"class":65,"line":3000},[63,15005,456],{"class":91},[63,15007,15008],{"class":102},"Benchmark",[63,15010,5150],{"class":91},[63,15012,15013,15015,15017,15020],{"class":65,"line":3018},[63,15014,483],{"class":439},[63,15016,2766],{"class":439},[63,15018,15019],{"class":95}," StringConcat",[63,15021,5131],{"class":91},[63,15023,15024],{"class":65,"line":3037},[63,15025,250],{"class":91},[63,15027,15028,15030,15032,15034,15036,15038,15040],{"class":65,"line":3056},[63,15029,525],{"class":439},[63,15031,4879],{"class":528},[63,15033,133],{"class":132},[63,15035,2766],{"class":439},[63,15037,92],{"class":91},[63,15039,2331],{"class":87},[63,15041,274],{"class":91},[63,15043,15044,15047,15049,15051,15053,15055,15057,15059,15061,15063,15066,15068,15070,15072,15074,15076],{"class":65,"line":5491},[63,15045,15046],{"class":439},"        for",[63,15048,3366],{"class":91},[63,15050,2067],{"class":439},[63,15052,6009],{"class":528},[63,15054,133],{"class":132},[63,15056,4464],{"class":289},[63,15058,6016],{"class":91},[63,15060,5610],{"class":528},[63,15062,6021],{"class":132},[63,15064,15065],{"class":87}," _strings",[63,15067,92],{"class":91},[63,15069,6898],{"class":87},[63,15071,6016],{"class":91},[63,15073,5610],{"class":528},[63,15075,6031],{"class":132},[63,15077,474],{"class":91},[63,15079,15080],{"class":65,"line":5515},[63,15081,1953],{"class":91},[63,15083,15084,15086,15088,15090,15092,15094],{"class":65,"line":5520},[63,15085,14353],{"class":528},[63,15087,4479],{"class":4478},[63,15089,15065],{"class":87},[63,15091,4353],{"class":91},[63,15093,5610],{"class":528},[63,15095,5867],{"class":91},[63,15097,15098,15100,15102,15104,15106,15108,15110,15112,15114,15116],{"class":65,"line":5545},[63,15099,7439],{"class":439},[63,15101,3366],{"class":91},[63,15103,5610],{"class":528},[63,15105,6021],{"class":132},[63,15107,15065],{"class":87},[63,15109,92],{"class":91},[63,15111,6898],{"class":87},[63,15113,4363],{"class":132},[63,15115,887],{"class":289},[63,15117,474],{"class":91},[63,15119,15120,15123,15125,15127],{"class":65,"line":5550},[63,15121,15122],{"class":528},"                result",[63,15124,4479],{"class":4478},[63,15126,14888],{"class":528},[63,15128,274],{"class":91},[63,15130,15131],{"class":65,"line":8275},[63,15132,7517],{"class":91},[63,15134,15135],{"class":65,"line":8280},[63,15136,588],{"emptyLinePlaceholder":587},[63,15138,15139,15141,15143],{"class":65,"line":8285},[63,15140,593],{"class":439},[63,15142,4879],{"class":528},[63,15144,274],{"class":91},[63,15146,15147],{"class":65,"line":8290},[63,15148,621],{"class":91},[63,15150,15151],{"class":65,"line":8299},[63,15152,14282],{"class":91},[63,15154,15155,15157,15159],{"class":65,"line":8304},[63,15156,456],{"class":91},[63,15158,15008],{"class":102},[63,15160,5150],{"class":91},[63,15162,15163,15165,15167,15170],{"class":65,"line":8309},[63,15164,483],{"class":439},[63,15166,2766],{"class":439},[63,15168,15169],{"class":95}," StringJoin",[63,15171,5131],{"class":91},[63,15173,15174],{"class":65,"line":8321},[63,15175,250],{"class":91},[63,15177,15178,15180,15182,15184,15187,15189,15191,15193,15195],{"class":65,"line":13006},[63,15179,593],{"class":439},[63,15181,2766],{"class":439},[63,15183,92],{"class":91},[63,15185,15186],{"class":95},"Join",[63,15188,142],{"class":91},[63,15190,14175],{"class":145},[63,15192,508],{"class":91},[63,15194,14868],{"class":528},[63,15196,149],{"class":91},[63,15198,15199],{"class":65,"line":13011},[63,15200,621],{"class":91},[63,15202,15203],{"class":65,"line":13025},[63,15204,14282],{"class":91},[63,15206,15207,15209,15211],{"class":65,"line":13030},[63,15208,456],{"class":91},[63,15210,15008],{"class":102},[63,15212,5150],{"class":91},[63,15214,15215,15217,15219,15222],{"class":65,"line":13044},[63,15216,483],{"class":439},[63,15218,2766],{"class":439},[63,15220,15221],{"class":95}," StringBuilder",[63,15223,5131],{"class":91},[63,15225,15226],{"class":65,"line":13049},[63,15227,250],{"class":91},[63,15229,15230,15232,15235,15237,15239,15242],{"class":65,"line":13080},[63,15231,525],{"class":439},[63,15233,15234],{"class":528}," sb",[63,15236,133],{"class":132},[63,15238,136],{"class":91},[63,15240,15241],{"class":102},"StringBuilder",[63,15243,403],{"class":91},[63,15245,15246,15248,15250,15252,15254,15256,15258,15260,15262,15264,15266,15268,15270,15272,15274,15276],{"class":65,"line":13085},[63,15247,15046],{"class":439},[63,15249,3366],{"class":91},[63,15251,2067],{"class":439},[63,15253,6009],{"class":528},[63,15255,133],{"class":132},[63,15257,4464],{"class":289},[63,15259,6016],{"class":91},[63,15261,5610],{"class":528},[63,15263,6021],{"class":132},[63,15265,15065],{"class":87},[63,15267,92],{"class":91},[63,15269,6898],{"class":87},[63,15271,6016],{"class":91},[63,15273,5610],{"class":528},[63,15275,6031],{"class":132},[63,15277,474],{"class":91},[63,15279,15280],{"class":65,"line":13090},[63,15281,1953],{"class":91},[63,15283,15285,15288,15290,15292,15294,15296,15298,15300],{"class":65,"line":15284},47,[63,15286,15287],{"class":87},"            sb",[63,15289,92],{"class":91},[63,15291,7101],{"class":95},[63,15293,142],{"class":91},[63,15295,14868],{"class":87},[63,15297,4353],{"class":91},[63,15299,5610],{"class":528},[63,15301,5044],{"class":91},[63,15303,15305,15307,15309,15311,15313,15315,15317,15319,15321,15323],{"class":65,"line":15304},48,[63,15306,7439],{"class":439},[63,15308,3366],{"class":91},[63,15310,5610],{"class":528},[63,15312,6021],{"class":132},[63,15314,15065],{"class":87},[63,15316,92],{"class":91},[63,15318,6898],{"class":87},[63,15320,4363],{"class":132},[63,15322,887],{"class":289},[63,15324,474],{"class":91},[63,15326,15328],{"class":65,"line":15327},49,[63,15329,7479],{"class":91},[63,15331,15333,15336,15338,15340,15342,15345],{"class":65,"line":15332},50,[63,15334,15335],{"class":87},"                sb",[63,15337,92],{"class":91},[63,15339,7101],{"class":95},[63,15341,142],{"class":91},[63,15343,15344],{"class":528},"_separator",[63,15346,149],{"class":91},[63,15348,15350],{"class":65,"line":15349},51,[63,15351,7499],{"class":91},[63,15353,15355],{"class":65,"line":15354},52,[63,15356,7517],{"class":91},[63,15358,15360],{"class":65,"line":15359},53,[63,15361,588],{"emptyLinePlaceholder":587},[63,15363,15365,15367,15369,15371,15373],{"class":65,"line":15364},54,[63,15366,593],{"class":439},[63,15368,15234],{"class":87},[63,15370,92],{"class":91},[63,15372,2232],{"class":95},[63,15374,403],{"class":91},[63,15376,15378],{"class":65,"line":15377},55,[63,15379,621],{"class":91},[63,15381,15383],{"class":65,"line":15382},56,[63,15384,14282],{"class":91},[63,15386,15388,15390,15392],{"class":65,"line":15387},57,[63,15389,456],{"class":91},[63,15391,15008],{"class":102},[63,15393,5150],{"class":91},[63,15395,15397,15399,15401,15404],{"class":65,"line":15396},58,[63,15398,483],{"class":439},[63,15400,2766],{"class":439},[63,15402,15403],{"class":95}," StringCreate",[63,15405,5131],{"class":91},[63,15407,15409],{"class":65,"line":15408},59,[63,15410,250],{"class":91},[63,15412,15414,15416,15419,15421,15423],{"class":65,"line":15413},60,[63,15415,525],{"class":439},[63,15417,15418],{"class":528}," totalSize",[63,15420,133],{"class":132},[63,15422,4464],{"class":289},[63,15424,274],{"class":91},[63,15426,15428,15430,15432,15434,15436,15438,15440,15442,15444,15446,15448,15450,15452,15454,15456,15458],{"class":65,"line":15427},61,[63,15429,15046],{"class":439},[63,15431,3366],{"class":91},[63,15433,2067],{"class":439},[63,15435,6009],{"class":528},[63,15437,133],{"class":132},[63,15439,4464],{"class":289},[63,15441,6016],{"class":91},[63,15443,5610],{"class":528},[63,15445,6021],{"class":132},[63,15447,15065],{"class":87},[63,15449,92],{"class":91},[63,15451,6898],{"class":87},[63,15453,6016],{"class":91},[63,15455,5610],{"class":528},[63,15457,6031],{"class":132},[63,15459,474],{"class":91},[63,15461,15463],{"class":65,"line":15462},62,[63,15464,1953],{"class":91},[63,15466,15468,15471,15473,15475,15477,15479,15482,15484],{"class":65,"line":15467},63,[63,15469,15470],{"class":528},"            totalSize",[63,15472,4479],{"class":4478},[63,15474,15065],{"class":87},[63,15476,4353],{"class":91},[63,15478,5610],{"class":528},[63,15480,15481],{"class":91},"].",[63,15483,12451],{"class":87},[63,15485,274],{"class":91},[63,15487,15489],{"class":65,"line":15488},64,[63,15490,7517],{"class":91},[63,15492,15494],{"class":65,"line":15493},65,[63,15495,14387],{"class":91},[63,15497,15499,15502,15504,15506,15508,15510,15512,15514,15516,15518,15520,15522],{"class":65,"line":15498},66,[63,15500,15501],{"class":528},"        totalSize",[63,15503,4479],{"class":4478},[63,15505,14888],{"class":87},[63,15507,92],{"class":91},[63,15509,12451],{"class":87},[63,15511,5198],{"class":132},[63,15513,15065],{"class":87},[63,15515,92],{"class":91},[63,15517,6898],{"class":87},[63,15519,4363],{"class":132},[63,15521,887],{"class":289},[63,15523,274],{"class":91},[63,15525,15527],{"class":65,"line":15526},67,[63,15528,14387],{"class":91},[63,15530,15532,15534,15536,15538,15540,15542,15545,15547,15549,15551,15553,15556,15559,15561,15564],{"class":65,"line":15531},68,[63,15533,593],{"class":439},[63,15535,2766],{"class":439},[63,15537,92],{"class":91},[63,15539,1968],{"class":95},[63,15541,142],{"class":91},[63,15543,15544],{"class":528},"totalSize",[63,15546,7575],{"class":91},[63,15548,14868],{"class":528},[63,15550,508],{"class":91},[63,15552,15344],{"class":528},[63,15554,15555],{"class":91},"), (",[63,15557,15558],{"class":87},"chars",[63,15560,508],{"class":91},[63,15562,15563],{"class":87},"state",[63,15565,1910],{"class":91},[63,15567,15569],{"class":65,"line":15568},69,[63,15570,1953],{"class":91},[63,15572,15574,15576,15579,15581,15583],{"class":65,"line":15573},70,[63,15575,8045],{"class":439},[63,15577,15578],{"class":528}," offset",[63,15580,133],{"class":132},[63,15582,4464],{"class":289},[63,15584,274],{"class":91},[63,15586,15588],{"class":65,"line":15587},71,[63,15589,15590],{"class":91},"                \n",[63,15592,15594,15596,15599,15601,15604,15606,15608,15610,15612],{"class":65,"line":15593},72,[63,15595,8045],{"class":439},[63,15597,15598],{"class":528}," separatorSpan",[63,15600,133],{"class":132},[63,15602,15603],{"class":87}," state",[63,15605,92],{"class":91},[63,15607,15344],{"class":87},[63,15609,92],{"class":91},[63,15611,13997],{"class":95},[63,15613,403],{"class":91},[63,15615,15617,15620,15622,15624,15626,15628,15630,15632,15634,15636,15638,15640,15642,15644,15646,15648,15650,15652],{"class":65,"line":15616},73,[63,15618,15619],{"class":439},"            for",[63,15621,142],{"class":91},[63,15623,2067],{"class":439},[63,15625,6009],{"class":528},[63,15627,133],{"class":132},[63,15629,4464],{"class":289},[63,15631,6016],{"class":91},[63,15633,5610],{"class":528},[63,15635,6021],{"class":132},[63,15637,15603],{"class":87},[63,15639,92],{"class":91},[63,15641,14868],{"class":87},[63,15643,92],{"class":91},[63,15645,6898],{"class":87},[63,15647,6016],{"class":91},[63,15649,5610],{"class":528},[63,15651,6031],{"class":132},[63,15653,474],{"class":91},[63,15655,15657],{"class":65,"line":15656},74,[63,15658,7479],{"class":91},[63,15660,15662,15664,15667,15669,15671,15673,15675,15677,15679],{"class":65,"line":15661},75,[63,15663,8105],{"class":439},[63,15665,15666],{"class":528}," currentStr",[63,15668,133],{"class":132},[63,15670,15603],{"class":87},[63,15672,92],{"class":91},[63,15674,14868],{"class":87},[63,15676,4353],{"class":91},[63,15678,5610],{"class":528},[63,15680,15681],{"class":91},"];                    \n",[63,15683,15685,15688,15690,15692,15695,15697,15699,15701,15703,15706],{"class":65,"line":15684},76,[63,15686,15687],{"class":87},"                currentStr",[63,15689,92],{"class":91},[63,15691,13997],{"class":95},[63,15693,15694],{"class":91},"().",[63,15696,12839],{"class":95},[63,15698,142],{"class":91},[63,15700,15558],{"class":87},[63,15702,4353],{"class":91},[63,15704,15705],{"class":528},"offset",[63,15707,15708],{"class":91},"..]);\n",[63,15710,15712,15715,15717,15719,15721,15723],{"class":65,"line":15711},77,[63,15713,15714],{"class":528},"                offset",[63,15716,4479],{"class":4478},[63,15718,15666],{"class":87},[63,15720,92],{"class":91},[63,15722,12451],{"class":87},[63,15724,274],{"class":91},[63,15726,15728],{"class":65,"line":15727},78,[63,15729,15590],{"class":91},[63,15731,15733,15736,15738,15740,15742,15744,15746,15748,15750,15752,15754,15756],{"class":65,"line":15732},79,[63,15734,15735],{"class":439},"                if",[63,15737,3366],{"class":91},[63,15739,5610],{"class":528},[63,15741,6021],{"class":132},[63,15743,15603],{"class":87},[63,15745,92],{"class":91},[63,15747,14868],{"class":87},[63,15749,92],{"class":91},[63,15751,6898],{"class":87},[63,15753,4363],{"class":132},[63,15755,887],{"class":289},[63,15757,474],{"class":91},[63,15759,15761],{"class":65,"line":15760},80,[63,15762,15763],{"class":91},"                {\n",[63,15765,15767,15770,15772,15774,15776,15778,15780,15782],{"class":65,"line":15766},81,[63,15768,15769],{"class":87},"                    separatorSpan",[63,15771,92],{"class":91},[63,15773,12839],{"class":95},[63,15775,142],{"class":91},[63,15777,15558],{"class":87},[63,15779,4353],{"class":91},[63,15781,15705],{"class":528},[63,15783,15708],{"class":91},[63,15785,15787,15790,15792,15794,15796,15798,15800,15802],{"class":65,"line":15786},82,[63,15788,15789],{"class":528},"                    offset",[63,15791,4479],{"class":4478},[63,15793,15603],{"class":87},[63,15795,92],{"class":91},[63,15797,15344],{"class":87},[63,15799,92],{"class":91},[63,15801,12451],{"class":87},[63,15803,274],{"class":91},[63,15805,15807],{"class":65,"line":15806},83,[63,15808,15809],{"class":91},"                }\n",[63,15811,15813],{"class":65,"line":15812},84,[63,15814,7499],{"class":91},[63,15816,15818],{"class":65,"line":15817},85,[63,15819,15820],{"class":91},"        });\n",[63,15822,15824],{"class":65,"line":15823},86,[63,15825,621],{"class":91},[63,15827,15829],{"class":65,"line":15828},87,[63,15830,588],{"emptyLinePlaceholder":587},[63,15832,15834,15836,15838,15840,15842,15845,15847],{"class":65,"line":15833},88,[63,15835,2964],{"class":439},[63,15837,2967],{"class":439},[63,15839,2970],{"class":439},[63,15841,14514],{"class":102},[63,15843,15844],{"class":2976}," r",[63,15846,133],{"class":132},[63,15848,15849],{"class":91}," new();\n",[63,15851,15853,15855,15857,15859,15861,15863,15865,15868,15870,15872,15875],{"class":65,"line":15852},89,[63,15854,2964],{"class":439},[63,15856,2967],{"class":439},[63,15858,14144],{"class":102},[63,15860,99],{"class":91},[63,15862,502],{"class":439},[63,15864,1847],{"class":91},[63,15866,15867],{"class":95},"GenerateRandomWords",[63,15869,142],{"class":91},[63,15871,856],{"class":439},[63,15873,15874],{"class":87}," count",[63,15876,1910],{"class":91},[63,15878,15880],{"class":65,"line":15879},90,[63,15881,15882],{"class":91},"    [\n",[63,15884,15886,15889,15892,15894,15897,15899,15901,15903,15906],{"class":65,"line":15885},91,[63,15887,15888],{"class":91},"        ..",[63,15890,15891],{"class":87},"Enumerable",[63,15893,92],{"class":91},[63,15895,15896],{"class":95},"Range",[63,15898,142],{"class":91},[63,15900,867],{"class":289},[63,15902,508],{"class":91},[63,15904,15905],{"class":528},"count",[63,15907,474],{"class":91},[63,15909,15911,15914,15916,15918,15920],{"class":65,"line":15910},92,[63,15912,15913],{"class":91},"            .",[63,15915,6347],{"class":95},[63,15917,142],{"class":91},[63,15919,13520],{"class":87},[63,15921,15922],{"class":91}," => \n",[63,15924,15926,15929,15931,15933,15935,15937,15939,15941,15943,15945,15948,15950,15952,15954,15956,15958,15961],{"class":65,"line":15925},93,[63,15927,15928],{"class":91},"                new ",[63,15930,502],{"class":439},[63,15932,142],{"class":91},[63,15934,15891],{"class":87},[63,15936,92],{"class":91},[63,15938,15896],{"class":95},[63,15940,142],{"class":91},[63,15942,867],{"class":289},[63,15944,508],{"class":91},[63,15946,15947],{"class":87},"r",[63,15949,92],{"class":91},[63,15951,14524],{"class":95},[63,15953,142],{"class":91},[63,15955,692],{"class":289},[63,15957,508],{"class":91},[63,15959,15960],{"class":289},"8",[63,15962,1106],{"class":91},[63,15964,15966,15969,15971,15973,15975,15978,15980,15982,15984,15986,15988,15990,15993,15995,15998,16000,16002],{"class":65,"line":15965},94,[63,15967,15968],{"class":91},"                    .",[63,15970,6347],{"class":95},[63,15972,142],{"class":91},[63,15974,13520],{"class":87},[63,15976,15977],{"class":91}," => (",[63,15979,13984],{"class":439},[63,15981,5359],{"class":91},[63,15983,15947],{"class":87},[63,15985,92],{"class":91},[63,15987,14524],{"class":95},[63,15989,142],{"class":91},[63,15991,15992],{"class":145},"'a'",[63,15994,508],{"class":91},[63,15996,15997],{"class":145},"'z'",[63,15999,6165],{"class":132},[63,16001,887],{"class":289},[63,16003,1106],{"class":91},[63,16005,16007,16009,16011],{"class":65,"line":16006},95,[63,16008,15968],{"class":91},[63,16010,6362],{"class":95},[63,16012,16013],{"class":91},"()))\n",[63,16015,16017],{"class":65,"line":16016},96,[63,16018,16019],{"class":91},"    ];\n",[63,16021,16023],{"class":65,"line":16022},97,[63,16024,626],{"class":91},[16,16026,14095],{},[16,16028,16029],{},[7330,16030],{"alt":16031,"src":16032},"image-20250218-022012.png","\u002Fblog\u002Freduce-memory-footprint-and-garbage-collection-pressure\u002Fimage_20250218_022012.png",[11,16034,16036],{"id":16035},"json-key","Json Key",[16,16038,16039],{},"Let's compare something as trivial as",[54,16041,16043],{"className":78,"code":16042,"language":80,"meta":59,"style":59},"JsonDocument::RootElement.TryGetProperty(\"MyKey\", out var _);\n",[32,16044,16045],{"__ignoreMap":59},[63,16046,16047,16050,16053,16056,16058,16061,16063,16066,16068,16070,16072,16075],{"class":65,"line":66},[63,16048,16049],{"class":528},"JsonDocument",[63,16051,16052],{"class":91},"::",[63,16054,16055],{"class":87},"RootElement",[63,16057,92],{"class":91},[63,16059,16060],{"class":95},"TryGetProperty",[63,16062,142],{"class":91},[63,16064,16065],{"class":145},"\"MyKey\"",[63,16067,508],{"class":91},[63,16069,7467],{"class":439},[63,16071,1690],{"class":439},[63,16073,16074],{"class":528}," _",[63,16076,149],{"class":91},[16,16078,13954],{},[54,16080,16082],{"className":78,"code":16081,"language":80,"meta":59,"style":59},"JsonDocument::RootElement.TryGetProperty(\"MyKey\"u8, out var _);\n",[32,16083,16084],{"__ignoreMap":59},[63,16085,16086,16088,16090,16092,16094,16096,16098,16100,16103,16105,16107,16109,16111],{"class":65,"line":66},[63,16087,16049],{"class":528},[63,16089,16052],{"class":91},[63,16091,16055],{"class":87},[63,16093,92],{"class":91},[63,16095,16060],{"class":95},[63,16097,142],{"class":91},[63,16099,16065],{"class":145},[63,16101,16102],{"class":528},"u8",[63,16104,508],{"class":91},[63,16106,7467],{"class":439},[63,16108,1690],{"class":439},[63,16110,16074],{"class":528},[63,16112,149],{"class":91},[16,16114,14095],{},[16,16116,16117],{},[7330,16118],{"alt":16119,"src":16120},"image-20250218-023528.png","\u002Fblog\u002Freduce-memory-footprint-and-garbage-collection-pressure\u002Fimage_20250218_023528.png",[11,16122,16124],{"id":16123},"ii-tryformat-and-stackalloc","II. TryFormat and stackalloc",[16,16126,16127,16130],{},[32,16128,16129],{},"TryFormat()"," is a modern API for converting values to their string representations without allocating memory. Many built in types (numbers, dates, GUIDs, etc.) implement this method, allowing direct writing to a span buffer.",[54,16132,16134],{"className":78,"code":16133,"language":80,"meta":59,"style":59},"public string FormatCurrency(decimal amount)\n{\n    Span\u003Cchar> buffer = stackalloc char[32]; \u002F\u002F Stack-allocated buffer\n    if (amount.TryFormat(buffer, out int written, \"C2\"))\n    {\n        return new string(buffer[..written]);\n    }\n}\n",[32,16135,16136,16155,16159,16191,16225,16229,16248,16252],{"__ignoreMap":59},[63,16137,16138,16140,16142,16145,16147,16150,16153],{"class":65,"line":66},[63,16139,440],{"class":439},[63,16141,2766],{"class":439},[63,16143,16144],{"class":95}," FormatCurrency",[63,16146,142],{"class":91},[63,16148,16149],{"class":439},"decimal",[63,16151,16152],{"class":87}," amount",[63,16154,474],{"class":91},[63,16156,16157],{"class":65,"line":115},[63,16158,118],{"class":91},[63,16160,16161,16164,16166,16168,16170,16173,16175,16178,16180,16182,16185,16188],{"class":65,"line":121},[63,16162,16163],{"class":102},"    Span",[63,16165,99],{"class":91},[63,16167,13984],{"class":439},[63,16169,1847],{"class":91},[63,16171,16172],{"class":528},"buffer",[63,16174,133],{"class":132},[63,16176,16177],{"class":91}," stackalloc ",[63,16179,13984],{"class":439},[63,16181,4353],{"class":91},[63,16183,16184],{"class":289},"32",[63,16186,16187],{"class":91},"]; ",[63,16189,16190],{"class":2731},"\u002F\u002F Stack-allocated buffer\n",[63,16192,16193,16195,16197,16200,16202,16205,16207,16209,16211,16213,16215,16218,16220,16223],{"class":65,"line":152},[63,16194,3410],{"class":439},[63,16196,3366],{"class":91},[63,16198,16199],{"class":87},"amount",[63,16201,92],{"class":91},[63,16203,16204],{"class":95},"TryFormat",[63,16206,142],{"class":91},[63,16208,16172],{"class":528},[63,16210,508],{"class":91},[63,16212,7467],{"class":439},[63,16214,14932],{"class":439},[63,16216,16217],{"class":528}," written",[63,16219,508],{"class":91},[63,16221,16222],{"class":145},"\"C2\"",[63,16224,1106],{"class":91},[63,16226,16227],{"class":65,"line":253},[63,16228,250],{"class":91},[63,16230,16231,16233,16235,16237,16239,16241,16243,16246],{"class":65,"line":277},[63,16232,593],{"class":439},[63,16234,136],{"class":91},[63,16236,502],{"class":439},[63,16238,142],{"class":91},[63,16240,16172],{"class":87},[63,16242,14411],{"class":91},[63,16244,16245],{"class":528},"written",[63,16247,5044],{"class":91},[63,16249,16250],{"class":65,"line":295},[63,16251,621],{"class":91},[63,16253,16254],{"class":65,"line":301},[63,16255,626],{"class":91},[54,16257,16259],{"className":78,"code":16258,"language":80,"meta":59,"style":59},"public string FormatDateAndTime(DateTime dateTime)\n{\n    const string format = \"yyyy-MM-dd HH:mm:ss\";\n    Span\u003Cchar> buffer = stackalloc char[format.Length];\n    \n    if (dateTime.TryFormat(buffer, out int written, format))\n    {\n        return new string(buffer[..written]);\n    }\n    return dateTime.ToString(format);\n}\n",[32,16260,16261,16280,16284,16301,16330,16334,16365,16369,16387,16391,16407],{"__ignoreMap":59},[63,16262,16263,16265,16267,16270,16272,16275,16278],{"class":65,"line":66},[63,16264,440],{"class":439},[63,16266,2766],{"class":439},[63,16268,16269],{"class":95}," FormatDateAndTime",[63,16271,142],{"class":91},[63,16273,16274],{"class":102},"DateTime",[63,16276,16277],{"class":87}," dateTime",[63,16279,474],{"class":91},[63,16281,16282],{"class":65,"line":115},[63,16283,118],{"class":91},[63,16285,16286,16289,16291,16294,16296,16299],{"class":65,"line":121},[63,16287,16288],{"class":439},"    const",[63,16290,2766],{"class":439},[63,16292,16293],{"class":528}," format",[63,16295,133],{"class":132},[63,16297,16298],{"class":145}," \"yyyy-MM-dd HH:mm:ss\"",[63,16300,274],{"class":91},[63,16302,16303,16305,16307,16309,16311,16313,16315,16317,16319,16321,16324,16326,16328],{"class":65,"line":152},[63,16304,16163],{"class":102},[63,16306,99],{"class":91},[63,16308,13984],{"class":439},[63,16310,1847],{"class":91},[63,16312,16172],{"class":528},[63,16314,133],{"class":132},[63,16316,16177],{"class":91},[63,16318,13984],{"class":439},[63,16320,4353],{"class":91},[63,16322,16323],{"class":87},"format",[63,16325,92],{"class":91},[63,16327,12451],{"class":87},[63,16329,5867],{"class":91},[63,16331,16332],{"class":65,"line":253},[63,16333,14282],{"class":91},[63,16335,16336,16338,16340,16343,16345,16347,16349,16351,16353,16355,16357,16359,16361,16363],{"class":65,"line":277},[63,16337,3410],{"class":439},[63,16339,3366],{"class":91},[63,16341,16342],{"class":87},"dateTime",[63,16344,92],{"class":91},[63,16346,16204],{"class":95},[63,16348,142],{"class":91},[63,16350,16172],{"class":528},[63,16352,508],{"class":91},[63,16354,7467],{"class":439},[63,16356,14932],{"class":439},[63,16358,16217],{"class":528},[63,16360,508],{"class":91},[63,16362,16323],{"class":528},[63,16364,1106],{"class":91},[63,16366,16367],{"class":65,"line":295},[63,16368,250],{"class":91},[63,16370,16371,16373,16375,16377,16379,16381,16383,16385],{"class":65,"line":301},[63,16372,593],{"class":439},[63,16374,136],{"class":91},[63,16376,502],{"class":439},[63,16378,142],{"class":91},[63,16380,16172],{"class":87},[63,16382,14411],{"class":91},[63,16384,16245],{"class":528},[63,16386,5044],{"class":91},[63,16388,16389],{"class":65,"line":313},[63,16390,621],{"class":91},[63,16392,16393,16395,16397,16399,16401,16403,16405],{"class":65,"line":318},[63,16394,1890],{"class":439},[63,16396,16277],{"class":87},[63,16398,92],{"class":91},[63,16400,2232],{"class":95},[63,16402,142],{"class":91},[63,16404,16323],{"class":528},[63,16406,149],{"class":91},[63,16408,16409],{"class":65,"line":340},[63,16410,626],{"class":91},[16,16412,16413,16414,3246,16416,16419],{},"Let's benchmark ",[32,16415,16129],{},[32,16417,16418],{},"ToString()"," for int, double and decimal. This is the code we will use for benchmarking",[54,16421,16423],{"className":78,"code":16422,"language":80,"meta":59,"style":59},"namespace Performance.Benchmarks;\n\n[MemoryDiagnoser]\npublic class NumberFormatBenchmarks\n{\n    private const int IntValue = 12345;\n    private const double DoubleValue = 123.456;\n    private const decimal DecimalValue = 123.456m;\n    private readonly char[] _charBuffer = new char[100];\n    \n    [Benchmark]\n    public string Int_ToString()\n    {\n        return IntValue.ToString();\n    }\n\n    [Benchmark]\n    public bool Int_TryFormat()\n    {\n        return IntValue.TryFormat(_charBuffer, out _);\n    }\n\n    [Benchmark]\n    public string Int_ToString_WithFormat()\n    {\n        return IntValue.ToString(\"D8\");\n    }\n\n    [Benchmark]\n    public bool Int_TryFormat_WithFormat()\n    {\n        return IntValue.TryFormat(_charBuffer, out _, \"D8\");\n    }\n\n    [Benchmark]\n    public string Double_ToString()\n    {\n        return DoubleValue.ToString();\n    }\n\n    [Benchmark]\n    public bool Double_TryFormat()\n    {\n        return DoubleValue.TryFormat(_charBuffer, out _);\n    }\n\n    [Benchmark]\n    public string Double_ToString_WithFormat()\n    {\n        return DoubleValue.ToString(\"F2\");\n    }\n\n    [Benchmark]\n    public bool Double_TryFormat_WithFormat()\n    {\n        return DoubleValue.TryFormat(_charBuffer, out _, \"F2\");\n    }\n\n    [Benchmark]\n    public string Decimal_ToString()\n    {\n        return DecimalValue.ToString();\n    }\n\n    [Benchmark]\n    public bool Decimal_TryFormat()\n    {\n        return DecimalValue.TryFormat(_charBuffer, out _);\n    }\n\n    [Benchmark]\n    public string Decimal_ToString_WithFormat()\n    {\n        return DecimalValue.ToString(\"C\");\n    }\n\n    [Benchmark]\n    public bool Decimal_TryFormat_WithFormat()\n    {\n        return DecimalValue.TryFormat(_charBuffer, out _, \"C\");\n    }\n    \n    [Benchmark]\n    public string Double_ToString_Culture()\n    {\n        return DoubleValue.ToString(System.Globalization.CultureInfo.InvariantCulture);\n    }\n\n    [Benchmark]\n    public bool Double_TryFormat_Culture()\n    {\n        return DoubleValue.TryFormat(_charBuffer, out _, provider: System.Globalization.CultureInfo.InvariantCulture);\n    }\n}\n",[32,16424,16425,16437,16441,16449,16458,16462,16480,16498,16517,16543,16547,16555,16566,16570,16582,16586,16590,16598,16609,16613,16635,16639,16643,16651,16662,16666,16683,16687,16691,16699,16710,16714,16740,16744,16748,16756,16767,16771,16783,16787,16791,16799,16810,16814,16836,16840,16844,16852,16863,16867,16884,16888,16892,16900,16911,16915,16941,16945,16949,16957,16968,16972,16984,16988,16992,17000,17011,17015,17037,17041,17045,17053,17064,17068,17085,17089,17093,17101,17112,17116,17142,17146,17150,17158,17169,17173,17205,17209,17213,17221,17232,17236,17279,17283],{"__ignoreMap":59},[63,16426,16427,16429,16431,16433,16435],{"class":65,"line":66},[63,16428,14817],{"class":439},[63,16430,14820],{"class":102},[63,16432,92],{"class":91},[63,16434,14825],{"class":102},[63,16436,274],{"class":91},[63,16438,16439],{"class":65,"line":115},[63,16440,588],{"emptyLinePlaceholder":587},[63,16442,16443,16445,16447],{"class":65,"line":121},[63,16444,4353],{"class":91},[63,16446,14838],{"class":102},[63,16448,5150],{"class":91},[63,16450,16451,16453,16455],{"class":65,"line":152},[63,16452,440],{"class":439},[63,16454,446],{"class":439},[63,16456,16457],{"class":102}," NumberFormatBenchmarks\n",[63,16459,16460],{"class":65,"line":253},[63,16461,118],{"class":91},[63,16463,16464,16466,16468,16470,16473,16475,16478],{"class":65,"line":277},[63,16465,2964],{"class":439},[63,16467,13848],{"class":439},[63,16469,14932],{"class":439},[63,16471,16472],{"class":2976}," IntValue",[63,16474,133],{"class":132},[63,16476,16477],{"class":289}," 12345",[63,16479,274],{"class":91},[63,16481,16482,16484,16486,16488,16491,16493,16496],{"class":65,"line":295},[63,16483,2964],{"class":439},[63,16485,13848],{"class":439},[63,16487,4432],{"class":439},[63,16489,16490],{"class":2976}," DoubleValue",[63,16492,133],{"class":132},[63,16494,16495],{"class":289}," 123.456",[63,16497,274],{"class":91},[63,16499,16500,16502,16504,16507,16510,16512,16515],{"class":65,"line":301},[63,16501,2964],{"class":439},[63,16503,13848],{"class":439},[63,16505,16506],{"class":439}," decimal",[63,16508,16509],{"class":2976}," DecimalValue",[63,16511,133],{"class":132},[63,16513,16514],{"class":289}," 123.456m",[63,16516,274],{"class":91},[63,16518,16519,16521,16523,16526,16528,16531,16533,16535,16537,16539,16541],{"class":65,"line":313},[63,16520,2964],{"class":439},[63,16522,2970],{"class":439},[63,16524,16525],{"class":439}," char",[63,16527,4442],{"class":91},[63,16529,16530],{"class":2976},"_charBuffer",[63,16532,133],{"class":132},[63,16534,136],{"class":91},[63,16536,13984],{"class":439},[63,16538,4353],{"class":91},[63,16540,1176],{"class":289},[63,16542,5867],{"class":91},[63,16544,16545],{"class":65,"line":318},[63,16546,14282],{"class":91},[63,16548,16549,16551,16553],{"class":65,"line":340},[63,16550,456],{"class":91},[63,16552,15008],{"class":102},[63,16554,5150],{"class":91},[63,16556,16557,16559,16561,16564],{"class":65,"line":369},[63,16558,483],{"class":439},[63,16560,2766],{"class":439},[63,16562,16563],{"class":95}," Int_ToString",[63,16565,5131],{"class":91},[63,16567,16568],{"class":65,"line":374},[63,16569,250],{"class":91},[63,16571,16572,16574,16576,16578,16580],{"class":65,"line":387},[63,16573,593],{"class":439},[63,16575,16472],{"class":87},[63,16577,92],{"class":91},[63,16579,2232],{"class":95},[63,16581,403],{"class":91},[63,16583,16584],{"class":65,"line":392},[63,16585,621],{"class":91},[63,16587,16588],{"class":65,"line":406},[63,16589,588],{"emptyLinePlaceholder":587},[63,16591,16592,16594,16596],{"class":65,"line":2931},[63,16593,456],{"class":91},[63,16595,15008],{"class":102},[63,16597,5150],{"class":91},[63,16599,16600,16602,16604,16607],{"class":65,"line":2937},[63,16601,483],{"class":439},[63,16603,14635],{"class":439},[63,16605,16606],{"class":95}," Int_TryFormat",[63,16608,5131],{"class":91},[63,16610,16611],{"class":65,"line":2956},[63,16612,250],{"class":91},[63,16614,16615,16617,16619,16621,16623,16625,16627,16629,16631,16633],{"class":65,"line":2961},[63,16616,593],{"class":439},[63,16618,16472],{"class":87},[63,16620,92],{"class":91},[63,16622,16204],{"class":95},[63,16624,142],{"class":91},[63,16626,16530],{"class":528},[63,16628,508],{"class":91},[63,16630,7467],{"class":439},[63,16632,16074],{"class":528},[63,16634,149],{"class":91},[63,16636,16637],{"class":65,"line":3000},[63,16638,621],{"class":91},[63,16640,16641],{"class":65,"line":3018},[63,16642,588],{"emptyLinePlaceholder":587},[63,16644,16645,16647,16649],{"class":65,"line":3037},[63,16646,456],{"class":91},[63,16648,15008],{"class":102},[63,16650,5150],{"class":91},[63,16652,16653,16655,16657,16660],{"class":65,"line":3056},[63,16654,483],{"class":439},[63,16656,2766],{"class":439},[63,16658,16659],{"class":95}," Int_ToString_WithFormat",[63,16661,5131],{"class":91},[63,16663,16664],{"class":65,"line":5491},[63,16665,250],{"class":91},[63,16667,16668,16670,16672,16674,16676,16678,16681],{"class":65,"line":5515},[63,16669,593],{"class":439},[63,16671,16472],{"class":87},[63,16673,92],{"class":91},[63,16675,2232],{"class":95},[63,16677,142],{"class":91},[63,16679,16680],{"class":145},"\"D8\"",[63,16682,149],{"class":91},[63,16684,16685],{"class":65,"line":5520},[63,16686,621],{"class":91},[63,16688,16689],{"class":65,"line":5545},[63,16690,588],{"emptyLinePlaceholder":587},[63,16692,16693,16695,16697],{"class":65,"line":5550},[63,16694,456],{"class":91},[63,16696,15008],{"class":102},[63,16698,5150],{"class":91},[63,16700,16701,16703,16705,16708],{"class":65,"line":8275},[63,16702,483],{"class":439},[63,16704,14635],{"class":439},[63,16706,16707],{"class":95}," Int_TryFormat_WithFormat",[63,16709,5131],{"class":91},[63,16711,16712],{"class":65,"line":8280},[63,16713,250],{"class":91},[63,16715,16716,16718,16720,16722,16724,16726,16728,16730,16732,16734,16736,16738],{"class":65,"line":8285},[63,16717,593],{"class":439},[63,16719,16472],{"class":87},[63,16721,92],{"class":91},[63,16723,16204],{"class":95},[63,16725,142],{"class":91},[63,16727,16530],{"class":528},[63,16729,508],{"class":91},[63,16731,7467],{"class":439},[63,16733,16074],{"class":528},[63,16735,508],{"class":91},[63,16737,16680],{"class":145},[63,16739,149],{"class":91},[63,16741,16742],{"class":65,"line":8290},[63,16743,621],{"class":91},[63,16745,16746],{"class":65,"line":8299},[63,16747,588],{"emptyLinePlaceholder":587},[63,16749,16750,16752,16754],{"class":65,"line":8304},[63,16751,456],{"class":91},[63,16753,15008],{"class":102},[63,16755,5150],{"class":91},[63,16757,16758,16760,16762,16765],{"class":65,"line":8309},[63,16759,483],{"class":439},[63,16761,2766],{"class":439},[63,16763,16764],{"class":95}," Double_ToString",[63,16766,5131],{"class":91},[63,16768,16769],{"class":65,"line":8321},[63,16770,250],{"class":91},[63,16772,16773,16775,16777,16779,16781],{"class":65,"line":13006},[63,16774,593],{"class":439},[63,16776,16490],{"class":87},[63,16778,92],{"class":91},[63,16780,2232],{"class":95},[63,16782,403],{"class":91},[63,16784,16785],{"class":65,"line":13011},[63,16786,621],{"class":91},[63,16788,16789],{"class":65,"line":13025},[63,16790,588],{"emptyLinePlaceholder":587},[63,16792,16793,16795,16797],{"class":65,"line":13030},[63,16794,456],{"class":91},[63,16796,15008],{"class":102},[63,16798,5150],{"class":91},[63,16800,16801,16803,16805,16808],{"class":65,"line":13044},[63,16802,483],{"class":439},[63,16804,14635],{"class":439},[63,16806,16807],{"class":95}," Double_TryFormat",[63,16809,5131],{"class":91},[63,16811,16812],{"class":65,"line":13049},[63,16813,250],{"class":91},[63,16815,16816,16818,16820,16822,16824,16826,16828,16830,16832,16834],{"class":65,"line":13080},[63,16817,593],{"class":439},[63,16819,16490],{"class":87},[63,16821,92],{"class":91},[63,16823,16204],{"class":95},[63,16825,142],{"class":91},[63,16827,16530],{"class":528},[63,16829,508],{"class":91},[63,16831,7467],{"class":439},[63,16833,16074],{"class":528},[63,16835,149],{"class":91},[63,16837,16838],{"class":65,"line":13085},[63,16839,621],{"class":91},[63,16841,16842],{"class":65,"line":13090},[63,16843,588],{"emptyLinePlaceholder":587},[63,16845,16846,16848,16850],{"class":65,"line":15284},[63,16847,456],{"class":91},[63,16849,15008],{"class":102},[63,16851,5150],{"class":91},[63,16853,16854,16856,16858,16861],{"class":65,"line":15304},[63,16855,483],{"class":439},[63,16857,2766],{"class":439},[63,16859,16860],{"class":95}," Double_ToString_WithFormat",[63,16862,5131],{"class":91},[63,16864,16865],{"class":65,"line":15327},[63,16866,250],{"class":91},[63,16868,16869,16871,16873,16875,16877,16879,16882],{"class":65,"line":15332},[63,16870,593],{"class":439},[63,16872,16490],{"class":87},[63,16874,92],{"class":91},[63,16876,2232],{"class":95},[63,16878,142],{"class":91},[63,16880,16881],{"class":145},"\"F2\"",[63,16883,149],{"class":91},[63,16885,16886],{"class":65,"line":15349},[63,16887,621],{"class":91},[63,16889,16890],{"class":65,"line":15354},[63,16891,588],{"emptyLinePlaceholder":587},[63,16893,16894,16896,16898],{"class":65,"line":15359},[63,16895,456],{"class":91},[63,16897,15008],{"class":102},[63,16899,5150],{"class":91},[63,16901,16902,16904,16906,16909],{"class":65,"line":15364},[63,16903,483],{"class":439},[63,16905,14635],{"class":439},[63,16907,16908],{"class":95}," Double_TryFormat_WithFormat",[63,16910,5131],{"class":91},[63,16912,16913],{"class":65,"line":15377},[63,16914,250],{"class":91},[63,16916,16917,16919,16921,16923,16925,16927,16929,16931,16933,16935,16937,16939],{"class":65,"line":15382},[63,16918,593],{"class":439},[63,16920,16490],{"class":87},[63,16922,92],{"class":91},[63,16924,16204],{"class":95},[63,16926,142],{"class":91},[63,16928,16530],{"class":528},[63,16930,508],{"class":91},[63,16932,7467],{"class":439},[63,16934,16074],{"class":528},[63,16936,508],{"class":91},[63,16938,16881],{"class":145},[63,16940,149],{"class":91},[63,16942,16943],{"class":65,"line":15387},[63,16944,621],{"class":91},[63,16946,16947],{"class":65,"line":15396},[63,16948,588],{"emptyLinePlaceholder":587},[63,16950,16951,16953,16955],{"class":65,"line":15408},[63,16952,456],{"class":91},[63,16954,15008],{"class":102},[63,16956,5150],{"class":91},[63,16958,16959,16961,16963,16966],{"class":65,"line":15413},[63,16960,483],{"class":439},[63,16962,2766],{"class":439},[63,16964,16965],{"class":95}," Decimal_ToString",[63,16967,5131],{"class":91},[63,16969,16970],{"class":65,"line":15427},[63,16971,250],{"class":91},[63,16973,16974,16976,16978,16980,16982],{"class":65,"line":15462},[63,16975,593],{"class":439},[63,16977,16509],{"class":87},[63,16979,92],{"class":91},[63,16981,2232],{"class":95},[63,16983,403],{"class":91},[63,16985,16986],{"class":65,"line":15467},[63,16987,621],{"class":91},[63,16989,16990],{"class":65,"line":15488},[63,16991,588],{"emptyLinePlaceholder":587},[63,16993,16994,16996,16998],{"class":65,"line":15493},[63,16995,456],{"class":91},[63,16997,15008],{"class":102},[63,16999,5150],{"class":91},[63,17001,17002,17004,17006,17009],{"class":65,"line":15498},[63,17003,483],{"class":439},[63,17005,14635],{"class":439},[63,17007,17008],{"class":95}," Decimal_TryFormat",[63,17010,5131],{"class":91},[63,17012,17013],{"class":65,"line":15526},[63,17014,250],{"class":91},[63,17016,17017,17019,17021,17023,17025,17027,17029,17031,17033,17035],{"class":65,"line":15531},[63,17018,593],{"class":439},[63,17020,16509],{"class":87},[63,17022,92],{"class":91},[63,17024,16204],{"class":95},[63,17026,142],{"class":91},[63,17028,16530],{"class":528},[63,17030,508],{"class":91},[63,17032,7467],{"class":439},[63,17034,16074],{"class":528},[63,17036,149],{"class":91},[63,17038,17039],{"class":65,"line":15568},[63,17040,621],{"class":91},[63,17042,17043],{"class":65,"line":15573},[63,17044,588],{"emptyLinePlaceholder":587},[63,17046,17047,17049,17051],{"class":65,"line":15587},[63,17048,456],{"class":91},[63,17050,15008],{"class":102},[63,17052,5150],{"class":91},[63,17054,17055,17057,17059,17062],{"class":65,"line":15593},[63,17056,483],{"class":439},[63,17058,2766],{"class":439},[63,17060,17061],{"class":95}," Decimal_ToString_WithFormat",[63,17063,5131],{"class":91},[63,17065,17066],{"class":65,"line":15616},[63,17067,250],{"class":91},[63,17069,17070,17072,17074,17076,17078,17080,17083],{"class":65,"line":15656},[63,17071,593],{"class":439},[63,17073,16509],{"class":87},[63,17075,92],{"class":91},[63,17077,2232],{"class":95},[63,17079,142],{"class":91},[63,17081,17082],{"class":145},"\"C\"",[63,17084,149],{"class":91},[63,17086,17087],{"class":65,"line":15661},[63,17088,621],{"class":91},[63,17090,17091],{"class":65,"line":15684},[63,17092,588],{"emptyLinePlaceholder":587},[63,17094,17095,17097,17099],{"class":65,"line":15711},[63,17096,456],{"class":91},[63,17098,15008],{"class":102},[63,17100,5150],{"class":91},[63,17102,17103,17105,17107,17110],{"class":65,"line":15727},[63,17104,483],{"class":439},[63,17106,14635],{"class":439},[63,17108,17109],{"class":95}," Decimal_TryFormat_WithFormat",[63,17111,5131],{"class":91},[63,17113,17114],{"class":65,"line":15732},[63,17115,250],{"class":91},[63,17117,17118,17120,17122,17124,17126,17128,17130,17132,17134,17136,17138,17140],{"class":65,"line":15760},[63,17119,593],{"class":439},[63,17121,16509],{"class":87},[63,17123,92],{"class":91},[63,17125,16204],{"class":95},[63,17127,142],{"class":91},[63,17129,16530],{"class":528},[63,17131,508],{"class":91},[63,17133,7467],{"class":439},[63,17135,16074],{"class":528},[63,17137,508],{"class":91},[63,17139,17082],{"class":145},[63,17141,149],{"class":91},[63,17143,17144],{"class":65,"line":15766},[63,17145,621],{"class":91},[63,17147,17148],{"class":65,"line":15786},[63,17149,14282],{"class":91},[63,17151,17152,17154,17156],{"class":65,"line":15806},[63,17153,456],{"class":91},[63,17155,15008],{"class":102},[63,17157,5150],{"class":91},[63,17159,17160,17162,17164,17167],{"class":65,"line":15812},[63,17161,483],{"class":439},[63,17163,2766],{"class":439},[63,17165,17166],{"class":95}," Double_ToString_Culture",[63,17168,5131],{"class":91},[63,17170,17171],{"class":65,"line":15817},[63,17172,250],{"class":91},[63,17174,17175,17177,17179,17181,17183,17185,17188,17190,17193,17195,17198,17200,17203],{"class":65,"line":15823},[63,17176,593],{"class":439},[63,17178,16490],{"class":87},[63,17180,92],{"class":91},[63,17182,2232],{"class":95},[63,17184,142],{"class":91},[63,17186,17187],{"class":87},"System",[63,17189,92],{"class":91},[63,17191,17192],{"class":87},"Globalization",[63,17194,92],{"class":91},[63,17196,17197],{"class":87},"CultureInfo",[63,17199,92],{"class":91},[63,17201,17202],{"class":87},"InvariantCulture",[63,17204,149],{"class":91},[63,17206,17207],{"class":65,"line":15828},[63,17208,621],{"class":91},[63,17210,17211],{"class":65,"line":15833},[63,17212,588],{"emptyLinePlaceholder":587},[63,17214,17215,17217,17219],{"class":65,"line":15852},[63,17216,456],{"class":91},[63,17218,15008],{"class":102},[63,17220,5150],{"class":91},[63,17222,17223,17225,17227,17230],{"class":65,"line":15879},[63,17224,483],{"class":439},[63,17226,14635],{"class":439},[63,17228,17229],{"class":95}," Double_TryFormat_Culture",[63,17231,5131],{"class":91},[63,17233,17234],{"class":65,"line":15885},[63,17235,250],{"class":91},[63,17237,17238,17240,17242,17244,17246,17248,17250,17252,17254,17256,17258,17261,17263,17265,17267,17269,17271,17273,17275,17277],{"class":65,"line":15910},[63,17239,593],{"class":439},[63,17241,16490],{"class":87},[63,17243,92],{"class":91},[63,17245,16204],{"class":95},[63,17247,142],{"class":91},[63,17249,16530],{"class":528},[63,17251,508],{"class":91},[63,17253,7467],{"class":439},[63,17255,16074],{"class":528},[63,17257,508],{"class":91},[63,17259,17260],{"class":87},"provider",[63,17262,227],{"class":91},[63,17264,17187],{"class":87},[63,17266,92],{"class":91},[63,17268,17192],{"class":87},[63,17270,92],{"class":91},[63,17272,17197],{"class":87},[63,17274,92],{"class":91},[63,17276,17202],{"class":87},[63,17278,149],{"class":91},[63,17280,17281],{"class":65,"line":15925},[63,17282,621],{"class":91},[63,17284,17285],{"class":65,"line":15965},[63,17286,626],{"class":91},[16,17288,14095],{},[16,17290,17291],{},[7330,17292],{"alt":17293,"src":17294},"image-20250218-025153.png","\u002Fblog\u002Freduce-memory-footprint-and-garbage-collection-pressure\u002Fimage_20250218_025153.png",[16,17296,17297],{},"Note:",[16,17299,17300,17302],{},[32,17301,6842],{}," allows you to allocate memory directly on the stack rather than the heap. This is particularly useful for short lived, fixed size buffers. So don't do the below if you don't know the input length. The guideline is 1KB.",[54,17304,17306],{"className":78,"code":17305,"language":80,"meta":59,"style":59},"Span\u003Cchar> buffer = stackalloc char[input.Length * 2]; \u002F\u002F TOO LARGE\n",[32,17307,17308],{"__ignoreMap":59},[63,17309,17310,17312,17314,17316,17318,17320,17322,17324,17326,17328,17331,17333,17335,17337,17339,17341],{"class":65,"line":66},[63,17311,14229],{"class":102},[63,17313,99],{"class":91},[63,17315,13984],{"class":439},[63,17317,1847],{"class":91},[63,17319,16172],{"class":528},[63,17321,133],{"class":132},[63,17323,16177],{"class":91},[63,17325,13984],{"class":439},[63,17327,4353],{"class":91},[63,17329,17330],{"class":87},"input",[63,17332,92],{"class":91},[63,17334,12451],{"class":87},[63,17336,5198],{"class":132},[63,17338,2113],{"class":289},[63,17340,16187],{"class":91},[63,17342,17343],{"class":2731},"\u002F\u002F TOO LARGE\n",[11,17345,17347],{"id":17346},"iii-collectionmarshal","III. CollectionMarshal",[16,17349,17350,17351,17353,17354,17357],{},"This provides a set of methods to access the underlying data representations of collections. Basically think of it as changing a Collection to ",[32,17352,5808],{}," . The equivalent to ",[32,17355,17356],{},"AsSpan()"," for Collection is",[54,17359,17361],{"className":78,"code":17360,"language":80,"meta":59,"style":59},"Span\u003CT> collectionSpan = CollectionsMarshal.AsSpan(list)\n",[32,17362,17363],{"__ignoreMap":59},[63,17364,17365,17367,17369,17372,17374,17377,17379,17382,17384,17386,17388,17391],{"class":65,"line":66},[63,17366,14229],{"class":102},[63,17368,99],{"class":91},[63,17370,17371],{"class":102},"T",[63,17373,1847],{"class":91},[63,17375,17376],{"class":528},"collectionSpan",[63,17378,133],{"class":132},[63,17380,17381],{"class":87}," CollectionsMarshal",[63,17383,92],{"class":91},[63,17385,13997],{"class":95},[63,17387,142],{"class":91},[63,17389,17390],{"class":528},"list",[63,17392,474],{"class":91},[11,17394,17396],{"id":17395},"loop-benchmark","Loop Benchmark",[16,17398,17399],{},"Let's compare looping through a List and aggregate items together using",[1789,17401,17402,17405,17408,17411,17414],{},[173,17403,17404],{},"For",[173,17406,17407],{},"ForEach",[173,17409,17410],{},"AsSpan + For",[173,17412,17413],{},"AsSpan + ForEach",[173,17415,17416],{},"Linq Aggregate\u002FForEach",[16,17418,17419],{},"For list of different size",[16,17421,17422],{},[7330,17423],{"alt":17424,"src":17425},"image-20250218-024914.png","\u002Fblog\u002Freduce-memory-footprint-and-garbage-collection-pressure\u002Fimage_20250218_024914.png",[16,17427,17428],{},"Takeaways:",[1789,17430,17431,17434],{},[173,17432,17433],{},"AsSpan is the fastest and allow For and ForEach to have similar performance",[173,17435,17436],{},"As the list size increases, Linq is faster while traditional For and ForEach are slower",[11,17438,17440],{"id":17439},"iv-arraypool","IV. ArrayPool",[16,17442,17443,17444,17447],{},"Array pooling is a technique to reuse arrays instead of creating new ones, which helps reduce garbage collection pressure and memory fragmentation. .NET provides the ",[32,17445,17446],{},"ArrayPool\u003CT>"," class for this purpose.",[11,17449,17451],{"id":17450},"why-use-array-pooling","Why Use Array Pooling?",[1789,17453,17454,17457,17460,17463],{},[173,17455,17456],{},"Reduced Memory Allocation: Instead of creating new arrays, you rent them from a pool",[173,17458,17459],{},"Less Garbage Collection: Fewer allocations mean less GC pressure",[173,17461,17462],{},"Better Performance: Reusing arrays is faster than creating new ones",[173,17464,17465],{},"Memory Fragmentation: Helps prevent memory fragmentation in long running applications",[11,17467,17469],{"id":17468},"when-to-use-array-pooling","When To Use Array Pooling?",[1789,17471,17472,17475,17478,17481],{},[173,17473,17474],{},"High frequency array allocations: When your application frequently creates and disposes of arrays, especially in performance critical paths.",[173,17476,17477],{},"Large arrays: When working with arrays larger than 85KB (as they go on the Large Object Heap).",[173,17479,17480],{},"Memory sensitive scenarios: In environments where memory pressure is a concern, like in microservices or high scale applications.",[173,17482,17483],{},"Temporary buffers: When you need temporary buffer space for operations like I\u002FO or string manipulation. This is why you can never read CSV faster than a library.",[11,17485,17487],{"id":17486},"why-not-use-array-pooling","Why Not Use Array Pooling?",[1789,17489,17490,17493,17496,17499,17502,17505],{},[173,17491,17492],{},"Arrays are smaller than 1KB",[173,17494,17495],{},"Arrays need to be long lived",[173,17497,17498],{},"Exact size matching is required",[173,17500,17501],{},"Working with sensitive data without proper clearing",[173,17503,17504],{},"In simple, hot path operations where allocation overhead is minimal",[173,17506,17507],{},"Working with large value types",[11,17509,17511],{"id":17510},"then-use-what","Then Use What?",[1789,17513,17514,17520,17523,17526],{},[173,17515,17516,17517,5359],{},"Direct array allocation (",[32,17518,17519],{},"new byte[]",[173,17521,17522],{},"stackalloc for small, temporary buffers",[173,17524,17525],{},"Dedicated arrays for long lived data",[173,17527,17528],{},"Custom array implementations for special requirements",[16,17530,17531],{},"So how to use it",[54,17533,17535],{"className":78,"code":17534,"language":80,"meta":59,"style":59},"public class PoolingComparison\n{\n    \u002F\u002F Traditional approach - creates new arrays\n    public void Traditional()\n    {\n        byte[] buffer1 = new byte[16384];\n        byte[] buffer2 = new byte[16384];\n        byte[] buffer3 = new byte[16384];\n        \n        \u002F\u002F Use buffers...\n        \n        \u002F\u002F Arrays become eligible for garbage collection\n    }\n    \n    \u002F\u002F Pooled approach - reuses arrays\n    public void Pooled()\n    {\n        byte[] buffer1 = ArrayPool\u003Cbyte>.Shared.Rent(16384);\n        byte[] buffer2 = ArrayPool\u003Cbyte>.Shared.Rent(16384);\n        byte[] buffer3 = ArrayPool\u003Cbyte>.Shared.Rent(16384);\n        \n        try\n        {\n            \u002F\u002F Use buffers...\n        }\n        finally\n        {\n            ArrayPool\u003Cbyte>.Shared.Return(buffer1);\n            ArrayPool\u003Cbyte>.Shared.Return(buffer2);\n            ArrayPool\u003Cbyte>.Shared.Return(buffer3);\n        }\n    }\n}\n",[32,17536,17537,17546,17550,17555,17565,17569,17593,17614,17635,17639,17644,17648,17653,17657,17661,17666,17677,17681,17714,17744,17774,17778,17782,17786,17791,17795,17800,17804,17828,17850,17872,17876,17880],{"__ignoreMap":59},[63,17538,17539,17541,17543],{"class":65,"line":66},[63,17540,440],{"class":439},[63,17542,446],{"class":439},[63,17544,17545],{"class":102}," PoolingComparison\n",[63,17547,17548],{"class":65,"line":115},[63,17549,118],{"class":91},[63,17551,17552],{"class":65,"line":121},[63,17553,17554],{"class":2731},"    \u002F\u002F Traditional approach - creates new arrays\n",[63,17556,17557,17559,17561,17563],{"class":65,"line":152},[63,17558,483],{"class":439},[63,17560,14967],{"class":439},[63,17562,13877],{"class":95},[63,17564,5131],{"class":91},[63,17566,17567],{"class":65,"line":253},[63,17568,250],{"class":91},[63,17570,17571,17574,17576,17579,17581,17583,17586,17588,17591],{"class":65,"line":277},[63,17572,17573],{"class":439},"        byte",[63,17575,4442],{"class":91},[63,17577,17578],{"class":528},"buffer1",[63,17580,133],{"class":132},[63,17582,136],{"class":91},[63,17584,17585],{"class":439},"byte",[63,17587,4353],{"class":91},[63,17589,17590],{"class":289},"16384",[63,17592,5867],{"class":91},[63,17594,17595,17597,17599,17602,17604,17606,17608,17610,17612],{"class":65,"line":295},[63,17596,17573],{"class":439},[63,17598,4442],{"class":91},[63,17600,17601],{"class":528},"buffer2",[63,17603,133],{"class":132},[63,17605,136],{"class":91},[63,17607,17585],{"class":439},[63,17609,4353],{"class":91},[63,17611,17590],{"class":289},[63,17613,5867],{"class":91},[63,17615,17616,17618,17620,17623,17625,17627,17629,17631,17633],{"class":65,"line":301},[63,17617,17573],{"class":439},[63,17619,4442],{"class":91},[63,17621,17622],{"class":528},"buffer3",[63,17624,133],{"class":132},[63,17626,136],{"class":91},[63,17628,17585],{"class":439},[63,17630,4353],{"class":91},[63,17632,17590],{"class":289},[63,17634,5867],{"class":91},[63,17636,17637],{"class":65,"line":313},[63,17638,14387],{"class":91},[63,17640,17641],{"class":65,"line":318},[63,17642,17643],{"class":2731},"        \u002F\u002F Use buffers...\n",[63,17645,17646],{"class":65,"line":340},[63,17647,14387],{"class":91},[63,17649,17650],{"class":65,"line":369},[63,17651,17652],{"class":2731},"        \u002F\u002F Arrays become eligible for garbage collection\n",[63,17654,17655],{"class":65,"line":374},[63,17656,621],{"class":91},[63,17658,17659],{"class":65,"line":387},[63,17660,14282],{"class":91},[63,17662,17663],{"class":65,"line":392},[63,17664,17665],{"class":2731},"    \u002F\u002F Pooled approach - reuses arrays\n",[63,17667,17668,17670,17672,17675],{"class":65,"line":406},[63,17669,483],{"class":439},[63,17671,14967],{"class":439},[63,17673,17674],{"class":95}," Pooled",[63,17676,5131],{"class":91},[63,17678,17679],{"class":65,"line":2931},[63,17680,250],{"class":91},[63,17682,17683,17685,17687,17689,17691,17694,17696,17698,17701,17703,17705,17708,17710,17712],{"class":65,"line":2937},[63,17684,17573],{"class":439},[63,17686,4442],{"class":91},[63,17688,17578],{"class":528},[63,17690,133],{"class":132},[63,17692,17693],{"class":87}," ArrayPool",[63,17695,99],{"class":91},[63,17697,17585],{"class":439},[63,17699,17700],{"class":91},">.",[63,17702,14519],{"class":87},[63,17704,92],{"class":91},[63,17706,17707],{"class":95},"Rent",[63,17709,142],{"class":91},[63,17711,17590],{"class":289},[63,17713,149],{"class":91},[63,17715,17716,17718,17720,17722,17724,17726,17728,17730,17732,17734,17736,17738,17740,17742],{"class":65,"line":2956},[63,17717,17573],{"class":439},[63,17719,4442],{"class":91},[63,17721,17601],{"class":528},[63,17723,133],{"class":132},[63,17725,17693],{"class":87},[63,17727,99],{"class":91},[63,17729,17585],{"class":439},[63,17731,17700],{"class":91},[63,17733,14519],{"class":87},[63,17735,92],{"class":91},[63,17737,17707],{"class":95},[63,17739,142],{"class":91},[63,17741,17590],{"class":289},[63,17743,149],{"class":91},[63,17745,17746,17748,17750,17752,17754,17756,17758,17760,17762,17764,17766,17768,17770,17772],{"class":65,"line":2961},[63,17747,17573],{"class":439},[63,17749,4442],{"class":91},[63,17751,17622],{"class":528},[63,17753,133],{"class":132},[63,17755,17693],{"class":87},[63,17757,99],{"class":91},[63,17759,17585],{"class":439},[63,17761,17700],{"class":91},[63,17763,14519],{"class":87},[63,17765,92],{"class":91},[63,17767,17707],{"class":95},[63,17769,142],{"class":91},[63,17771,17590],{"class":289},[63,17773,149],{"class":91},[63,17775,17776],{"class":65,"line":3000},[63,17777,14387],{"class":91},[63,17779,17780],{"class":65,"line":3018},[63,17781,12914],{"class":439},[63,17783,17784],{"class":65,"line":3037},[63,17785,1953],{"class":91},[63,17787,17788],{"class":65,"line":3056},[63,17789,17790],{"class":2731},"            \u002F\u002F Use buffers...\n",[63,17792,17793],{"class":65,"line":5491},[63,17794,7517],{"class":91},[63,17796,17797],{"class":65,"line":5515},[63,17798,17799],{"class":439},"        finally\n",[63,17801,17802],{"class":65,"line":5520},[63,17803,1953],{"class":91},[63,17805,17806,17809,17811,17813,17815,17817,17819,17822,17824,17826],{"class":65,"line":5545},[63,17807,17808],{"class":87},"            ArrayPool",[63,17810,99],{"class":91},[63,17812,17585],{"class":439},[63,17814,17700],{"class":91},[63,17816,14519],{"class":87},[63,17818,92],{"class":91},[63,17820,17821],{"class":95},"Return",[63,17823,142],{"class":91},[63,17825,17578],{"class":528},[63,17827,149],{"class":91},[63,17829,17830,17832,17834,17836,17838,17840,17842,17844,17846,17848],{"class":65,"line":5550},[63,17831,17808],{"class":87},[63,17833,99],{"class":91},[63,17835,17585],{"class":439},[63,17837,17700],{"class":91},[63,17839,14519],{"class":87},[63,17841,92],{"class":91},[63,17843,17821],{"class":95},[63,17845,142],{"class":91},[63,17847,17601],{"class":528},[63,17849,149],{"class":91},[63,17851,17852,17854,17856,17858,17860,17862,17864,17866,17868,17870],{"class":65,"line":8275},[63,17853,17808],{"class":87},[63,17855,99],{"class":91},[63,17857,17585],{"class":439},[63,17859,17700],{"class":91},[63,17861,14519],{"class":87},[63,17863,92],{"class":91},[63,17865,17821],{"class":95},[63,17867,142],{"class":91},[63,17869,17622],{"class":528},[63,17871,149],{"class":91},[63,17873,17874],{"class":65,"line":8280},[63,17875,7517],{"class":91},[63,17877,17878],{"class":65,"line":8285},[63,17879,621],{"class":91},[63,17881,17882],{"class":65,"line":8290},[63,17883,626],{"class":91},[16,17885,17886],{},"Let's do some benchmark between these 2.",[16,17888,17889],{},[7330,17890],{"alt":17891,"src":17892},"image-20250218-030849.png","\u002Fblog\u002Freduce-memory-footprint-and-garbage-collection-pressure\u002Fimage_20250218_030849.png",[16,17894,17895],{},"Extra:",[16,17897,17898],{},"Dictionary lookup vs Alternate lookup",[16,17900,17901],{},"I'll leave the code below",[16,17903,17904],{},[7330,17905],{"alt":17906,"src":17907},"image-20250228-033501.png","\u002Fblog\u002Freduce-memory-footprint-and-garbage-collection-pressure\u002Fimage_20250228_033501.png",[54,17909,17911],{"className":78,"code":17910,"language":80,"meta":59,"style":59},"private Dictionary\u003Cstring, int> _dictionary = null!;\nprivate Dictionary\u003Cstring, int>.AlternateLookup\u003CReadOnlySpan\u003Cchar>> _alternateLookup;\nprivate string _keys = null!;\n\n[GlobalSetup]\npublic void Setup()\n{\n    _dictionary = new Dictionary\u003Cstring, int>\n    {\n        { \"foo\", 10 },\n        { \"bar\", 20 },\n        { \"baz\", 30 },\n        { \"qux\", 40 },\n        { \"quux\", 50 },\n        { \"corge\", 60 },\n        { \"grault\", 70 },\n        { \"garply\", 80 },\n        { \"waldo\", 90 },\n        { \"fred\", 100 }\n    };\n    \n    _alternateLookup = _dictionary.GetAlternateLookup\u003CReadOnlySpan\u003Cchar>>();\n    \n    _keys = \"foo, bar, baz, qux, quux, corge, grault, garply, waldo, fred\";\n}\n\n[Benchmark(Baseline = true)]\npublic int StandardDictionaryLookup()\n{\n    var sum = 0;\n    foreach (var key in _keys.Split(','))\n    {\n        var trimmedKey = key.Trim();\n        sum += _dictionary[trimmedKey];\n    }\n    return sum;\n}\n\n[Benchmark]\npublic int AlternateLookup()\n{\n    var sum = 0;\n    foreach (Range range in _keys.AsSpan().Split(','))\n    {\n        ReadOnlySpan\u003Cchar> key = _keys.AsSpan(range).Trim();\n        sum += _alternateLookup[key];\n    }\n    return sum;\n}\n",[32,17912,17913,17941,17976,17993,17997,18006,18017,18021,18043,18047,18061,18075,18088,18102,18115,18129,18143,18157,18171,18185,18189,18193,18219,18223,18235,18239,18243,18260,18271,18275,18287,18313,18317,18335,18351,18355,18363,18367,18371,18379,18390,18394,18406,18435,18439,18472,18487,18491,18499],{"__ignoreMap":59},[63,17914,17915,17917,17920,17922,17924,17926,17928,17930,17933,17935,17937,17939],{"class":65,"line":66},[63,17916,5789],{"class":439},[63,17918,17919],{"class":102}," Dictionary",[63,17921,99],{"class":91},[63,17923,502],{"class":439},[63,17925,508],{"class":91},[63,17927,856],{"class":439},[63,17929,1847],{"class":91},[63,17931,17932],{"class":528},"_dictionary",[63,17934,133],{"class":132},[63,17936,3607],{"class":289},[63,17938,3369],{"class":132},[63,17940,274],{"class":91},[63,17942,17943,17945,17947,17949,17951,17953,17955,17957,17960,17962,17964,17966,17968,17971,17974],{"class":65,"line":115},[63,17944,5789],{"class":439},[63,17946,17919],{"class":87},[63,17948,99],{"class":91},[63,17950,502],{"class":439},[63,17952,508],{"class":91},[63,17954,856],{"class":439},[63,17956,17700],{"class":91},[63,17958,17959],{"class":102},"AlternateLookup",[63,17961,99],{"class":91},[63,17963,5808],{"class":102},[63,17965,99],{"class":91},[63,17967,13984],{"class":439},[63,17969,17970],{"class":91},">> ",[63,17972,17973],{"class":528},"_alternateLookup",[63,17975,274],{"class":91},[63,17977,17978,17980,17982,17985,17987,17989,17991],{"class":65,"line":121},[63,17979,5789],{"class":439},[63,17981,2766],{"class":439},[63,17983,17984],{"class":528}," _keys",[63,17986,133],{"class":132},[63,17988,3607],{"class":289},[63,17990,3369],{"class":132},[63,17992,274],{"class":91},[63,17994,17995],{"class":65,"line":152},[63,17996,588],{"emptyLinePlaceholder":587},[63,17998,17999,18001,18004],{"class":65,"line":253},[63,18000,4353],{"class":91},[63,18002,18003],{"class":102},"GlobalSetup",[63,18005,5150],{"class":91},[63,18007,18008,18010,18012,18015],{"class":65,"line":277},[63,18009,440],{"class":439},[63,18011,14967],{"class":439},[63,18013,18014],{"class":95}," Setup",[63,18016,5131],{"class":91},[63,18018,18019],{"class":65,"line":295},[63,18020,118],{"class":91},[63,18022,18023,18026,18028,18030,18032,18034,18036,18038,18040],{"class":65,"line":301},[63,18024,18025],{"class":528},"    _dictionary",[63,18027,133],{"class":132},[63,18029,136],{"class":91},[63,18031,556],{"class":102},[63,18033,99],{"class":91},[63,18035,502],{"class":439},[63,18037,508],{"class":91},[63,18039,856],{"class":439},[63,18041,18042],{"class":91},">\n",[63,18044,18045],{"class":65,"line":313},[63,18046,250],{"class":91},[63,18048,18049,18052,18055,18057,18059],{"class":65,"line":318},[63,18050,18051],{"class":91},"        { ",[63,18053,18054],{"class":145},"\"foo\"",[63,18056,508],{"class":91},[63,18058,13929],{"class":289},[63,18060,890],{"class":91},[63,18062,18063,18065,18068,18070,18073],{"class":65,"line":340},[63,18064,18051],{"class":91},[63,18066,18067],{"class":145},"\"bar\"",[63,18069,508],{"class":91},[63,18071,18072],{"class":289},"20",[63,18074,890],{"class":91},[63,18076,18077,18079,18082,18084,18086],{"class":65,"line":369},[63,18078,18051],{"class":91},[63,18080,18081],{"class":145},"\"baz\"",[63,18083,508],{"class":91},[63,18085,988],{"class":289},[63,18087,890],{"class":91},[63,18089,18090,18092,18095,18097,18100],{"class":65,"line":374},[63,18091,18051],{"class":91},[63,18093,18094],{"class":145},"\"qux\"",[63,18096,508],{"class":91},[63,18098,18099],{"class":289},"40",[63,18101,890],{"class":91},[63,18103,18104,18106,18109,18111,18113],{"class":65,"line":387},[63,18105,18051],{"class":91},[63,18107,18108],{"class":145},"\"quux\"",[63,18110,508],{"class":91},[63,18112,14555],{"class":289},[63,18114,890],{"class":91},[63,18116,18117,18119,18122,18124,18127],{"class":65,"line":392},[63,18118,18051],{"class":91},[63,18120,18121],{"class":145},"\"corge\"",[63,18123,508],{"class":91},[63,18125,18126],{"class":289},"60",[63,18128,890],{"class":91},[63,18130,18131,18133,18136,18138,18141],{"class":65,"line":406},[63,18132,18051],{"class":91},[63,18134,18135],{"class":145},"\"grault\"",[63,18137,508],{"class":91},[63,18139,18140],{"class":289},"70",[63,18142,890],{"class":91},[63,18144,18145,18147,18150,18152,18155],{"class":65,"line":2931},[63,18146,18051],{"class":91},[63,18148,18149],{"class":145},"\"garply\"",[63,18151,508],{"class":91},[63,18153,18154],{"class":289},"80",[63,18156,890],{"class":91},[63,18158,18159,18161,18164,18166,18169],{"class":65,"line":2937},[63,18160,18051],{"class":91},[63,18162,18163],{"class":145},"\"waldo\"",[63,18165,508],{"class":91},[63,18167,18168],{"class":289},"90",[63,18170,890],{"class":91},[63,18172,18173,18175,18178,18180,18182],{"class":65,"line":2956},[63,18174,18051],{"class":91},[63,18176,18177],{"class":145},"\"fred\"",[63,18179,508],{"class":91},[63,18181,1176],{"class":289},[63,18183,18184],{"class":91}," }\n",[63,18186,18187],{"class":65,"line":2961},[63,18188,7522],{"class":91},[63,18190,18191],{"class":65,"line":3000},[63,18192,14282],{"class":91},[63,18194,18195,18198,18200,18203,18205,18208,18210,18212,18214,18216],{"class":65,"line":3018},[63,18196,18197],{"class":528},"    _alternateLookup",[63,18199,133],{"class":132},[63,18201,18202],{"class":87}," _dictionary",[63,18204,92],{"class":91},[63,18206,18207],{"class":95},"GetAlternateLookup",[63,18209,99],{"class":91},[63,18211,5808],{"class":102},[63,18213,99],{"class":91},[63,18215,13984],{"class":439},[63,18217,18218],{"class":91},">>();\n",[63,18220,18221],{"class":65,"line":3037},[63,18222,14282],{"class":91},[63,18224,18225,18228,18230,18233],{"class":65,"line":3056},[63,18226,18227],{"class":528},"    _keys",[63,18229,133],{"class":132},[63,18231,18232],{"class":145}," \"foo, bar, baz, qux, quux, corge, grault, garply, waldo, fred\"",[63,18234,274],{"class":91},[63,18236,18237],{"class":65,"line":5491},[63,18238,626],{"class":91},[63,18240,18241],{"class":65,"line":5515},[63,18242,588],{"emptyLinePlaceholder":587},[63,18244,18245,18247,18249,18251,18254,18256,18258],{"class":65,"line":5520},[63,18246,4353],{"class":91},[63,18248,15008],{"class":102},[63,18250,142],{"class":91},[63,18252,18253],{"class":2769},"Baseline",[63,18255,133],{"class":132},[63,18257,1515],{"class":289},[63,18259,12602],{"class":91},[63,18261,18262,18264,18266,18269],{"class":65,"line":5545},[63,18263,440],{"class":439},[63,18265,14932],{"class":439},[63,18267,18268],{"class":95}," StandardDictionaryLookup",[63,18270,5131],{"class":91},[63,18272,18273],{"class":65,"line":5550},[63,18274,118],{"class":91},[63,18276,18277,18279,18281,18283,18285],{"class":65,"line":8275},[63,18278,6335],{"class":439},[63,18280,6371],{"class":528},[63,18282,133],{"class":132},[63,18284,4464],{"class":289},[63,18286,274],{"class":91},[63,18288,18289,18292,18294,18296,18299,18301,18303,18305,18307,18309,18311],{"class":65,"line":8280},[63,18290,18291],{"class":439},"    foreach",[63,18293,3366],{"class":91},[63,18295,2067],{"class":439},[63,18297,18298],{"class":528}," key",[63,18300,5215],{"class":439},[63,18302,17984],{"class":87},[63,18304,92],{"class":91},[63,18306,14170],{"class":95},[63,18308,142],{"class":91},[63,18310,14175],{"class":145},[63,18312,1106],{"class":91},[63,18314,18315],{"class":65,"line":8285},[63,18316,250],{"class":91},[63,18318,18319,18321,18324,18326,18328,18330,18333],{"class":65,"line":8290},[63,18320,525],{"class":439},[63,18322,18323],{"class":528}," trimmedKey",[63,18325,133],{"class":132},[63,18327,18298],{"class":87},[63,18329,92],{"class":91},[63,18331,18332],{"class":95},"Trim",[63,18334,403],{"class":91},[63,18336,18337,18340,18342,18344,18346,18349],{"class":65,"line":8299},[63,18338,18339],{"class":528},"        sum",[63,18341,4479],{"class":4478},[63,18343,18202],{"class":87},[63,18345,4353],{"class":91},[63,18347,18348],{"class":528},"trimmedKey",[63,18350,5867],{"class":91},[63,18352,18353],{"class":65,"line":8304},[63,18354,621],{"class":91},[63,18356,18357,18359,18361],{"class":65,"line":8309},[63,18358,1890],{"class":439},[63,18360,6371],{"class":528},[63,18362,274],{"class":91},[63,18364,18365],{"class":65,"line":8321},[63,18366,626],{"class":91},[63,18368,18369],{"class":65,"line":13006},[63,18370,588],{"emptyLinePlaceholder":587},[63,18372,18373,18375,18377],{"class":65,"line":13011},[63,18374,4353],{"class":91},[63,18376,15008],{"class":102},[63,18378,5150],{"class":91},[63,18380,18381,18383,18385,18388],{"class":65,"line":13025},[63,18382,440],{"class":439},[63,18384,14932],{"class":439},[63,18386,18387],{"class":95}," AlternateLookup",[63,18389,5131],{"class":91},[63,18391,18392],{"class":65,"line":13030},[63,18393,118],{"class":91},[63,18395,18396,18398,18400,18402,18404],{"class":65,"line":13044},[63,18397,6335],{"class":439},[63,18399,6371],{"class":528},[63,18401,133],{"class":132},[63,18403,4464],{"class":289},[63,18405,274],{"class":91},[63,18407,18408,18410,18412,18414,18417,18419,18421,18423,18425,18427,18429,18431,18433],{"class":65,"line":13049},[63,18409,18291],{"class":439},[63,18411,3366],{"class":91},[63,18413,15896],{"class":102},[63,18415,18416],{"class":528}," range",[63,18418,5215],{"class":439},[63,18420,17984],{"class":87},[63,18422,92],{"class":91},[63,18424,13997],{"class":95},[63,18426,15694],{"class":91},[63,18428,14170],{"class":95},[63,18430,142],{"class":91},[63,18432,14175],{"class":145},[63,18434,1106],{"class":91},[63,18436,18437],{"class":65,"line":13080},[63,18438,250],{"class":91},[63,18440,18441,18444,18446,18448,18450,18453,18455,18457,18459,18461,18463,18466,18468,18470],{"class":65,"line":13085},[63,18442,18443],{"class":102},"        ReadOnlySpan",[63,18445,99],{"class":91},[63,18447,13984],{"class":439},[63,18449,1847],{"class":91},[63,18451,18452],{"class":528},"key",[63,18454,133],{"class":132},[63,18456,17984],{"class":87},[63,18458,92],{"class":91},[63,18460,13997],{"class":95},[63,18462,142],{"class":91},[63,18464,18465],{"class":528},"range",[63,18467,6359],{"class":91},[63,18469,18332],{"class":95},[63,18471,403],{"class":91},[63,18473,18474,18476,18478,18481,18483,18485],{"class":65,"line":13090},[63,18475,18339],{"class":528},[63,18477,4479],{"class":4478},[63,18479,18480],{"class":87}," _alternateLookup",[63,18482,4353],{"class":91},[63,18484,18452],{"class":528},[63,18486,5867],{"class":91},[63,18488,18489],{"class":65,"line":15284},[63,18490,621],{"class":91},[63,18492,18493,18495,18497],{"class":65,"line":15304},[63,18494,1890],{"class":439},[63,18496,6371],{"class":528},[63,18498,274],{"class":91},[63,18500,18501],{"class":65,"line":15327},[63,18502,626],{"class":91},[2563,18504,18505],{},"html pre.shiki code .sLKXg, html code.shiki .sLKXg{--shiki-default:#A626A4;--shiki-dark:#C678DD}html pre.shiki code .sz0mV, html code.shiki .sz0mV{--shiki-default:#383A42;--shiki-dark:#E06C75}html pre.shiki code .sknuh, html code.shiki .sknuh{--shiki-default:#383A42;--shiki-dark:#56B6C2}html pre.shiki code .sDhpE, html code.shiki .sDhpE{--shiki-default:#50A14F;--shiki-dark:#98C379}html pre.shiki code .s5ixo, html code.shiki .s5ixo{--shiki-default:#383A42;--shiki-dark:#ABB2BF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sAdtL, html code.shiki .sAdtL{--shiki-default:#4078F2;--shiki-dark:#61AFEF}html pre.shiki code .s7GmK, html code.shiki .s7GmK{--shiki-default:#383A42;--shiki-dark:#E5C07B}html pre.shiki code .sAGMh, html code.shiki .sAGMh{--shiki-default:#986801;--shiki-dark:#D19A66}html pre.shiki code .sC09Y, html code.shiki .sC09Y{--shiki-default:#C18401;--shiki-dark:#E5C07B}html pre.shiki code .sJa8x, html code.shiki .sJa8x{--shiki-default:#E45649;--shiki-dark:#E06C75}html pre.shiki code .siaei, html code.shiki .siaei{--shiki-default:#4078F2;--shiki-dark:#ABB2BF}html pre.shiki code .sblXP, html code.shiki .sblXP{--shiki-default:#383A42;--shiki-dark:#C678DD}html pre.shiki code .sW2Sy, html code.shiki .sW2Sy{--shiki-default:#A0A1A7;--shiki-default-font-style:italic;--shiki-dark:#7F848E;--shiki-dark-font-style:italic}",{"title":59,"searchDepth":115,"depth":115,"links":18507},[18508,18509,18510,18511,18512,18513,18514,18515,18516,18517,18518,18519,18520,18521],{"id":13798,"depth":115,"text":13799},{"id":13832,"depth":115,"text":13833},{"id":14104,"depth":115,"text":14105},{"id":14468,"depth":115,"text":14469},{"id":14785,"depth":115,"text":14786},{"id":16035,"depth":115,"text":16036},{"id":16123,"depth":115,"text":16124},{"id":17346,"depth":115,"text":17347},{"id":17395,"depth":115,"text":17396},{"id":17439,"depth":115,"text":17440},{"id":17450,"depth":115,"text":17451},{"id":17468,"depth":115,"text":17469},{"id":17486,"depth":115,"text":17487},{"id":17510,"depth":115,"text":17511},"2025-02-17","Practical ways to reduce memory usage and garbage collection pressure.",{},"\u002Fblog\u002Freduce-memory-footprint-and-garbage-collection-pressure",{"title":13787,"description":18523},"blog\u002Freduce-memory-footprint-and-garbage-collection-pressure",[2594],"NuxUlJUW5tNDlPT5NZb4gHB_9LyGnSOa5hF6MqwXmQg",{"id":18531,"title":18532,"body":18533,"book":2585,"date":18883,"description":18884,"extension":2588,"meta":18885,"navigation":587,"path":18886,"seo":18887,"stem":18888,"tags":18889,"__hash__":18890},"blog\u002Fblog\u002Fslimming-down-your-docker-image.md","Slimming down your Docker Image",{"type":8,"value":18534,"toc":18871},[18535,18538,18542,18545,18549,18576,18580,18618,18638,18642,18654,18658,18670,18674,18703,18707,18733,18736,18768,18772,18799,18803,18826,18833,18836,18868],[16,18536,18537],{},"Docker images can quickly become bloated, leading to slower deployments, increased storage costs, and reduced performance. In this guide, I explain some way you can reduce your docker image size for .NET and python application.",[11,18539,18541],{"id":18540},"i-clean-caches","I. Clean Caches",[16,18543,18544],{},"Clean any of the caches if you use them",[720,18546,18548],{"id":18547},"clean-package-manager-cache","Clean package manager cache",[54,18550,18552],{"className":9969,"code":18551,"language":9971,"meta":59,"style":59},"RUN apt-get update && apt-get install -y \\\n    package1 \\\n    && apt-get clean \\\n    && rm -rf \u002Fvar\u002Flib\u002Fapt\u002Flists\u002F*\n",[32,18553,18554,18561,18566,18571],{"__ignoreMap":59},[63,18555,18556,18558],{"class":65,"line":66},[63,18557,10004],{"class":95},[63,18559,18560],{"class":91}," apt-get update && apt-get install -y \\\n",[63,18562,18563],{"class":65,"line":115},[63,18564,18565],{"class":91},"    package1 \\\n",[63,18567,18568],{"class":65,"line":121},[63,18569,18570],{"class":91},"    && apt-get clean \\\n",[63,18572,18573],{"class":65,"line":152},[63,18574,18575],{"class":91},"    && rm -rf \u002Fvar\u002Flib\u002Fapt\u002Flists\u002F*\n",[720,18577,18579],{"id":18578},"clean-conda-cache","Clean conda cache",[54,18581,18583],{"className":9969,"code":18582,"language":9971,"meta":59,"style":59},"RUN conda env update -n base -f environment.yml \\\n    && conda clean -afy \\\n    && find \u002Fopt\u002Fconda\u002F -follow -type f -name '*.a' -delete \\\n    && find \u002Fopt\u002Fconda\u002F -follow -type f -name '*.js.map' -delete\n",[32,18584,18585,18592,18597,18608],{"__ignoreMap":59},[63,18586,18587,18589],{"class":65,"line":66},[63,18588,10004],{"class":95},[63,18590,18591],{"class":91}," conda env update -n base -f environment.yml \\\n",[63,18593,18594],{"class":65,"line":115},[63,18595,18596],{"class":91},"    && conda clean -afy \\\n",[63,18598,18599,18602,18605],{"class":65,"line":121},[63,18600,18601],{"class":91},"    && find \u002Fopt\u002Fconda\u002F -follow -type f -name ",[63,18603,18604],{"class":145},"'*.a'",[63,18606,18607],{"class":91}," -delete \\\n",[63,18609,18610,18612,18615],{"class":65,"line":152},[63,18611,18601],{"class":91},[63,18613,18614],{"class":145},"'*.js.map'",[63,18616,18617],{"class":91}," -delete\n",[1789,18619,18620,18626,18632],{},[173,18621,18622,18625],{},[32,18623,18624],{},"conda clean -afy"," will clean caches (all, forced, say yes to all prompts)",[173,18627,18628,18631],{},[32,18629,18630],{},"*.a"," are conda static files",[173,18633,18634,18637],{},[32,18635,18636],{},"*.js.map"," are javascript package static files usually used in jupyter notebook or other graphical libraries",[720,18639,18641],{"id":18640},"clean-pip-cache","Clean pip cache",[54,18643,18645],{"className":9969,"code":18644,"language":9971,"meta":59,"style":59},"RUN pip install --no-cache-dir -r requirements.txt\n",[32,18646,18647],{"__ignoreMap":59},[63,18648,18649,18651],{"class":65,"line":66},[63,18650,10004],{"class":95},[63,18652,18653],{"class":91}," pip install --no-cache-dir -r requirements.txt\n",[720,18655,18657],{"id":18656},"clean-pixi-cache","Clean pixi cache",[54,18659,18661],{"className":9969,"code":18660,"language":9971,"meta":59,"style":59},"RUN rm -rf \u002Froot\u002F.cache\u002Frattler\n",[32,18662,18663],{"__ignoreMap":59},[63,18664,18665,18667],{"class":65,"line":66},[63,18666,10004],{"class":95},[63,18668,18669],{"class":91}," rm -rf \u002Froot\u002F.cache\u002Frattler\n",[11,18671,18673],{"id":18672},"ii-delete-files-you-dont-need","II. Delete files you don't need",[54,18675,18677],{"className":9969,"code":18676,"language":9971,"meta":59,"style":59},"COPY pixi.toml .\nRUN pixi install && \\\n    rm pixi.lock && \\\n    rm -rf \u002Froot\u002F.cache\u002Frattler && \\\n",[32,18678,18679,18686,18693,18698],{"__ignoreMap":59},[63,18680,18681,18683],{"class":65,"line":66},[63,18682,10074],{"class":95},[63,18684,18685],{"class":91}," pixi.toml .\n",[63,18687,18688,18690],{"class":65,"line":115},[63,18689,10004],{"class":95},[63,18691,18692],{"class":91}," pixi install && \\\n",[63,18694,18695],{"class":65,"line":121},[63,18696,18697],{"class":91},"    rm pixi.lock && \\\n",[63,18699,18700],{"class":65,"line":152},[63,18701,18702],{"class":91},"    rm -rf \u002Froot\u002F.cache\u002Frattler && \\\n",[11,18704,18706],{"id":18705},"iii-combine-run-commands-to-reduce-layer","III. Combine RUN commands to reduce layer",[54,18708,18710],{"className":9969,"code":18709,"language":9971,"meta":59,"style":59},"RUN pixi install && \\\n    rm pixi.lock && \\\n    rm -rf \u002Froot\u002F.cache\u002Frattler && \\\n    rm pixi.toml\n",[32,18711,18712,18718,18723,18728],{"__ignoreMap":59},[63,18713,18714,18716],{"class":65,"line":66},[63,18715,10004],{"class":95},[63,18717,18692],{"class":91},[63,18719,18720],{"class":65,"line":115},[63,18721,18722],{"class":91},"    rm pixi.lock && \\\n",[63,18724,18725],{"class":65,"line":121},[63,18726,18727],{"class":91},"    rm -rf \u002Froot\u002F.cache\u002Frattler && \\\n",[63,18729,18730],{"class":65,"line":152},[63,18731,18732],{"class":91},"    rm pixi.toml\n",[16,18734,18735],{},"is better than",[54,18737,18739],{"className":9969,"code":18738,"language":9971,"meta":59,"style":59},"RUN pixi install\nRUN pixi.lock\nRUN rm -rf \u002Froot\u002F.cache\u002Frattler\nRUN rm pixi.toml\n",[32,18740,18741,18748,18755,18761],{"__ignoreMap":59},[63,18742,18743,18745],{"class":65,"line":66},[63,18744,10004],{"class":95},[63,18746,18747],{"class":91}," pixi install\n",[63,18749,18750,18752],{"class":65,"line":115},[63,18751,10004],{"class":95},[63,18753,18754],{"class":91}," pixi.lock\n",[63,18756,18757,18759],{"class":65,"line":121},[63,18758,10004],{"class":95},[63,18760,18669],{"class":91},[63,18762,18763,18765],{"class":65,"line":152},[63,18764,10004],{"class":95},[63,18766,18767],{"class":91}," rm pixi.toml\n",[11,18769,18771],{"id":18770},"iv-use-dockerignore-to-exclude-unnecessary-files-from-build-context","IV. Use .dockerignore to exclude unnecessary files from build context",[54,18773,18777],{"className":18774,"code":18775,"language":18776,"meta":59,"style":59},"language-dockerignore shiki shiki-themes one-light one-dark-pro","**\u002F.git\n**\u002Fnode_modules\n**\u002Fbin\n**\u002Fobj\n","dockerignore",[32,18778,18779,18784,18789,18794],{"__ignoreMap":59},[63,18780,18781],{"class":65,"line":66},[63,18782,18783],{},"**\u002F.git\n",[63,18785,18786],{"class":65,"line":115},[63,18787,18788],{},"**\u002Fnode_modules\n",[63,18790,18791],{"class":65,"line":121},[63,18792,18793],{},"**\u002Fbin\n",[63,18795,18796],{"class":65,"line":152},[63,18797,18798],{},"**\u002Fobj\n",[11,18800,18802],{"id":18801},"v-use-net-optimization-flag","V. Use .NET optimization flag",[54,18804,18806],{"className":9969,"code":18805,"language":9971,"meta":59,"style":59},"RUN dotnet publish \"MyApp.csproj\" -c Release -o \u002Fapp\u002Fpublish \\\n    \u002Fp:UseAppHost=false\n",[32,18807,18808,18821],{"__ignoreMap":59},[63,18809,18810,18812,18815,18818],{"class":65,"line":66},[63,18811,10004],{"class":95},[63,18813,18814],{"class":91}," dotnet publish ",[63,18816,18817],{"class":145},"\"MyApp.csproj\"",[63,18819,18820],{"class":91}," -c Release -o \u002Fapp\u002Fpublish \\\n",[63,18822,18823],{"class":65,"line":115},[63,18824,18825],{"class":91},"    \u002Fp:UseAppHost=false\n",[16,18827,18828,18829,18832],{},"Matter of fact, we should always use ",[32,18830,18831],{},"\u002Fp:UseAppHost=false"," in our project",[16,18834,18835],{},"What does it do?. There are a few .NET optimization flag",[1789,18837,18838,18851,18862],{},[173,18839,18840,18842,18843],{},[32,18841,18831],{},": Prevents generation of native executable",[1789,18844,18845,18848],{},[173,18846,18847],{},"You need this if you use IIS",[173,18849,18850],{},"If your environment already runs asp net runtime (docker), you don't need this, this will prevent generating .dll files needed to run .NET",[173,18852,18853,18856,18857],{},[32,18854,18855],{},"\u002Fp:PublishTrimmed=true",": Removes unused assemblies (careful with reflection)",[1789,18858,18859],{},[173,18860,18861],{},"This is the best if your project is AOT, it does the equivalent of Tree Shaking in JavaScript",[173,18863,18864,18867],{},[32,18865,18866],{},"\u002Fp:PublishSingleFile=true",": Creates a single executable file",[2563,18869,18870],{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sAdtL, html code.shiki .sAdtL{--shiki-default:#4078F2;--shiki-dark:#61AFEF}html pre.shiki code .s5ixo, html code.shiki .s5ixo{--shiki-default:#383A42;--shiki-dark:#ABB2BF}html pre.shiki code .sDhpE, html code.shiki .sDhpE{--shiki-default:#50A14F;--shiki-dark:#98C379}",{"title":59,"searchDepth":115,"depth":115,"links":18872},[18873,18879,18880,18881,18882],{"id":18540,"depth":115,"text":18541,"children":18874},[18875,18876,18877,18878],{"id":18547,"depth":121,"text":18548},{"id":18578,"depth":121,"text":18579},{"id":18640,"depth":121,"text":18641},{"id":18656,"depth":121,"text":18657},{"id":18672,"depth":115,"text":18673},{"id":18705,"depth":115,"text":18706},{"id":18770,"depth":115,"text":18771},{"id":18801,"depth":115,"text":18802},"2025-02-10","Simple Docker image changes that reduce size and risk.",{},"\u002Fblog\u002Fslimming-down-your-docker-image",{"title":18532,"description":18884},"blog\u002Fslimming-down-your-docker-image",[2594],"RXnz60dBwp0OMk_YKxS4bPU3l9jZJjYU_SFCKaWiS24",{"id":18892,"title":23,"body":18893,"book":2585,"date":20635,"description":20636,"extension":2588,"meta":20637,"navigation":587,"path":22,"seo":20638,"stem":20639,"tags":20640,"__hash__":20641},"blog\u002Fblog\u002Fload-balancing-long-lived-connections-in-kubernetes.md",{"type":8,"value":18894,"toc":20621},[18895,18898,18902,18905,18908,18914,18917,18920,18924,18935,18939,18942,18945,18948,18959,18962,18966,18970,18973,18976,18998,19091,19094,19097,19100,19104,19107,19113,19116,19119,19122,19207,19210,19221,19224,19228,19231,19237,19240,19246,19249,19252,19255,19259,19328,19331,19335,19338,19342,20618],[16,18896,18897],{},"TL;DR: Kubernetes doesn't load balance long lived connections, and some Pods might receive more requests than others. Consider client side load balancing or a proxy if you're using HTTP\u002F2, gRPC, long lived HttpClient or other long lived database connection.",[11,18899,18901],{"id":18900},"service-clusterip-nodeport-in-k8s","Service (ClusterIP, NodePort) in k8s",[16,18903,18904],{},"Kubernetes Services don't exist.",[16,18906,18907],{},"There's no process for listening to the IP address and port of the Service. Even the IP address can't be found anywhere.",[16,18909,18910,18911,92],{},"You can check that this is the case by accessing any node in your Kubernetes cluster and executing ",[32,18912,18913],{},"netstat -ntlp",[16,18915,18916],{},"Kube proxy reads the list of IP addresses for all Services and writes rules in every node.",[16,18918,18919],{},"The rules are meant to say, \" If you see this Service IP address, rewrite the request and pick one of the Pods as the destination.\" The IP Address is only a placeholder.",[720,18921,18923],{"id":18922},"by-default-kubernetes-uses-iptables-to-implement-services-iptables-traditionally-does-not-implement-load-balancing-or-round-robin-strategy-but-kubernetes-implements-an-interesting-rule-into-their-iptable-for-example-with-3-pods-under-this-service-ip-they-will-implement-the-rule","By default, Kubernetes uses iptables to implement Services. iptables traditionally does not implement load balancing or round robin strategy, but Kubernetes implements an interesting rule into their iptable. For example, with 3 pods under this service IP, they will implement the rule:",[170,18925,18926,18929,18932],{},[173,18927,18928],{},"With a likelihood of 33%, select Pod 1 as the destination. Otherwise, proceed to the following rule.",[173,18930,18931],{},"With a probability of 50%, choose Pod 2 as the destination. Otherwise, proceed to the following rule.",[173,18933,18934],{},"Select Pod 3 as the destination (no probability).",[11,18936,18938],{"id":18937},"long-lived-connections-dont-get-load-balanced","Long lived connections don't get load balanced",[16,18940,18941],{},"What happened if the clients keep sending requests to the same Service? They will be sent to the same pods",[16,18943,18944],{},"You can reproduce this with a small test service that sends repeated requests through a single long lived client.",[16,18946,18947],{},"Why is the traffic not distributed?",[1789,18949,18950,18953,18956],{},[173,18951,18952],{},"A single TCP connection is open, and the iptables rule was invoked the first time.",[173,18954,18955],{},"One of the three Pods was selected as the destination.",[173,18957,18958],{},"Since all subsequent requests are channelled through the same TCP connection, iptables doesn't invoke anymore.",[16,18960,18961],{},"So now you achieve better throughput and latency but completely lost the ability to scale your backend services.",[11,18963,18965],{"id":18964},"solutions","Solutions",[720,18967,18969],{"id":18968},"_1-client-side-load-balancing","1. Client side load balancing",[16,18971,18972],{},"This is the most common solution for larger systems.",[16,18974,18975],{},"The client side code that executes the load balancing should follow the logic below:",[170,18977,18978,18981,18984,18987,18990],{},[173,18979,18980],{},"Retrieve a list of endpoints from the Service.",[173,18982,18983],{},"For each of them, open a connection and keep it open.",[173,18985,18986],{},"Pick one of the open connections When you need to make a request.",[173,18988,18989],{},"At regular intervals, refresh the list of endpoints and remove or add new connections.",[173,18991,18992,18993],{},"Refresh the list of endpoints when you have host error",[170,18994,18995],{},[173,18996,18997],{},"In .NET, you can detect host error with this (you can't use http status code for this error)",[54,18999,19001],{"className":78,"code":19000,"language":80,"meta":59,"style":59},"catch (HttpRequestException ex) when \n    (ex.InnerException is SocketException socketEx && \n     (socketEx.ErrorCode == 11001 || socketEx.ErrorCode == 11004))\n{\n    Console.WriteLine(\"IP Resolution Failed\");\n}\n",[32,19002,19003,19017,19041,19066,19070,19087],{"__ignoreMap":59},[63,19004,19005,19007,19009,19012,19014],{"class":65,"line":66},[63,19006,12143],{"class":439},[63,19008,3366],{"class":91},[63,19010,19011],{"class":102},"HttpRequestException",[63,19013,12151],{"class":528},[63,19015,19016],{"class":91},") when \n",[63,19018,19019,19022,19024,19026,19029,19032,19035,19038],{"class":65,"line":115},[63,19020,19021],{"class":91},"    (",[63,19023,12176],{"class":102},[63,19025,92],{"class":91},[63,19027,19028],{"class":102},"InnerException",[63,19030,19031],{"class":528}," is",[63,19033,19034],{"class":102}," SocketException",[63,19036,19037],{"class":528}," socketEx",[63,19039,19040],{"class":91}," && \n",[63,19042,19043,19046,19049,19051,19054,19057,19059,19061,19063],{"class":65,"line":121},[63,19044,19045],{"class":91},"     (",[63,19047,19048],{"class":102},"socketEx",[63,19050,92],{"class":91},[63,19052,19053],{"class":102},"ErrorCode",[63,19055,19056],{"class":91}," == 11001 || ",[63,19058,19048],{"class":102},[63,19060,92],{"class":91},[63,19062,19053],{"class":102},[63,19064,19065],{"class":91}," == 11004))\n",[63,19067,19068],{"class":65,"line":152},[63,19069,118],{"class":91},[63,19071,19072,19075,19077,19080,19082,19085],{"class":65,"line":253},[63,19073,19074],{"class":87},"    Console",[63,19076,92],{"class":91},[63,19078,19079],{"class":95},"WriteLine",[63,19081,142],{"class":91},[63,19083,19084],{"class":145},"\"IP Resolution Failed\"",[63,19086,149],{"class":91},[63,19088,19089],{"class":65,"line":277},[63,19090,626],{"class":91},[16,19092,19093],{},"Or you can do something more complicated with a dedicated client side load balancer.",[16,19095,19096],{},"You can check sample code at the end.",[16,19098,19099],{},"Cons: Complicated",[720,19101,19103],{"id":19102},"_2-service-mesh-to-the-rescue","2. Service Mesh to the rescue",[16,19105,19106],{},"A service mesh decouples the communication between services from the application layer to the infrastructure layer. The abstraction at the infrastructure level happens by proxying the traffic between services. Many companies use Service Mesh",[16,19108,19109],{},[7330,19110],{"alt":19111,"src":19112},"image-20241206-073028.png","\u002Fblog\u002Fload-balancing-long-lived-connections-in-kubernetes\u002Fimage_20241206_073028.png",[16,19114,19115],{},"One of the key benefit of this is the proxy will do the load balancing for you.",[16,19117,19118],{},"There are 2 big players in the market: istio vs linkerd",[16,19120,19121],{},"You can check the CNCF landscape or service mesh project documentation for more options.",[54,19123,19127],{"className":19124,"code":19125,"language":19126,"meta":59,"style":59},"language-yaml shiki shiki-themes one-light one-dark-pro","apiVersion: networking.istio.io\u002Fv1alpha3\nkind: DestinationRule\nmetadata:\n  name: my-service-destination\nspec:\n  host: my-service\n  trafficPolicy:\n    loadBalancer:\n      simple: ROUND_ROBIN\n","yaml",[32,19128,19129,19139,19149,19156,19166,19173,19183,19190,19197],{"__ignoreMap":59},[63,19130,19131,19134,19136],{"class":65,"line":66},[63,19132,19133],{"class":2976},"apiVersion",[63,19135,227],{"class":91},[63,19137,19138],{"class":145},"networking.istio.io\u002Fv1alpha3\n",[63,19140,19141,19144,19146],{"class":65,"line":115},[63,19142,19143],{"class":2976},"kind",[63,19145,227],{"class":91},[63,19147,19148],{"class":145},"DestinationRule\n",[63,19150,19151,19154],{"class":65,"line":121},[63,19152,19153],{"class":2976},"metadata",[63,19155,10453],{"class":91},[63,19157,19158,19161,19163],{"class":65,"line":152},[63,19159,19160],{"class":2976},"  name",[63,19162,227],{"class":91},[63,19164,19165],{"class":145},"my-service-destination\n",[63,19167,19168,19171],{"class":65,"line":253},[63,19169,19170],{"class":2976},"spec",[63,19172,10453],{"class":91},[63,19174,19175,19178,19180],{"class":65,"line":277},[63,19176,19177],{"class":2976},"  host",[63,19179,227],{"class":91},[63,19181,19182],{"class":145},"my-service\n",[63,19184,19185,19188],{"class":65,"line":295},[63,19186,19187],{"class":2976},"  trafficPolicy",[63,19189,10453],{"class":91},[63,19191,19192,19195],{"class":65,"line":301},[63,19193,19194],{"class":2976},"    loadBalancer",[63,19196,10453],{"class":91},[63,19198,19199,19202,19204],{"class":65,"line":313},[63,19200,19201],{"class":2976},"      simple",[63,19203,227],{"class":91},[63,19205,19206],{"class":145},"ROUND_ROBIN\n",[16,19208,19209],{},"Cons:",[1789,19211,19212,19215,19218],{},[173,19213,19214],{},"You have an extra container in every pod",[173,19216,19217],{},"Costs more to operate",[173,19219,19220],{},"Better to be maintained by a dedicated IT team",[16,19222,19223],{},"Service Mesh basically handles client side load balancing for you.",[720,19225,19227],{"id":19226},"_3-scaled-down-number-of-pods","3. Scaled down number of pods",[16,19229,19230],{},"If you have more clients than servers, there should be limited issue. The servers will be more utilized",[16,19232,19233],{},[7330,19234],{"alt":19235,"src":19236},"image-20241206-073629.png","\u002Fblog\u002Fload-balancing-long-lived-connections-in-kubernetes\u002Fimage_20241206_073629.png",[16,19238,19239],{},"The opposite scenario is the troublesome one",[16,19241,19242],{},[7330,19243],{"alt":19244,"src":19245},"image-20241206-073740.png","\u002Fblog\u002Fload-balancing-long-lived-connections-in-kubernetes\u002Fimage_20241206_073740.png",[16,19247,19248],{},"This is why Horizontal Scaling will not help unless you can load balance long lived connection properly.",[16,19250,19251],{},"So obviously 1 of the solution is to scale down, potentially to even 1 pod, and do Vertical Scaling if needed",[16,19253,19254],{},"Cons: what are the cons of fewer pods, or 1 pod, with vertical scaling?",[720,19256,19258],{"id":19257},"_4-shorter-long-lived-clients","4. Shorter long lived clients",[54,19260,19262],{"className":78,"code":19261,"language":80,"meta":59,"style":59},"services.AddHttpClient\u003CTClient, TImplementation>()\n  .ConfigurePrimaryHttpMessageHandler(_ => new SocketsHttpHandler\n  {\n      PooledConnectionLifetime = TimeSpan.FromMinutes(60)\n  });\n",[32,19263,19264,19284,19299,19304,19323],{"__ignoreMap":59},[63,19265,19266,19268,19270,19272,19274,19277,19279,19282],{"class":65,"line":66},[63,19267,88],{"class":87},[63,19269,92],{"class":91},[63,19271,96],{"class":95},[63,19273,99],{"class":91},[63,19275,19276],{"class":102},"TClient",[63,19278,508],{"class":91},[63,19280,19281],{"class":102},"TImplementation",[63,19283,13508],{"class":91},[63,19285,19286,19289,19291,19293,19295,19297],{"class":65,"line":115},[63,19287,19288],{"class":91},"  .",[63,19290,13515],{"class":95},[63,19292,142],{"class":91},[63,19294,13520],{"class":87},[63,19296,13523],{"class":91},[63,19298,13526],{"class":102},[63,19300,19301],{"class":65,"line":121},[63,19302,19303],{"class":91},"  {\n",[63,19305,19306,19309,19311,19313,19315,19317,19319,19321],{"class":65,"line":152},[63,19307,19308],{"class":528},"      PooledConnectionLifetime",[63,19310,133],{"class":132},[63,19312,682],{"class":87},[63,19314,92],{"class":91},[63,19316,3939],{"class":95},[63,19318,142],{"class":91},[63,19320,18126],{"class":289},[63,19322,474],{"class":91},[63,19324,19325],{"class":65,"line":253},[63,19326,19327],{"class":91},"  });\n",[16,19329,19330],{},"We can make the connection lifetime shorter. You cannot make it too short or else there is no point of having a long lived client",[720,19332,19334],{"id":19333},"_5-changing-architecture","5. Changing architecture",[16,19336,19337],{},"This is not possible most of the time, but if you use Event driven architecture, load balancing task is handled by Kafka\u002FRabbitMQ or whatever library you use",[720,19339,19341],{"id":19340},"code-for-number-1","Code for number 1",[54,19343,19345],{"className":78,"code":19344,"language":80,"meta":59,"style":59},"public class RoundRobinK8sClientHandler : HttpClientHandler\n{\n    private ConcurrentDictionary\u003Cstring, HttpClient> _persistentClients;\n    private ReaderWriterLockSlim _clientsLock = new ReaderWriterLockSlim();\n    private int _currentIndex = 0;\n\n    private readonly string _namespace;\n    private readonly string _serviceName;\n    private readonly Kubernetes _k8sClient = new Kubernetes(KubernetesClientConfiguration.BuildDefaultConfig());\n\n    public RoundRobinK8sClientHandler(string namespace, string serviceName)\n    {\n        _namespace = namespace;\n        _serviceName = serviceName;\n        _persistentClients = new ConcurrentDictionary\u003Cstring, HttpClient>();\n        RefreshClient();\n    }\n\n    private async Task\u003CList\u003CPodInfo>> GetPodsByServiceWithIpAsync()\n    {\n        var service = await _k8sClient.ReadNamespacedServiceAsync(_serviceName, _namespace);\n        var podList = await _k8sClient.ListNamespacedPodAsync(\n            _namespace, \n            labelSelector: string.Join(\",\", \n                service.Spec.Selector.Select(kvp => $\"{kvp.Key}={kvp.Value}\"))\n        );\n\n        return podList.Items.Select(pod => pod.Status.PodIP).ToList();\n    }\n\n    private void RefreshClient()\n    {\n        _clientsLock.EnterWriteLock();\n        try\n        {\n            var endpoints = GetPodsByServiceWithIpAsync().GetAwaiter().GetResult();\n\n            var newClients = new ConcurrentDictionary\u003Cstring, HttpClient>();\n            foreach (var endpoint in endpoints)\n            {\n                if (!_persistentClients.TryGetValue(endpoint, out var existingClient))\n                {\n                    existingClient = new HttpClient(new HttpClientHandler())\n                    {\n                        BaseAddress = new Uri(endpoint)\n                    };\n                }\n                newClients[endpoint] = existingClient;\n            }\n\n            _persistentClients = newClients;\n        }\n        finally \n        {\n            _clientsLock.ExitWriteLock();\n        }\n    }\n\n    protected override async Task\u003CHttpResponseMessage> SendAsync(\n        HttpRequestMessage request, \n        CancellationToken cancellationToken)\n    {\n        _clientsLock.EnterReadLock();\n        try\n        {\n            if (_persistentClients.Count == 0)\n            {\n                RefreshClient();\n            }\n\n            var endpoints = _persistentClients.Keys.ToList();\n            if (endpoints.Count == 0)\n            {\n                throw new InvalidOperationException(\"No endpoints available\");\n            }\n\n            \u002F\u002F Round-robin endpoint selection\n            string selectedEndpoint = endpoints[\n                Interlocked.Increment(ref _currentIndex) % endpoints.Count];\n\n            var client = _persistentClients[selectedEndpoint];\n\n            \u002F\u002F Clone the request for the specific endpoint\n            var newRequest = new HttpRequestMessage(request.Method, request.RequestUri);\n            newRequest.Content = request.Content;\n            foreach (var header in request.Headers)\n            {\n                newRequest.Headers.TryAddWithoutValidation(header.Key, header.Value);\n            }\n\n            return await client.SendAsync(newRequest, cancellationToken);\n        }\n        catch (HttpRequestException ex) when \n            (ex.InnerException is SocketException socketEx && \n              (socketEx.ErrorCode == 11001 || socketEx.ErrorCode == 11004))\n        {\n            RefreshClient();\n        }\n        finally \n        {\n            _clientsLock.ExitReadLock();\n        }\n    }\n}\n\n\nservices.AddHttpClient(\"MyClient\")\n    .ConfigurePrimaryHttpMessageHandler(sp => new RoundRobinK8sClientHandler(\"app-namespace\", \"sample-service\"));\n",[32,19346,19347,19361,19365,19387,19406,19421,19425,19438,19451,19483,19487,19509,19513,19524,19535,19557,19564,19568,19572,19596,19600,19631,19651,19659,19679,19735,19740,19744,19784,19788,19792,19803,19807,19819,19823,19827,19851,19855,19878,19896,19900,19930,19934,19953,19958,19975,19980,19984,20001,20005,20009,20020,20024,20032,20036,20048,20052,20056,20060,20080,20089,20099,20103,20114,20118,20122,20140,20144,20151,20155,20159,20181,20200,20204,20221,20225,20229,20234,20249,20279,20283,20300,20304,20309,20342,20362,20383,20387,20420,20424,20428,20452,20456,20468,20487,20508,20512,20519,20524,20531,20536,20548,20553,20558,20563,20568,20573,20589],{"__ignoreMap":59},[63,19348,19349,19351,19353,19356,19358],{"class":65,"line":66},[63,19350,440],{"class":439},[63,19352,446],{"class":439},[63,19354,19355],{"class":102}," RoundRobinK8sClientHandler",[63,19357,12614],{"class":91},[63,19359,19360],{"class":102},"HttpClientHandler\n",[63,19362,19363],{"class":65,"line":115},[63,19364,118],{"class":91},[63,19366,19367,19369,19372,19374,19376,19378,19380,19382,19385],{"class":65,"line":121},[63,19368,2964],{"class":439},[63,19370,19371],{"class":102}," ConcurrentDictionary",[63,19373,99],{"class":91},[63,19375,502],{"class":439},[63,19377,508],{"class":91},[63,19379,34],{"class":102},[63,19381,1847],{"class":91},[63,19383,19384],{"class":2976},"_persistentClients",[63,19386,274],{"class":91},[63,19388,19389,19391,19394,19397,19399,19401,19404],{"class":65,"line":152},[63,19390,2964],{"class":439},[63,19392,19393],{"class":102}," ReaderWriterLockSlim",[63,19395,19396],{"class":2976}," _clientsLock",[63,19398,133],{"class":132},[63,19400,136],{"class":91},[63,19402,19403],{"class":102},"ReaderWriterLockSlim",[63,19405,403],{"class":91},[63,19407,19408,19410,19412,19415,19417,19419],{"class":65,"line":253},[63,19409,2964],{"class":439},[63,19411,14932],{"class":439},[63,19413,19414],{"class":2976}," _currentIndex",[63,19416,133],{"class":132},[63,19418,4464],{"class":289},[63,19420,274],{"class":91},[63,19422,19423],{"class":65,"line":277},[63,19424,588],{"emptyLinePlaceholder":587},[63,19426,19427,19429,19431,19433,19436],{"class":65,"line":295},[63,19428,2964],{"class":439},[63,19430,2970],{"class":439},[63,19432,2766],{"class":439},[63,19434,19435],{"class":2976}," _namespace",[63,19437,274],{"class":91},[63,19439,19440,19442,19444,19446,19449],{"class":65,"line":301},[63,19441,2964],{"class":439},[63,19443,2970],{"class":439},[63,19445,2766],{"class":439},[63,19447,19448],{"class":2976}," _serviceName",[63,19450,274],{"class":91},[63,19452,19453,19455,19457,19460,19463,19465,19467,19470,19472,19475,19477,19480],{"class":65,"line":313},[63,19454,2964],{"class":439},[63,19456,2970],{"class":439},[63,19458,19459],{"class":102}," Kubernetes",[63,19461,19462],{"class":2976}," _k8sClient",[63,19464,133],{"class":132},[63,19466,136],{"class":91},[63,19468,19469],{"class":102},"Kubernetes",[63,19471,142],{"class":91},[63,19473,19474],{"class":87},"KubernetesClientConfiguration",[63,19476,92],{"class":91},[63,19478,19479],{"class":95},"BuildDefaultConfig",[63,19481,19482],{"class":91},"());\n",[63,19484,19485],{"class":65,"line":318},[63,19486,588],{"emptyLinePlaceholder":587},[63,19488,19489,19491,19493,19495,19497,19500,19502,19504,19507],{"class":65,"line":340},[63,19490,483],{"class":439},[63,19492,19355],{"class":95},[63,19494,142],{"class":91},[63,19496,502],{"class":439},[63,19498,19499],{"class":87}," namespace",[63,19501,508],{"class":91},[63,19503,502],{"class":439},[63,19505,19506],{"class":87}," serviceName",[63,19508,474],{"class":91},[63,19510,19511],{"class":65,"line":369},[63,19512,250],{"class":91},[63,19514,19515,19518,19520,19522],{"class":65,"line":374},[63,19516,19517],{"class":528},"        _namespace",[63,19519,133],{"class":132},[63,19521,19499],{"class":528},[63,19523,274],{"class":91},[63,19525,19526,19529,19531,19533],{"class":65,"line":387},[63,19527,19528],{"class":528},"        _serviceName",[63,19530,133],{"class":132},[63,19532,19506],{"class":528},[63,19534,274],{"class":91},[63,19536,19537,19540,19542,19544,19547,19549,19551,19553,19555],{"class":65,"line":392},[63,19538,19539],{"class":528},"        _persistentClients",[63,19541,133],{"class":132},[63,19543,136],{"class":91},[63,19545,19546],{"class":102},"ConcurrentDictionary",[63,19548,99],{"class":91},[63,19550,502],{"class":439},[63,19552,508],{"class":91},[63,19554,34],{"class":102},[63,19556,14277],{"class":91},[63,19558,19559,19562],{"class":65,"line":406},[63,19560,19561],{"class":95},"        RefreshClient",[63,19563,403],{"class":91},[63,19565,19566],{"class":65,"line":2931},[63,19567,621],{"class":91},[63,19569,19570],{"class":65,"line":2937},[63,19571,588],{"emptyLinePlaceholder":587},[63,19573,19574,19576,19578,19580,19582,19584,19586,19589,19591,19594],{"class":65,"line":2956},[63,19575,2964],{"class":439},[63,19577,2256],{"class":439},[63,19579,486],{"class":102},[63,19581,99],{"class":91},[63,19583,14270],{"class":102},[63,19585,99],{"class":91},[63,19587,19588],{"class":102},"PodInfo",[63,19590,17970],{"class":91},[63,19592,19593],{"class":95},"GetPodsByServiceWithIpAsync",[63,19595,5131],{"class":91},[63,19597,19598],{"class":65,"line":2961},[63,19599,250],{"class":91},[63,19601,19602,19604,19607,19609,19611,19614,19616,19619,19621,19624,19626,19629],{"class":65,"line":3000},[63,19603,525],{"class":439},[63,19605,19606],{"class":528}," service",[63,19608,133],{"class":132},[63,19610,2301],{"class":91},[63,19612,19613],{"class":87},"_k8sClient",[63,19615,92],{"class":91},[63,19617,19618],{"class":95},"ReadNamespacedServiceAsync",[63,19620,142],{"class":91},[63,19622,19623],{"class":528},"_serviceName",[63,19625,508],{"class":91},[63,19627,19628],{"class":528},"_namespace",[63,19630,149],{"class":91},[63,19632,19633,19635,19638,19640,19642,19644,19646,19649],{"class":65,"line":3018},[63,19634,525],{"class":439},[63,19636,19637],{"class":528}," podList",[63,19639,133],{"class":132},[63,19641,2301],{"class":91},[63,19643,19613],{"class":87},[63,19645,92],{"class":91},[63,19647,19648],{"class":95},"ListNamespacedPodAsync",[63,19650,219],{"class":91},[63,19652,19653,19656],{"class":65,"line":3037},[63,19654,19655],{"class":528},"            _namespace",[63,19657,19658],{"class":91},", \n",[63,19660,19661,19664,19666,19668,19670,19672,19674,19677],{"class":65,"line":3056},[63,19662,19663],{"class":87},"            labelSelector",[63,19665,227],{"class":91},[63,19667,502],{"class":439},[63,19669,92],{"class":91},[63,19671,15186],{"class":95},[63,19673,142],{"class":91},[63,19675,19676],{"class":145},"\",\"",[63,19678,19658],{"class":91},[63,19680,19681,19684,19686,19689,19691,19694,19696,19698,19700,19703,19705,19708,19710,19712,19714,19717,19719,19721,19723,19725,19727,19729,19731,19733],{"class":65,"line":5491},[63,19682,19683],{"class":87},"                service",[63,19685,92],{"class":91},[63,19687,19688],{"class":87},"Spec",[63,19690,92],{"class":91},[63,19692,19693],{"class":87},"Selector",[63,19695,92],{"class":91},[63,19697,6347],{"class":95},[63,19699,142],{"class":91},[63,19701,19702],{"class":87},"kvp",[63,19704,784],{"class":91},[63,19706,19707],{"class":145},"$\"",[63,19709,5231],{"class":6466},[63,19711,19702],{"class":87},[63,19713,92],{"class":6466},[63,19715,19716],{"class":87},"Key",[63,19718,5237],{"class":6466},[63,19720,577],{"class":145},[63,19722,5231],{"class":6466},[63,19724,19702],{"class":87},[63,19726,92],{"class":6466},[63,19728,1135],{"class":87},[63,19730,5237],{"class":6466},[63,19732,5228],{"class":145},[63,19734,1106],{"class":91},[63,19736,19737],{"class":65,"line":5515},[63,19738,19739],{"class":91},"        );\n",[63,19741,19742],{"class":65,"line":5520},[63,19743,588],{"emptyLinePlaceholder":587},[63,19745,19746,19748,19750,19752,19755,19757,19759,19761,19764,19766,19768,19770,19773,19775,19778,19780,19782],{"class":65,"line":5545},[63,19747,593],{"class":439},[63,19749,19637],{"class":87},[63,19751,92],{"class":91},[63,19753,19754],{"class":87},"Items",[63,19756,92],{"class":91},[63,19758,6347],{"class":95},[63,19760,142],{"class":91},[63,19762,19763],{"class":87},"pod",[63,19765,784],{"class":91},[63,19767,19763],{"class":87},[63,19769,92],{"class":91},[63,19771,19772],{"class":87},"Status",[63,19774,92],{"class":91},[63,19776,19777],{"class":87},"PodIP",[63,19779,6359],{"class":91},[63,19781,14201],{"class":95},[63,19783,403],{"class":91},[63,19785,19786],{"class":65,"line":5550},[63,19787,621],{"class":91},[63,19789,19790],{"class":65,"line":8275},[63,19791,588],{"emptyLinePlaceholder":587},[63,19793,19794,19796,19798,19801],{"class":65,"line":8280},[63,19795,2964],{"class":439},[63,19797,14967],{"class":439},[63,19799,19800],{"class":95}," RefreshClient",[63,19802,5131],{"class":91},[63,19804,19805],{"class":65,"line":8285},[63,19806,250],{"class":91},[63,19808,19809,19812,19814,19817],{"class":65,"line":8290},[63,19810,19811],{"class":87},"        _clientsLock",[63,19813,92],{"class":91},[63,19815,19816],{"class":95},"EnterWriteLock",[63,19818,403],{"class":91},[63,19820,19821],{"class":65,"line":8299},[63,19822,12914],{"class":439},[63,19824,19825],{"class":65,"line":8304},[63,19826,1953],{"class":91},[63,19828,19829,19831,19834,19836,19839,19841,19844,19846,19849],{"class":65,"line":8309},[63,19830,8045],{"class":439},[63,19832,19833],{"class":528}," endpoints",[63,19835,133],{"class":132},[63,19837,19838],{"class":95}," GetPodsByServiceWithIpAsync",[63,19840,15694],{"class":91},[63,19842,19843],{"class":95},"GetAwaiter",[63,19845,15694],{"class":91},[63,19847,19848],{"class":95},"GetResult",[63,19850,403],{"class":91},[63,19852,19853],{"class":65,"line":8321},[63,19854,588],{"emptyLinePlaceholder":587},[63,19856,19857,19859,19862,19864,19866,19868,19870,19872,19874,19876],{"class":65,"line":13006},[63,19858,8045],{"class":439},[63,19860,19861],{"class":528}," newClients",[63,19863,133],{"class":132},[63,19865,136],{"class":91},[63,19867,19546],{"class":102},[63,19869,99],{"class":91},[63,19871,502],{"class":439},[63,19873,508],{"class":91},[63,19875,34],{"class":102},[63,19877,14277],{"class":91},[63,19879,19880,19883,19885,19887,19890,19892,19894],{"class":65,"line":13011},[63,19881,19882],{"class":439},"            foreach",[63,19884,3366],{"class":91},[63,19886,2067],{"class":439},[63,19888,19889],{"class":528}," endpoint",[63,19891,5215],{"class":439},[63,19893,19833],{"class":528},[63,19895,474],{"class":91},[63,19897,19898],{"class":65,"line":13025},[63,19899,7479],{"class":91},[63,19901,19902,19904,19906,19908,19910,19912,19914,19916,19919,19921,19923,19925,19928],{"class":65,"line":13030},[63,19903,15735],{"class":439},[63,19905,3366],{"class":91},[63,19907,3369],{"class":132},[63,19909,19384],{"class":87},[63,19911,92],{"class":91},[63,19913,7458],{"class":95},[63,19915,142],{"class":91},[63,19917,19918],{"class":528},"endpoint",[63,19920,508],{"class":91},[63,19922,7467],{"class":439},[63,19924,1690],{"class":439},[63,19926,19927],{"class":528}," existingClient",[63,19929,1106],{"class":91},[63,19931,19932],{"class":65,"line":13044},[63,19933,15763],{"class":91},[63,19935,19936,19939,19941,19943,19945,19948,19950],{"class":65,"line":13049},[63,19937,19938],{"class":528},"                    existingClient",[63,19940,133],{"class":132},[63,19942,136],{"class":91},[63,19944,34],{"class":102},[63,19946,19947],{"class":91},"(new ",[63,19949,13645],{"class":102},[63,19951,19952],{"class":91},"())\n",[63,19954,19955],{"class":65,"line":13080},[63,19956,19957],{"class":91},"                    {\n",[63,19959,19960,19963,19965,19967,19969,19971,19973],{"class":65,"line":13085},[63,19961,19962],{"class":528},"                        BaseAddress",[63,19964,133],{"class":132},[63,19966,136],{"class":91},[63,19968,139],{"class":102},[63,19970,142],{"class":91},[63,19972,19918],{"class":528},[63,19974,474],{"class":91},[63,19976,19977],{"class":65,"line":13090},[63,19978,19979],{"class":91},"                    };\n",[63,19981,19982],{"class":65,"line":15284},[63,19983,15809],{"class":91},[63,19985,19986,19989,19991,19993,19995,19997,19999],{"class":65,"line":15304},[63,19987,19988],{"class":87},"                newClients",[63,19990,4353],{"class":91},[63,19992,19918],{"class":528},[63,19994,574],{"class":91},[63,19996,577],{"class":132},[63,19998,19927],{"class":528},[63,20000,274],{"class":91},[63,20002,20003],{"class":65,"line":15327},[63,20004,7499],{"class":91},[63,20006,20007],{"class":65,"line":15332},[63,20008,588],{"emptyLinePlaceholder":587},[63,20010,20011,20014,20016,20018],{"class":65,"line":15349},[63,20012,20013],{"class":528},"            _persistentClients",[63,20015,133],{"class":132},[63,20017,19861],{"class":528},[63,20019,274],{"class":91},[63,20021,20022],{"class":65,"line":15354},[63,20023,7517],{"class":91},[63,20025,20026,20029],{"class":65,"line":15359},[63,20027,20028],{"class":439},"        finally",[63,20030,20031],{"class":91}," \n",[63,20033,20034],{"class":65,"line":15364},[63,20035,1953],{"class":91},[63,20037,20038,20041,20043,20046],{"class":65,"line":15377},[63,20039,20040],{"class":87},"            _clientsLock",[63,20042,92],{"class":91},[63,20044,20045],{"class":95},"ExitWriteLock",[63,20047,403],{"class":91},[63,20049,20050],{"class":65,"line":15382},[63,20051,7517],{"class":91},[63,20053,20054],{"class":65,"line":15387},[63,20055,621],{"class":91},[63,20057,20058],{"class":65,"line":15396},[63,20059,588],{"emptyLinePlaceholder":587},[63,20061,20062,20064,20066,20068,20070,20072,20074,20076,20078],{"class":65,"line":15408},[63,20063,2782],{"class":439},[63,20065,2763],{"class":439},[63,20067,2256],{"class":439},[63,20069,486],{"class":102},[63,20071,99],{"class":91},[63,20073,1844],{"class":102},[63,20075,1847],{"class":91},[63,20077,1827],{"class":95},[63,20079,219],{"class":91},[63,20081,20082,20085,20087],{"class":65,"line":15413},[63,20083,20084],{"class":102},"        HttpRequestMessage",[63,20086,1924],{"class":87},[63,20088,19658],{"class":91},[63,20090,20091,20094,20097],{"class":65,"line":15427},[63,20092,20093],{"class":102},"        CancellationToken",[63,20095,20096],{"class":87}," cancellationToken",[63,20098,474],{"class":91},[63,20100,20101],{"class":65,"line":15462},[63,20102,250],{"class":91},[63,20104,20105,20107,20109,20112],{"class":65,"line":15467},[63,20106,19811],{"class":87},[63,20108,92],{"class":91},[63,20110,20111],{"class":95},"EnterReadLock",[63,20113,403],{"class":91},[63,20115,20116],{"class":65,"line":15488},[63,20117,12914],{"class":439},[63,20119,20120],{"class":65,"line":15493},[63,20121,1953],{"class":91},[63,20123,20124,20126,20128,20130,20132,20134,20136,20138],{"class":65,"line":15498},[63,20125,7439],{"class":439},[63,20127,3366],{"class":91},[63,20129,19384],{"class":87},[63,20131,92],{"class":91},[63,20133,6898],{"class":87},[63,20135,8032],{"class":132},[63,20137,4464],{"class":289},[63,20139,474],{"class":91},[63,20141,20142],{"class":65,"line":15526},[63,20143,7479],{"class":91},[63,20145,20146,20149],{"class":65,"line":15531},[63,20147,20148],{"class":95},"                RefreshClient",[63,20150,403],{"class":91},[63,20152,20153],{"class":65,"line":15568},[63,20154,7499],{"class":91},[63,20156,20157],{"class":65,"line":15573},[63,20158,588],{"emptyLinePlaceholder":587},[63,20160,20161,20163,20165,20167,20170,20172,20175,20177,20179],{"class":65,"line":15587},[63,20162,8045],{"class":439},[63,20164,19833],{"class":528},[63,20166,133],{"class":132},[63,20168,20169],{"class":87}," _persistentClients",[63,20171,92],{"class":91},[63,20173,20174],{"class":87},"Keys",[63,20176,92],{"class":91},[63,20178,14201],{"class":95},[63,20180,403],{"class":91},[63,20182,20183,20185,20187,20190,20192,20194,20196,20198],{"class":65,"line":15593},[63,20184,7439],{"class":439},[63,20186,3366],{"class":91},[63,20188,20189],{"class":87},"endpoints",[63,20191,92],{"class":91},[63,20193,6898],{"class":87},[63,20195,8032],{"class":132},[63,20197,4464],{"class":289},[63,20199,474],{"class":91},[63,20201,20202],{"class":65,"line":15616},[63,20203,7479],{"class":91},[63,20205,20206,20209,20211,20214,20216,20219],{"class":65,"line":15656},[63,20207,20208],{"class":439},"                throw",[63,20210,136],{"class":91},[63,20212,20213],{"class":102},"InvalidOperationException",[63,20215,142],{"class":91},[63,20217,20218],{"class":145},"\"No endpoints available\"",[63,20220,149],{"class":91},[63,20222,20223],{"class":65,"line":15661},[63,20224,7499],{"class":91},[63,20226,20227],{"class":65,"line":15684},[63,20228,588],{"emptyLinePlaceholder":587},[63,20230,20231],{"class":65,"line":15711},[63,20232,20233],{"class":2731},"            \u002F\u002F Round-robin endpoint selection\n",[63,20235,20236,20239,20242,20244,20246],{"class":65,"line":15727},[63,20237,20238],{"class":439},"            string",[63,20240,20241],{"class":528}," selectedEndpoint",[63,20243,133],{"class":132},[63,20245,19833],{"class":87},[63,20247,20248],{"class":91},"[\n",[63,20250,20251,20254,20256,20259,20261,20264,20266,20268,20271,20273,20275,20277],{"class":65,"line":15732},[63,20252,20253],{"class":87},"                Interlocked",[63,20255,92],{"class":91},[63,20257,20258],{"class":95},"Increment",[63,20260,142],{"class":91},[63,20262,20263],{"class":439},"ref",[63,20265,19414],{"class":528},[63,20267,3420],{"class":91},[63,20269,20270],{"class":132},"%",[63,20272,19833],{"class":87},[63,20274,92],{"class":91},[63,20276,6898],{"class":87},[63,20278,5867],{"class":91},[63,20280,20281],{"class":65,"line":15760},[63,20282,588],{"emptyLinePlaceholder":587},[63,20284,20285,20287,20289,20291,20293,20295,20298],{"class":65,"line":15766},[63,20286,8045],{"class":439},[63,20288,2013],{"class":528},[63,20290,133],{"class":132},[63,20292,20169],{"class":87},[63,20294,4353],{"class":91},[63,20296,20297],{"class":528},"selectedEndpoint",[63,20299,5867],{"class":91},[63,20301,20302],{"class":65,"line":15786},[63,20303,588],{"emptyLinePlaceholder":587},[63,20305,20306],{"class":65,"line":15806},[63,20307,20308],{"class":2731},"            \u002F\u002F Clone the request for the specific endpoint\n",[63,20310,20311,20313,20316,20318,20320,20322,20324,20326,20328,20331,20333,20335,20337,20340],{"class":65,"line":15812},[63,20312,8045],{"class":439},[63,20314,20315],{"class":528}," newRequest",[63,20317,133],{"class":132},[63,20319,136],{"class":91},[63,20321,1931],{"class":102},[63,20323,142],{"class":91},[63,20325,2022],{"class":87},[63,20327,92],{"class":91},[63,20329,20330],{"class":87},"Method",[63,20332,508],{"class":91},[63,20334,2022],{"class":87},[63,20336,92],{"class":91},[63,20338,20339],{"class":87},"RequestUri",[63,20341,149],{"class":91},[63,20343,20344,20347,20349,20352,20354,20356,20358,20360],{"class":65,"line":15817},[63,20345,20346],{"class":87},"            newRequest",[63,20348,92],{"class":91},[63,20350,20351],{"class":87},"Content",[63,20353,133],{"class":132},[63,20355,1924],{"class":87},[63,20357,92],{"class":91},[63,20359,20351],{"class":87},[63,20361,274],{"class":91},[63,20363,20364,20366,20368,20370,20373,20375,20377,20379,20381],{"class":65,"line":15823},[63,20365,19882],{"class":439},[63,20367,3366],{"class":91},[63,20369,2067],{"class":439},[63,20371,20372],{"class":528}," header",[63,20374,5215],{"class":439},[63,20376,1924],{"class":87},[63,20378,92],{"class":91},[63,20380,1990],{"class":87},[63,20382,474],{"class":91},[63,20384,20385],{"class":65,"line":15828},[63,20386,7479],{"class":91},[63,20388,20389,20392,20394,20396,20398,20401,20403,20406,20408,20410,20412,20414,20416,20418],{"class":65,"line":15833},[63,20390,20391],{"class":87},"                newRequest",[63,20393,92],{"class":91},[63,20395,1990],{"class":87},[63,20397,92],{"class":91},[63,20399,20400],{"class":95},"TryAddWithoutValidation",[63,20402,142],{"class":91},[63,20404,20405],{"class":87},"header",[63,20407,92],{"class":91},[63,20409,19716],{"class":87},[63,20411,508],{"class":91},[63,20413,20405],{"class":87},[63,20415,92],{"class":91},[63,20417,1135],{"class":87},[63,20419,149],{"class":91},[63,20421,20422],{"class":65,"line":15852},[63,20423,7499],{"class":91},[63,20425,20426],{"class":65,"line":15879},[63,20427,588],{"emptyLinePlaceholder":587},[63,20429,20430,20432,20434,20436,20438,20440,20442,20445,20447,20450],{"class":65,"line":15885},[63,20431,4817],{"class":439},[63,20433,2301],{"class":91},[63,20435,1902],{"class":87},[63,20437,92],{"class":91},[63,20439,1827],{"class":95},[63,20441,142],{"class":91},[63,20443,20444],{"class":528},"newRequest",[63,20446,508],{"class":91},[63,20448,20449],{"class":528},"cancellationToken",[63,20451,149],{"class":91},[63,20453,20454],{"class":65,"line":15910},[63,20455,7517],{"class":91},[63,20457,20458,20460,20462,20464,20466],{"class":65,"line":15925},[63,20459,13033],{"class":439},[63,20461,3366],{"class":91},[63,20463,19011],{"class":102},[63,20465,12151],{"class":528},[63,20467,19016],{"class":91},[63,20469,20470,20473,20475,20477,20479,20481,20483,20485],{"class":65,"line":15965},[63,20471,20472],{"class":91},"            (",[63,20474,12176],{"class":102},[63,20476,92],{"class":91},[63,20478,19028],{"class":102},[63,20480,19031],{"class":528},[63,20482,19034],{"class":102},[63,20484,19037],{"class":528},[63,20486,19040],{"class":91},[63,20488,20489,20492,20494,20496,20498,20500,20502,20504,20506],{"class":65,"line":16006},[63,20490,20491],{"class":91},"              (",[63,20493,19048],{"class":102},[63,20495,92],{"class":91},[63,20497,19053],{"class":102},[63,20499,19056],{"class":91},[63,20501,19048],{"class":102},[63,20503,92],{"class":91},[63,20505,19053],{"class":102},[63,20507,19065],{"class":91},[63,20509,20510],{"class":65,"line":16016},[63,20511,1953],{"class":91},[63,20513,20514,20517],{"class":65,"line":16022},[63,20515,20516],{"class":95},"            RefreshClient",[63,20518,403],{"class":91},[63,20520,20522],{"class":65,"line":20521},98,[63,20523,7517],{"class":91},[63,20525,20527,20529],{"class":65,"line":20526},99,[63,20528,20028],{"class":439},[63,20530,20031],{"class":91},[63,20532,20534],{"class":65,"line":20533},100,[63,20535,1953],{"class":91},[63,20537,20539,20541,20543,20546],{"class":65,"line":20538},101,[63,20540,20040],{"class":87},[63,20542,92],{"class":91},[63,20544,20545],{"class":95},"ExitReadLock",[63,20547,403],{"class":91},[63,20549,20551],{"class":65,"line":20550},102,[63,20552,7517],{"class":91},[63,20554,20556],{"class":65,"line":20555},103,[63,20557,621],{"class":91},[63,20559,20561],{"class":65,"line":20560},104,[63,20562,626],{"class":91},[63,20564,20566],{"class":65,"line":20565},105,[63,20567,588],{"emptyLinePlaceholder":587},[63,20569,20571],{"class":65,"line":20570},106,[63,20572,588],{"emptyLinePlaceholder":587},[63,20574,20576,20578,20580,20582,20584,20587],{"class":65,"line":20575},107,[63,20577,88],{"class":87},[63,20579,92],{"class":91},[63,20581,96],{"class":95},[63,20583,142],{"class":91},[63,20585,20586],{"class":145},"\"MyClient\"",[63,20588,474],{"class":91},[63,20590,20592,20594,20596,20598,20601,20603,20606,20608,20611,20613,20616],{"class":65,"line":20591},108,[63,20593,1111],{"class":91},[63,20595,13515],{"class":95},[63,20597,142],{"class":91},[63,20599,20600],{"class":87},"sp",[63,20602,13523],{"class":91},[63,20604,20605],{"class":102},"RoundRobinK8sClientHandler",[63,20607,142],{"class":91},[63,20609,20610],{"class":145},"\"app-namespace\"",[63,20612,508],{"class":91},[63,20614,20615],{"class":145},"\"sample-service\"",[63,20617,366],{"class":91},[2563,20619,20620],{},"html pre.shiki code .sLKXg, html code.shiki .sLKXg{--shiki-default:#A626A4;--shiki-dark:#C678DD}html pre.shiki code .s5ixo, html code.shiki .s5ixo{--shiki-default:#383A42;--shiki-dark:#ABB2BF}html pre.shiki code .sC09Y, html code.shiki .sC09Y{--shiki-default:#C18401;--shiki-dark:#E5C07B}html pre.shiki code .sz0mV, html code.shiki .sz0mV{--shiki-default:#383A42;--shiki-dark:#E06C75}html pre.shiki code .s7GmK, html code.shiki .s7GmK{--shiki-default:#383A42;--shiki-dark:#E5C07B}html pre.shiki code .sAdtL, html code.shiki .sAdtL{--shiki-default:#4078F2;--shiki-dark:#61AFEF}html pre.shiki code .sDhpE, html code.shiki .sDhpE{--shiki-default:#50A14F;--shiki-dark:#98C379}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sJa8x, html code.shiki .sJa8x{--shiki-default:#E45649;--shiki-dark:#E06C75}html pre.shiki code .sknuh, html code.shiki .sknuh{--shiki-default:#383A42;--shiki-dark:#56B6C2}html pre.shiki code .sAGMh, html code.shiki .sAGMh{--shiki-default:#986801;--shiki-dark:#D19A66}html pre.shiki code .sMj0N, html code.shiki .sMj0N{--shiki-default:#50A14F;--shiki-dark:#ABB2BF}html pre.shiki code .sW2Sy, html code.shiki .sW2Sy{--shiki-default:#A0A1A7;--shiki-default-font-style:italic;--shiki-dark:#7F848E;--shiki-dark-font-style:italic}",{"title":59,"searchDepth":115,"depth":115,"links":20622},[20623,20626,20627],{"id":18900,"depth":115,"text":18901,"children":20624},[20625],{"id":18922,"depth":121,"text":18923},{"id":18937,"depth":115,"text":18938},{"id":18964,"depth":115,"text":18965,"children":20628},[20629,20630,20631,20632,20633,20634],{"id":18968,"depth":121,"text":18969},{"id":19102,"depth":121,"text":19103},{"id":19226,"depth":121,"text":19227},{"id":19257,"depth":121,"text":19258},{"id":19333,"depth":121,"text":19334},{"id":19340,"depth":121,"text":19341},"2024-12-09","How long lived connections interact with Kubernetes load balancing.",{},{"title":23,"description":20636},"blog\u002Fload-balancing-long-lived-connections-in-kubernetes",[2594],"NtCHtc8ODlX24UAKr_NdHFZb6vKZcXrCt_pY07tZOL4",{"id":20643,"title":20644,"body":20645,"book":2585,"date":21129,"description":21130,"extension":2588,"meta":21131,"navigation":587,"path":21132,"seo":21133,"stem":21134,"tags":21135,"__hash__":21136},"blog\u002Fblog\u002Fserver-sent-event-sse.md","Server Sent Event (SSE)",{"type":8,"value":20646,"toc":21123},[20647,20651,20654,20711,20715,20718,20721,20973,21006,21009,21057,21060,21066,21069,21072,21076,21079,21093,21096,21102,21106,21109,21120],[11,20648,20650],{"id":20649},"communication-type","Communication type",[16,20652,20653],{},"In the realm of real time transportations, there are some options to communicate between 2 entities (server and client)",[170,20655,20656,20670,20684,20697],{},[173,20657,20658,20659],{},"Long Polling",[170,20660,20661,20664,20667],{},[173,20662,20663],{},"The client sent an HTTP request to the server, the server will hold the request open until new data is available or time out occur",[173,20665,20666],{},"Pros: Work with older browser",[173,20668,20669],{},"Cons: Inefficient, high latency",[173,20671,20672,20673],{},"Short Polling",[170,20674,20675,20678,20681],{},[173,20676,20677],{},"Client sent an HTTP request to the server, the server will respond. Pull based request",[173,20679,20680],{},"Pros: Flexible with many systems",[173,20682,20683],{},"Cons: Frequent polling needed if the client doesn't know when new data is ready. Also slow to frequent update",[173,20685,20644,20686],{},[170,20687,20688,20691,20694],{},[173,20689,20690],{},"The client creates a persistent HTTP connection with the server. The server can send events to the client and the client can act accordingly",[173,20692,20693],{},"Pros: Simple one way server to client push based communication with low overhead",[173,20695,20696],{},"Cons: It only does what it does (no 2 way communication)",[173,20698,20699,20700],{},"WebSockets",[170,20701,20702,20705,20708],{},[173,20703,20704],{},"The client and server establish a full duplex communication channel over a single TCP connection. Both parties can send and receive data at any time.",[173,20706,20707],{},"Pros: 2 way communication",[173,20709,20710],{},"Cons: Most complex and not supported by all browsers",[11,20712,20714],{"id":20713},"sse","SSE",[16,20716,20717],{},"I will focus on Server Sent Events, their use cases and how to implement them.",[16,20719,20720],{},"In .NET Core, using minimal API, the server will expose this endpoint:",[54,20722,20724],{"className":78,"code":20723,"language":80,"meta":59,"style":59},"app.MapGet(\"\u002Fdata-stream\", async (HttpContext ctx, CancellationToken ct) =>\n{\n    ctx.Response.Headers.Append(\"Content-Type\", \"text\u002Fevent-stream\");\n    \n    while (!ct.IsCancellationRequested)\n    {\n        var randomNumber = new Random().Next(0, 100);\n        var data = $\"data: {randomNumber}\\n\\n\";\n        \n        await ctx.Response.WriteAsync(data, cancellationToken: ct);\n        await ctx.Response.Body.FlushAsync(cancellationToken: ct);\n        \n        await Task.Delay(1000, ct); \u002F\u002F Send a new number every second\n    }\n});\n",[32,20725,20726,20759,20763,20792,20796,20812,20816,20843,20869,20873,20905,20935,20939,20965,20969],{"__ignoreMap":59},[63,20727,20728,20730,20732,20735,20737,20740,20742,20744,20746,20748,20751,20753,20755,20757],{"class":65,"line":66},[63,20729,7562],{"class":87},[63,20731,92],{"class":91},[63,20733,20734],{"class":95},"MapGet",[63,20736,142],{"class":91},[63,20738,20739],{"class":145},"\"\u002Fdata-stream\"",[63,20741,508],{"class":91},[63,20743,7863],{"class":439},[63,20745,3366],{"class":91},[63,20747,7578],{"class":102},[63,20749,20750],{"class":87}," ctx",[63,20752,508],{"class":91},[63,20754,511],{"class":102},[63,20756,514],{"class":87},[63,20758,1910],{"class":91},[63,20760,20761],{"class":65,"line":115},[63,20762,118],{"class":91},[63,20764,20765,20768,20770,20772,20774,20776,20778,20780,20782,20785,20787,20790],{"class":65,"line":121},[63,20766,20767],{"class":87},"    ctx",[63,20769,92],{"class":91},[63,20771,7091],{"class":87},[63,20773,92],{"class":91},[63,20775,1990],{"class":87},[63,20777,92],{"class":91},[63,20779,7101],{"class":95},[63,20781,142],{"class":91},[63,20783,20784],{"class":145},"\"Content-Type\"",[63,20786,508],{"class":91},[63,20788,20789],{"class":145},"\"text\u002Fevent-stream\"",[63,20791,149],{"class":91},[63,20793,20794],{"class":65,"line":152},[63,20795,14282],{"class":91},[63,20797,20798,20800,20802,20804,20806,20808,20810],{"class":65,"line":253},[63,20799,5827],{"class":439},[63,20801,3366],{"class":91},[63,20803,3369],{"class":132},[63,20805,614],{"class":87},[63,20807,92],{"class":91},[63,20809,3376],{"class":87},[63,20811,474],{"class":91},[63,20813,20814],{"class":65,"line":277},[63,20815,250],{"class":91},[63,20817,20818,20820,20823,20825,20827,20829,20831,20833,20835,20837,20839,20841],{"class":65,"line":295},[63,20819,525],{"class":439},[63,20821,20822],{"class":528}," randomNumber",[63,20824,133],{"class":132},[63,20826,136],{"class":91},[63,20828,14560],{"class":102},[63,20830,15694],{"class":91},[63,20832,14524],{"class":95},[63,20834,142],{"class":91},[63,20836,867],{"class":289},[63,20838,508],{"class":91},[63,20840,1176],{"class":289},[63,20842,149],{"class":91},[63,20844,20845,20847,20850,20852,20855,20857,20860,20862,20865,20867],{"class":65,"line":301},[63,20846,525],{"class":439},[63,20848,20849],{"class":528}," data",[63,20851,133],{"class":132},[63,20853,20854],{"class":145}," $\"data: ",[63,20856,5231],{"class":6466},[63,20858,20859],{"class":528},"randomNumber",[63,20861,5237],{"class":6466},[63,20863,20864],{"class":5259},"\\n\\n",[63,20866,5228],{"class":145},[63,20868,274],{"class":91},[63,20870,20871],{"class":65,"line":313},[63,20872,14387],{"class":91},[63,20874,20875,20878,20881,20883,20885,20887,20890,20892,20895,20897,20899,20901,20903],{"class":65,"line":318},[63,20876,20877],{"class":91},"        await ",[63,20879,20880],{"class":87},"ctx",[63,20882,92],{"class":91},[63,20884,7091],{"class":87},[63,20886,92],{"class":91},[63,20888,20889],{"class":95},"WriteAsync",[63,20891,142],{"class":91},[63,20893,20894],{"class":528},"data",[63,20896,508],{"class":91},[63,20898,20449],{"class":87},[63,20900,227],{"class":91},[63,20902,614],{"class":528},[63,20904,149],{"class":91},[63,20906,20907,20909,20911,20913,20915,20917,20920,20922,20925,20927,20929,20931,20933],{"class":65,"line":340},[63,20908,20877],{"class":91},[63,20910,20880],{"class":87},[63,20912,92],{"class":91},[63,20914,7091],{"class":87},[63,20916,92],{"class":91},[63,20918,20919],{"class":87},"Body",[63,20921,92],{"class":91},[63,20923,20924],{"class":95},"FlushAsync",[63,20926,142],{"class":91},[63,20928,20449],{"class":87},[63,20930,227],{"class":91},[63,20932,614],{"class":528},[63,20934,149],{"class":91},[63,20936,20937],{"class":65,"line":369},[63,20938,14387],{"class":91},[63,20940,20941,20943,20946,20948,20951,20953,20956,20958,20960,20962],{"class":65,"line":374},[63,20942,20877],{"class":91},[63,20944,20945],{"class":87},"Task",[63,20947,92],{"class":91},[63,20949,20950],{"class":95},"Delay",[63,20952,142],{"class":91},[63,20954,20955],{"class":289},"1000",[63,20957,508],{"class":91},[63,20959,614],{"class":528},[63,20961,2994],{"class":91},[63,20963,20964],{"class":2731},"\u002F\u002F Send a new number every second\n",[63,20966,20967],{"class":65,"line":387},[63,20968,621],{"class":91},[63,20970,20971],{"class":65,"line":392},[63,20972,155],{"class":91},[170,20974,20975,20983],{},[173,20976,20977,20978,3246,20980,20982],{},"A ",[32,20979,7578],{},[32,20981,511],{}," are needed.",[173,20984,20985,20986,4020,20988,20990],{},"We directly modify the ",[32,20987,7578],{},[32,20989,7091],{},[170,20991,20992,21001],{},[173,20993,11699,20994,20997,20998],{},[32,20995,20996],{},"Content-Type"," header will be ",[32,20999,21000],{},"text\u002Fevent-stream",[173,21002,21003,21004,1860],{},"Data is directly written to the ",[32,21005,7091],{},[16,21007,21008],{},"Any Javascript Client that want to receive the event will implement",[54,21010,21014],{"className":21011,"code":21012,"language":21013,"meta":59,"style":59},"language-javascript shiki shiki-themes one-light one-dark-pro","const eventSource = new EventSource(url + \"\u002Fdata-stream\");\n\neventSource.onmessage = function(event) {\n    console.log(event.data)\n}\n\neventSource.onerror = function(event) {\n    \u002F\u002F log error\n}\n","javascript",[32,21015,21016,21021,21025,21030,21035,21039,21043,21048,21053],{"__ignoreMap":59},[63,21017,21018],{"class":65,"line":66},[63,21019,21020],{},"const eventSource = new EventSource(url + \"\u002Fdata-stream\");\n",[63,21022,21023],{"class":65,"line":115},[63,21024,588],{"emptyLinePlaceholder":587},[63,21026,21027],{"class":65,"line":121},[63,21028,21029],{},"eventSource.onmessage = function(event) {\n",[63,21031,21032],{"class":65,"line":152},[63,21033,21034],{},"    console.log(event.data)\n",[63,21036,21037],{"class":65,"line":253},[63,21038,626],{},[63,21040,21041],{"class":65,"line":277},[63,21042,588],{"emptyLinePlaceholder":587},[63,21044,21045],{"class":65,"line":295},[63,21046,21047],{},"eventSource.onerror = function(event) {\n",[63,21049,21050],{"class":65,"line":301},[63,21051,21052],{},"    \u002F\u002F log error\n",[63,21054,21055],{"class":65,"line":313},[63,21056,626],{},[16,21058,21059],{},"If you inspect the client, you will see",[16,21061,21062],{},[7330,21063],{"alt":21064,"src":21065},"image-20241206-043340.png","\u002Fblog\u002Fserver-sent-event-sse\u002Fimage_20241206_043340.png",[16,21067,21068],{},"Any time a new client join, they will start receiving events",[16,21070,21071],{},"Note that there is no way to manage clients from server side (how many clients are connected, manually disconnect client, etc.) due to the nature of SSE.",[11,21073,21075],{"id":21074},"use-cases","Use cases",[16,21077,21078],{},"The most common use cases are real time updates and notifications",[1789,21080,21081,21084,21087,21090],{},[173,21082,21083],{},"Social media notifications",[173,21085,21086],{},"Notification service events",[173,21088,21089],{},"News and stock tickers",[173,21091,21092],{},"Financial Dashboard",[16,21094,21095],{},"An example is a social platform pushing engagement events (like count, view count, reply count, etc.) to the client using SSE",[16,21097,21098],{},[7330,21099],{"alt":21100,"src":21101},"image-20241206-044035.png","\u002Fblog\u002Fserver-sent-event-sse\u002Fimage_20241206_044035.png",[11,21103,21105],{"id":21104},"extra-thing-to-consider","Extra thing to consider",[16,21107,21108],{},"Even though SSE is very simple, there are a few things you can consider with regard to the event stream",[1789,21110,21111,21114,21117],{},[173,21112,21113],{},"Compress data and optimize data format (JSON or Protocol Buffer) if the data size is big",[173,21115,21116],{},"Batch data together (send every interval with batches instead of instantly if UX is not affected)",[173,21118,21119],{},"Prioritize Events: important events can be sent first, and\u002For more often",[2563,21121,21122],{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .s7GmK, html code.shiki .s7GmK{--shiki-default:#383A42;--shiki-dark:#E5C07B}html pre.shiki code .s5ixo, html code.shiki .s5ixo{--shiki-default:#383A42;--shiki-dark:#ABB2BF}html pre.shiki code .sAdtL, html code.shiki .sAdtL{--shiki-default:#4078F2;--shiki-dark:#61AFEF}html pre.shiki code .sDhpE, html code.shiki .sDhpE{--shiki-default:#50A14F;--shiki-dark:#98C379}html pre.shiki code .sLKXg, html code.shiki .sLKXg{--shiki-default:#A626A4;--shiki-dark:#C678DD}html pre.shiki code .sC09Y, html code.shiki .sC09Y{--shiki-default:#C18401;--shiki-dark:#E5C07B}html pre.shiki code .sknuh, html code.shiki .sknuh{--shiki-default:#383A42;--shiki-dark:#56B6C2}html pre.shiki code .sz0mV, html code.shiki .sz0mV{--shiki-default:#383A42;--shiki-dark:#E06C75}html pre.shiki code .sAGMh, html code.shiki .sAGMh{--shiki-default:#986801;--shiki-dark:#D19A66}html pre.shiki code .sMj0N, html code.shiki .sMj0N{--shiki-default:#50A14F;--shiki-dark:#ABB2BF}html pre.shiki code .s_Sar, html code.shiki .s_Sar{--shiki-default:#0184BC;--shiki-dark:#56B6C2}html pre.shiki code .sW2Sy, html code.shiki .sW2Sy{--shiki-default:#A0A1A7;--shiki-default-font-style:italic;--shiki-dark:#7F848E;--shiki-dark-font-style:italic}",{"title":59,"searchDepth":115,"depth":115,"links":21124},[21125,21126,21127,21128],{"id":20649,"depth":115,"text":20650},{"id":20713,"depth":115,"text":20714},{"id":21074,"depth":115,"text":21075},{"id":21104,"depth":115,"text":21105},"2024-11-25","A practical look at Server Sent Events and when to use them.",{},"\u002Fblog\u002Fserver-sent-event-sse",{"title":20644,"description":21130},"blog\u002Fserver-sent-event-sse",[2594],"Iz2eE0FortTeX5hSinziaYj0s3YI30qk03YlOR6MGP0",{"id":21138,"title":21139,"body":21140,"book":2585,"date":22029,"description":22030,"extension":2588,"meta":22031,"navigation":587,"path":22032,"seo":22033,"stem":22034,"tags":22035,"__hash__":22036},"blog\u002Fblog\u002Fhiding-sensitive-information-source-code.md","Hiding sensitive information source code",{"type":8,"value":21141,"toc":22021},[21142,21145,21149,21152,21160,21234,21237,21366,21369,21417,21420,21453,21456,21459,21464,21468,21471,21603,21606,21642,21644,21655,21658,21666,21670,21673,21761,21764,21864,21867,21871,21874,21897,21900,21904,21907,21922,22007,22011,22018],[16,21143,21144],{},"A k8s Secret is a small an object that stores a small sensitive amount of data, that will be put into your Deployment or Pod specification instead of your source code. Because Secret can be created and deployed separately and independently of the Pod that uses them, there is less risk of exposure.",[11,21146,21148],{"id":21147},"use-secret-as-environment-variables","Use Secret as environment variables",[16,21150,21151],{},"First you can create your Secret like this. Note that:",[1789,21153,21154,21157],{},[173,21155,21156],{},"Secret is namespace scope, you cannot share between namespaces.",[173,21158,21159],{},"Value of the secret must be base64 encoded, even though the original value will be stored.",[54,21161,21163],{"className":19124,"code":21162,"language":19126,"meta":59,"style":59},"apiVersion: v1\nkind: Secret\nmetadata:\n  name: app-secrets\n  namespace: app-namespace\ntype: Opaque\ndata:\n  redis-connection-string: \u003Cyour value as a base64 value>\n",[32,21164,21165,21174,21183,21189,21198,21208,21218,21224],{"__ignoreMap":59},[63,21166,21167,21169,21171],{"class":65,"line":66},[63,21168,19133],{"class":2976},[63,21170,227],{"class":91},[63,21172,21173],{"class":145},"v1\n",[63,21175,21176,21178,21180],{"class":65,"line":115},[63,21177,19143],{"class":2976},[63,21179,227],{"class":91},[63,21181,21182],{"class":145},"Secret\n",[63,21184,21185,21187],{"class":65,"line":121},[63,21186,19153],{"class":2976},[63,21188,10453],{"class":91},[63,21190,21191,21193,21195],{"class":65,"line":152},[63,21192,19160],{"class":2976},[63,21194,227],{"class":91},[63,21196,21197],{"class":145},"app-secrets\n",[63,21199,21200,21203,21205],{"class":65,"line":253},[63,21201,21202],{"class":2976},"  namespace",[63,21204,227],{"class":91},[63,21206,21207],{"class":145},"app-namespace\n",[63,21209,21210,21213,21215],{"class":65,"line":277},[63,21211,21212],{"class":2976},"type",[63,21214,227],{"class":91},[63,21216,21217],{"class":145},"Opaque\n",[63,21219,21220,21222],{"class":65,"line":295},[63,21221,20894],{"class":2976},[63,21223,10453],{"class":91},[63,21225,21226,21229,21231],{"class":65,"line":301},[63,21227,21228],{"class":2976},"  redis-connection-string",[63,21230,227],{"class":91},[63,21232,21233],{"class":145},"\u003Cyour value as a base64 value>\n",[16,21235,21236],{},"After that you can use this variable in your Deployment",[54,21238,21240],{"className":19124,"code":21239,"language":19126,"meta":59,"style":59},"apiVersion: apps\u002Fv1\nkind: Deployment\nmetadata:\n  name: your-app\nspec:\n  template:\n    spec:\n      containers:\n      - name: your-app\n        env:\n        - name: ConnectionStrings__Redis\n          valueFrom:\n            secretKeyRef:\n              name: app-secrets\n              key: redis-connection-string\n",[32,21241,21242,21251,21260,21266,21275,21281,21288,21295,21302,21314,21321,21333,21340,21347,21356],{"__ignoreMap":59},[63,21243,21244,21246,21248],{"class":65,"line":66},[63,21245,19133],{"class":2976},[63,21247,227],{"class":91},[63,21249,21250],{"class":145},"apps\u002Fv1\n",[63,21252,21253,21255,21257],{"class":65,"line":115},[63,21254,19143],{"class":2976},[63,21256,227],{"class":91},[63,21258,21259],{"class":145},"Deployment\n",[63,21261,21262,21264],{"class":65,"line":121},[63,21263,19153],{"class":2976},[63,21265,10453],{"class":91},[63,21267,21268,21270,21272],{"class":65,"line":152},[63,21269,19160],{"class":2976},[63,21271,227],{"class":91},[63,21273,21274],{"class":145},"your-app\n",[63,21276,21277,21279],{"class":65,"line":253},[63,21278,19170],{"class":2976},[63,21280,10453],{"class":91},[63,21282,21283,21286],{"class":65,"line":277},[63,21284,21285],{"class":2976},"  template",[63,21287,10453],{"class":91},[63,21289,21290,21293],{"class":65,"line":295},[63,21291,21292],{"class":2976},"    spec",[63,21294,10453],{"class":91},[63,21296,21297,21300],{"class":65,"line":301},[63,21298,21299],{"class":2976},"      containers",[63,21301,10453],{"class":91},[63,21303,21304,21307,21310,21312],{"class":65,"line":313},[63,21305,21306],{"class":91},"      - ",[63,21308,21309],{"class":2976},"name",[63,21311,227],{"class":91},[63,21313,21274],{"class":145},[63,21315,21316,21319],{"class":65,"line":318},[63,21317,21318],{"class":2976},"        env",[63,21320,10453],{"class":91},[63,21322,21323,21326,21328,21330],{"class":65,"line":340},[63,21324,21325],{"class":91},"        - ",[63,21327,21309],{"class":2976},[63,21329,227],{"class":91},[63,21331,21332],{"class":145},"ConnectionStrings__Redis\n",[63,21334,21335,21338],{"class":65,"line":369},[63,21336,21337],{"class":2976},"          valueFrom",[63,21339,10453],{"class":91},[63,21341,21342,21345],{"class":65,"line":374},[63,21343,21344],{"class":2976},"            secretKeyRef",[63,21346,10453],{"class":91},[63,21348,21349,21352,21354],{"class":65,"line":387},[63,21350,21351],{"class":2976},"              name",[63,21353,227],{"class":91},[63,21355,21197],{"class":145},[63,21357,21358,21361,21363],{"class":65,"line":392},[63,21359,21360],{"class":2976},"              key",[63,21362,227],{"class":91},[63,21364,21365],{"class":145},"redis-connection-string\n",[16,21367,21368],{},"And by calling",[54,21370,21372],{"className":78,"code":21371,"language":80,"meta":59,"style":59},"var builder = WebApplication.CreateBuilder(args);   \nbuilder.Configuration\n    .AddEnvironmentVariables();\n",[32,21373,21374,21399,21408],{"__ignoreMap":59},[63,21375,21376,21378,21381,21383,21386,21388,21391,21393,21396],{"class":65,"line":66},[63,21377,2067],{"class":439},[63,21379,21380],{"class":528}," builder",[63,21382,133],{"class":132},[63,21384,21385],{"class":87}," WebApplication",[63,21387,92],{"class":91},[63,21389,21390],{"class":95},"CreateBuilder",[63,21392,142],{"class":91},[63,21394,21395],{"class":528},"args",[63,21397,21398],{"class":91},");   \n",[63,21400,21401,21403,21405],{"class":65,"line":115},[63,21402,206],{"class":87},[63,21404,92],{"class":91},[63,21406,21407],{"class":87},"Configuration\n",[63,21409,21410,21412,21415],{"class":65,"line":121},[63,21411,1111],{"class":91},[63,21413,21414],{"class":95},"AddEnvironmentVariables",[63,21416,403],{"class":91},[16,21418,21419],{},"you will have the same effect of",[54,21421,21423],{"className":3806,"code":21422,"language":3808,"meta":59,"style":59},"{\n  \"ConnectionStrings\": {\n    \"Redis\": \"\u003Cyour value>\"\n  }\n}\n",[32,21424,21425,21429,21435,21445,21449],{"__ignoreMap":59},[63,21426,21427],{"class":65,"line":66},[63,21428,118],{"class":91},[63,21430,21431,21433],{"class":65,"line":115},[63,21432,3819],{"class":2976},[63,21434,3840],{"class":91},[63,21436,21437,21440,21442],{"class":65,"line":121},[63,21438,21439],{"class":2976},"    \"Redis\"",[63,21441,227],{"class":91},[63,21443,21444],{"class":145},"\"\u003Cyour value>\"\n",[63,21446,21447],{"class":65,"line":152},[63,21448,3879],{"class":91},[63,21450,21451],{"class":65,"line":253},[63,21452,626],{"class":91},[16,21454,21455],{},"in your configuration.",[16,21457,21458],{},"Note that:",[1789,21460,21461],{},[173,21462,21463],{},"Since Secrets are namespace specific, you can use it to store common secret data across your apps like your Redis password, Kafka credentials, third party accounts, etc. that are shared between your microservices",[11,21465,21467],{"id":21466},"mount-the-secret-as-volume","Mount the Secret as volume",[16,21469,21470],{},"You can also mount the secret as a Persistent Volume",[54,21472,21474],{"className":19124,"code":21473,"language":19126,"meta":59,"style":59},"kind: Deployment\nmetadata:\n  name: your-app\nspec:\n  template:\n    spec:\n      containers:\n      - name: your-app\n        volumeMounts:\n        - name: secrets\n          mountPath: \u002Fetc\u002Fsecrets\n          readOnly: true\n      volumes:\n      - name: secrets\n        secret:\n          secretName: app-secrets\n",[32,21475,21476,21484,21490,21498,21504,21510,21516,21522,21532,21539,21550,21560,21570,21577,21587,21594],{"__ignoreMap":59},[63,21477,21478,21480,21482],{"class":65,"line":66},[63,21479,19143],{"class":2976},[63,21481,227],{"class":91},[63,21483,21259],{"class":145},[63,21485,21486,21488],{"class":65,"line":115},[63,21487,19153],{"class":2976},[63,21489,10453],{"class":91},[63,21491,21492,21494,21496],{"class":65,"line":121},[63,21493,19160],{"class":2976},[63,21495,227],{"class":91},[63,21497,21274],{"class":145},[63,21499,21500,21502],{"class":65,"line":152},[63,21501,19170],{"class":2976},[63,21503,10453],{"class":91},[63,21505,21506,21508],{"class":65,"line":253},[63,21507,21285],{"class":2976},[63,21509,10453],{"class":91},[63,21511,21512,21514],{"class":65,"line":277},[63,21513,21292],{"class":2976},[63,21515,10453],{"class":91},[63,21517,21518,21520],{"class":65,"line":295},[63,21519,21299],{"class":2976},[63,21521,10453],{"class":91},[63,21523,21524,21526,21528,21530],{"class":65,"line":301},[63,21525,21306],{"class":91},[63,21527,21309],{"class":2976},[63,21529,227],{"class":91},[63,21531,21274],{"class":145},[63,21533,21534,21537],{"class":65,"line":313},[63,21535,21536],{"class":2976},"        volumeMounts",[63,21538,10453],{"class":91},[63,21540,21541,21543,21545,21547],{"class":65,"line":318},[63,21542,21325],{"class":91},[63,21544,21309],{"class":2976},[63,21546,227],{"class":91},[63,21548,21549],{"class":145},"secrets\n",[63,21551,21552,21555,21557],{"class":65,"line":340},[63,21553,21554],{"class":2976},"          mountPath",[63,21556,227],{"class":91},[63,21558,21559],{"class":145},"\u002Fetc\u002Fsecrets\n",[63,21561,21562,21565,21567],{"class":65,"line":369},[63,21563,21564],{"class":2976},"          readOnly",[63,21566,227],{"class":91},[63,21568,21569],{"class":289},"true\n",[63,21571,21572,21575],{"class":65,"line":374},[63,21573,21574],{"class":2976},"      volumes",[63,21576,10453],{"class":91},[63,21578,21579,21581,21583,21585],{"class":65,"line":387},[63,21580,21306],{"class":91},[63,21582,21309],{"class":2976},[63,21584,227],{"class":91},[63,21586,21549],{"class":145},[63,21588,21589,21592],{"class":65,"line":392},[63,21590,21591],{"class":2976},"        secret",[63,21593,10453],{"class":91},[63,21595,21596,21599,21601],{"class":65,"line":406},[63,21597,21598],{"class":2976},"          secretName",[63,21600,227],{"class":91},[63,21602,21197],{"class":145},[16,21604,21605],{},"and add every key value pairs from that directory to your code by calling",[54,21607,21609],{"className":78,"code":21608,"language":80,"meta":59,"style":59},"builder.Configuration\n    .AddKeyPerFile(\"\u002Fetc\u002Fsecrets\", optional: true);\n",[32,21610,21611,21619],{"__ignoreMap":59},[63,21612,21613,21615,21617],{"class":65,"line":66},[63,21614,206],{"class":87},[63,21616,92],{"class":91},[63,21618,21407],{"class":87},[63,21620,21621,21623,21626,21628,21631,21633,21636,21638,21640],{"class":65,"line":115},[63,21622,1111],{"class":91},[63,21624,21625],{"class":95},"AddKeyPerFile",[63,21627,142],{"class":91},[63,21629,21630],{"class":145},"\"\u002Fetc\u002Fsecrets\"",[63,21632,508],{"class":91},[63,21634,21635],{"class":87},"optional",[63,21637,227],{"class":91},[63,21639,1206],{"class":289},[63,21641,149],{"class":91},[16,21643,19209],{},[1789,21645,21646,21649,21652],{},[173,21647,21648],{},"More complicated",[173,21650,21651],{},"Written to file system so there is higher attack surface compared to in memory",[173,21653,21654],{},"Slower startup time to read from mounted volume",[16,21656,21657],{},"Pros:",[1789,21659,21660,21663],{},[173,21661,21662],{},"When your secret is actually a file like a cert or configuration file (e.g. cloud provider credentials)",[173,21664,21665],{},"When you need to handle a large numbers of environment variables",[11,21667,21669],{"id":21668},"mount-the-secret-file-as-volume","Mount the secret FILE as volume",[16,21671,21672],{},"You can move the data sensitive part of your configuration and mount it as a volume. And you can merge that configuration with the non sensitive part on startup.",[54,21674,21676],{"className":19124,"code":21675,"language":19126,"meta":59,"style":59},"apiVersion: v1\nkind: Secret\nmetadata:\n  name: app-settings\ntype: Opaque\ndata:\n  appsettings.Secrets.Production.json: |\n    {\n      \"RedisConfiguration\": {\n        \"Enable\": true,\n        \"ConnectionString\": \u003Csome string>\n      }\n    }\n",[32,21677,21678,21686,21694,21700,21709,21717,21723,21733,21737,21742,21747,21752,21757],{"__ignoreMap":59},[63,21679,21680,21682,21684],{"class":65,"line":66},[63,21681,19133],{"class":2976},[63,21683,227],{"class":91},[63,21685,21173],{"class":145},[63,21687,21688,21690,21692],{"class":65,"line":115},[63,21689,19143],{"class":2976},[63,21691,227],{"class":91},[63,21693,21182],{"class":145},[63,21695,21696,21698],{"class":65,"line":121},[63,21697,19153],{"class":2976},[63,21699,10453],{"class":91},[63,21701,21702,21704,21706],{"class":65,"line":152},[63,21703,19160],{"class":2976},[63,21705,227],{"class":91},[63,21707,21708],{"class":145},"app-settings\n",[63,21710,21711,21713,21715],{"class":65,"line":253},[63,21712,21212],{"class":2976},[63,21714,227],{"class":91},[63,21716,21217],{"class":145},[63,21718,21719,21721],{"class":65,"line":277},[63,21720,20894],{"class":2976},[63,21722,10453],{"class":91},[63,21724,21725,21728,21730],{"class":65,"line":295},[63,21726,21727],{"class":2976},"  appsettings.Secrets.Production.json",[63,21729,227],{"class":91},[63,21731,21732],{"class":439},"|\n",[63,21734,21735],{"class":65,"line":301},[63,21736,250],{"class":145},[63,21738,21739],{"class":65,"line":313},[63,21740,21741],{"class":145},"      \"RedisConfiguration\": {\n",[63,21743,21744],{"class":65,"line":318},[63,21745,21746],{"class":145},"        \"Enable\": true,\n",[63,21748,21749],{"class":65,"line":340},[63,21750,21751],{"class":145},"        \"ConnectionString\": \u003Csome string>\n",[63,21753,21754],{"class":65,"line":369},[63,21755,21756],{"class":145},"      }\n",[63,21758,21759],{"class":65,"line":374},[63,21760,621],{"class":145},[16,21762,21763],{},"Then you can merge them together",[54,21765,21767],{"className":78,"code":21766,"language":80,"meta":59,"style":59},"builder.Configuration\n    .AddJsonFile(\"appsettings.json\", true)\n    .AddJsonFile($\"appsettings.{builder.Environment.EnvironmentName}.json\", true)\n    .AddJsonFile($\"\u002Fetc\u002Fsecrets\u002Fappsettings.Secrets.{builder.Environment.EnvironmentName}.json\", true);\n",[32,21768,21769,21777,21795,21831],{"__ignoreMap":59},[63,21770,21771,21773,21775],{"class":65,"line":66},[63,21772,206],{"class":87},[63,21774,92],{"class":91},[63,21776,21407],{"class":87},[63,21778,21779,21781,21784,21786,21789,21791,21793],{"class":65,"line":115},[63,21780,1111],{"class":91},[63,21782,21783],{"class":95},"AddJsonFile",[63,21785,142],{"class":91},[63,21787,21788],{"class":145},"\"appsettings.json\"",[63,21790,508],{"class":91},[63,21792,1206],{"class":289},[63,21794,474],{"class":91},[63,21796,21797,21799,21801,21803,21806,21808,21810,21812,21815,21817,21820,21822,21825,21827,21829],{"class":65,"line":121},[63,21798,1111],{"class":91},[63,21800,21783],{"class":95},[63,21802,142],{"class":91},[63,21804,21805],{"class":145},"$\"appsettings.",[63,21807,5231],{"class":6466},[63,21809,206],{"class":87},[63,21811,92],{"class":6466},[63,21813,21814],{"class":87},"Environment",[63,21816,92],{"class":6466},[63,21818,21819],{"class":87},"EnvironmentName",[63,21821,5237],{"class":6466},[63,21823,21824],{"class":145},".json\"",[63,21826,508],{"class":91},[63,21828,1206],{"class":289},[63,21830,474],{"class":91},[63,21832,21833,21835,21837,21839,21842,21844,21846,21848,21850,21852,21854,21856,21858,21860,21862],{"class":65,"line":152},[63,21834,1111],{"class":91},[63,21836,21783],{"class":95},[63,21838,142],{"class":91},[63,21840,21841],{"class":145},"$\"\u002Fetc\u002Fsecrets\u002Fappsettings.Secrets.",[63,21843,5231],{"class":6466},[63,21845,206],{"class":87},[63,21847,92],{"class":6466},[63,21849,21814],{"class":87},[63,21851,92],{"class":6466},[63,21853,21819],{"class":87},[63,21855,5237],{"class":6466},[63,21857,21824],{"class":145},[63,21859,508],{"class":91},[63,21861,1206],{"class":289},[63,21863,149],{"class":91},[16,21865,21866],{},"This is the most complicated but good when you have complicated configuration file",[11,21868,21870],{"id":21869},"how-about-secret-locally","How about Secret locally",[16,21872,21873],{},"The best practice is to not have any secrets in your local development, but sometimes, you cannot avoid that. There are 2 main ways to avoid this:",[170,21875,21876,21883],{},[173,21877,21878,21879,21882],{},"You can use ",[32,21880,21881],{},"user-secrets",". For more information, you can check out the ASP.NET Core documentation for safe local secret storage.",[173,21884,21885,21886,21889,21890,21893,21894,21896],{},"You don't check in ",[32,21887,21888],{},"appsettings.Development.json",". Make sure ",[32,21891,21892],{},"appsettings.json"," has every key ",[32,21895,21892],{}," has.",[16,21898,21899],{},"In both case, your team should know about the setup for this project code base.",[11,21901,21903],{"id":21902},"how-about-secret-in-pipeline","How about Secret in pipeline",[16,21905,21906],{},"If you need some kind of Secret in pipeline (Integration or end to end testings) and have no access to k8s Secrets, you need to use the CI\u002FCD tools to inject the Secret or the file:",[170,21908,21909,21919],{},[173,21910,21911,21912,3246,21915,21918],{},"Using CI\u002FCD variables: use ",[32,21913,21914],{},"sed",[32,21916,21917],{},"cat"," to inject or create the file and add in repository before script",[173,21920,21921],{},"Using secured files: if you need to add an entire file in the pipeline, use the secured file feature from your CI\u002FCD platform.",[54,21923,21925],{"className":19124,"code":21924,"language":19126,"meta":59,"style":59},"test:\n  stage: test\n  tags:\n    - docker\n  image: mcr.microsoft.com\u002Fdotnet\u002Fsdk:8.0\n  variables:\n    SECURE_FILES_DOWNLOAD_PATH: '.\u002FMyProject\u002F'\n  script:\n    - curl --silent \"https:\u002F\u002Fci.example.com\u002Fsecure_files\u002Finstaller\" | bash\n    - dotnet test MyProject.IntegrationTests\n",[32,21926,21927,21934,21944,21951,21959,21969,21976,21986,21993,22000],{"__ignoreMap":59},[63,21928,21929,21932],{"class":65,"line":66},[63,21930,21931],{"class":2976},"test",[63,21933,10453],{"class":91},[63,21935,21936,21939,21941],{"class":65,"line":115},[63,21937,21938],{"class":2976},"  stage",[63,21940,227],{"class":91},[63,21942,21943],{"class":145},"test\n",[63,21945,21946,21949],{"class":65,"line":121},[63,21947,21948],{"class":2976},"  tags",[63,21950,10453],{"class":91},[63,21952,21953,21956],{"class":65,"line":152},[63,21954,21955],{"class":91},"    - ",[63,21957,21958],{"class":145},"docker\n",[63,21960,21961,21964,21966],{"class":65,"line":253},[63,21962,21963],{"class":2976},"  image",[63,21965,227],{"class":91},[63,21967,21968],{"class":145},"mcr.microsoft.com\u002Fdotnet\u002Fsdk:8.0\n",[63,21970,21971,21974],{"class":65,"line":277},[63,21972,21973],{"class":2976},"  variables",[63,21975,10453],{"class":91},[63,21977,21978,21981,21983],{"class":65,"line":295},[63,21979,21980],{"class":2976},"    SECURE_FILES_DOWNLOAD_PATH",[63,21982,227],{"class":91},[63,21984,21985],{"class":145},"'.\u002FMyProject\u002F'\n",[63,21987,21988,21991],{"class":65,"line":301},[63,21989,21990],{"class":2976},"  script",[63,21992,10453],{"class":91},[63,21994,21995,21997],{"class":65,"line":313},[63,21996,21955],{"class":91},[63,21998,21999],{"class":145},"curl --silent \"https:\u002F\u002Fci.example.com\u002Fsecure_files\u002Finstaller\" | bash\n",[63,22001,22002,22004],{"class":65,"line":318},[63,22003,21955],{"class":91},[63,22005,22006],{"class":145},"dotnet test MyProject.IntegrationTests\n",[11,22008,22010],{"id":22009},"what-is-the-difference-between-configmap-and-secret","What is the difference between ConfigMap and Secret",[16,22012,22013,22014],{},"The difference is mainly the intention. You can check out the answer from the creator of them here: ",[20,22015,22016],{"href":22016,"rel":22017},"https:\u002F\u002Fstackoverflow.com\u002Fa\u002F36925553",[52],[2563,22019,22020],{},"html pre.shiki code .sJa8x, html code.shiki .sJa8x{--shiki-default:#E45649;--shiki-dark:#E06C75}html pre.shiki code .s5ixo, html code.shiki .s5ixo{--shiki-default:#383A42;--shiki-dark:#ABB2BF}html pre.shiki code .sDhpE, html code.shiki .sDhpE{--shiki-default:#50A14F;--shiki-dark:#98C379}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sLKXg, html code.shiki .sLKXg{--shiki-default:#A626A4;--shiki-dark:#C678DD}html pre.shiki code .sz0mV, html code.shiki .sz0mV{--shiki-default:#383A42;--shiki-dark:#E06C75}html pre.shiki code .sknuh, html code.shiki .sknuh{--shiki-default:#383A42;--shiki-dark:#56B6C2}html pre.shiki code .s7GmK, html code.shiki .s7GmK{--shiki-default:#383A42;--shiki-dark:#E5C07B}html pre.shiki code .sAdtL, html code.shiki .sAdtL{--shiki-default:#4078F2;--shiki-dark:#61AFEF}html pre.shiki code .sAGMh, html code.shiki .sAGMh{--shiki-default:#986801;--shiki-dark:#D19A66}html pre.shiki code .sMj0N, html code.shiki .sMj0N{--shiki-default:#50A14F;--shiki-dark:#ABB2BF}",{"title":59,"searchDepth":115,"depth":115,"links":22022},[22023,22024,22025,22026,22027,22028],{"id":21147,"depth":115,"text":21148},{"id":21466,"depth":115,"text":21467},{"id":21668,"depth":115,"text":21669},{"id":21869,"depth":115,"text":21870},{"id":21902,"depth":115,"text":21903},{"id":22009,"depth":115,"text":22010},"2024-11-11","Ways to keep sensitive information out of source code.",{},"\u002Fblog\u002Fhiding-sensitive-information-source-code",{"title":21139,"description":22030},"blog\u002Fhiding-sensitive-information-source-code",[2594],"POi45ZA5RhZw6HtuDN26pxPaN3MUqZH7SIVCvJ1v7bw",{"id":22038,"title":22039,"body":22040,"book":22071,"date":22075,"description":22076,"extension":2588,"meta":22077,"navigation":587,"path":22078,"seo":22079,"stem":22080,"tags":22081,"__hash__":22083},"blog\u002Fblog\u002Fhard-boiled-wonderland.md","Hard-Boiled Wonderland and the End of the World",{"type":8,"value":22041,"toc":22069},[22042,22045,22048,22051,22054,22057,22060,22063,22066],[16,22043,22044],{},"I finished this in eight hours of flying and transferring. It's now one of my favorite science fiction\u002Fsurrealist books and probably the best Murakami I've read, though that's out of only six or seven so far.",[16,22046,22047],{},"This book actually has an ending, unlike most of Murakami's work. The ending is very frustrating, but it's a logical one, and that actually makes the book memorable to me.",[16,22049,22050],{},"The book has a huge plot twist I won't spoil. It's tied to the title, and the title only makes sense once the twist lands. The marker for when it lands is the third consciousness chip thingy, which by itself isn't really a spoiler.",[16,22052,22053],{},"Most Murakami books feel like an abstraction of a big grey space of lost love and loneliness, full of lonely men and strange women, with nothing solid to grip on. I might be wrong since I haven't read everything, but that's the feel of the ones I've picked up. This one has the same atmosphere, but it has a spine in a way that his other books don't. The whole lecture about brain-wave shuffling and using frozen subconsciousness as an encryption key is a leap and a half, but it's a coherent leap inside the world the book builds. The Wall reads like a Lenin-style perfect world. The shadow is what's left over when you can think, which is also what makes you imperfect. The woods are where you end up if your shadow doesn't fully die during the ceremony. Nobody explains any of this. You sort of pick it up.",[16,22055,22056],{},"At the end of the book, the narrator has a way out and he doesn't take it. He has a list of reasons: the librarian, the instruments, \"I'm stubborn\", finding something worth doing in whatever's left. None of them are clean. He's coping, and the shadow basically tells him so.",[16,22058,22059],{},"The inner world is his, though. He built it from his own subconsciousness, except not entirely, because the scientist edited it, mapped it, rearranged it, and re-uploaded it. So the narrator is trapped in a version of himself that someone else has been messing with. That makes the choice to stay even weirder. It's still changing because he's still living in it. The shadow keeps telling him he can leave, but he's the one who made the place and he stays.",[16,22061,22062],{},"Most of the End of the World half is in dim light or pitch darkness, which really adds to the atmosphere and vibe of what was going on.",[16,22064,22065],{},"Another thing: this was published in 1985. Murakami is writing about brain-wave snapshots, encryption keys derived from your own subconsciousness, and black-box computation before computers were really a household item. Much of the explanation in the later half of the book resonated with me as a computer science graduate.",[16,22067,22068],{},"Anyway, it was a good book.",{"title":59,"searchDepth":115,"depth":115,"links":22070},[],{"title":22039,"author":22072,"year":22073,"cover":22074},"Haruki Murakami",1985,"\u002Fblog\u002Fhard-boiled-wonderland\u002Fcover.jpg","2024-05-18","Short notes on Murakami's 1985 novel, the one I think is his best.",{},"\u002Fblog\u002Fhard-boiled-wonderland",{"title":22039,"description":22076},"blog\u002Fhard-boiled-wonderland",[22082],"reading","Pmxyl92U_EbNh8EsK1anadJz_FvqwxFw2ofx0ya6KFo",1778998257112]