Tuesday, 21 July 2020

How do we control web page caching, across all browsers?

How do we control web page caching, across all browsers?


Introduction

The correct minimum set of headers that works across all mentioned clients (and proxies):

Cache-Control: no-cache, no-store, must-revalidate
Pragma: no-cache
Expires: 0

The Cache-Control is per the HTTP 1.1 spec for clients and proxies (and implicitly required by some clients next to Expires). The Pragma is per the HTTP 1.0 spec for prehistoric clients. The Expires is per the HTTP 1.0 and 1.1 specs for clients and proxies. In HTTP 1.1, the Cache-Control takes precedence over Expires, so it's after all for HTTP 1.0 proxies only.

If you don't care about IE6 and its broken caching when serving pages over HTTPS with only no-store, then you could omit Cache-Control: no-cache.

Cache-Control: no-store, must-revalidate
Pragma: no-cache
Expires: 0

If you don't care about IE6 nor HTTP 1.0 clients (HTTP 1.1 was introduced 1997), then you could omit Pragma.

Cache-Control: no-store, must-revalidate
Expires: 0

If you don't care about HTTP 1.0 proxies either, then you could omit Expires.

Cache-Control: no-store, must-revalidate

On the other hand, if the server auto-includes a valid Date header, then you could theoretically omit Cache-Control too and rely on Expires only.

Date: Wed, 24 Aug 2016 18:32:02 GMT
Expires: 0

But that may fail if e.g. the end-user manipulates the operating system date and the client software is relying on it.

Other Cache-Control parameters such as max-age are irrelevant if the abovementioned Cache-Control parameters are specified. The Last-Modified header as included in most other answers here is only interesting if you actually want to cache the request, so you don't need to specify it at all.

How to set it?

Using PHP:

header("Cache-Control: no-cache, no-store, must-revalidate"); // HTTP 1.1.
header("Pragma: no-cache"); // HTTP 1.0.
header("Expires: 0"); // Proxies.

Using Java Servlet, or Node.js:

response.setHeader("Cache-Control", "no-cache, no-store, must-revalidate"); // HTTP 1.1.
response.setHeader("Pragma", "no-cache"); // HTTP 1.0.
response.setHeader("Expires", "0"); // Proxies.

Using ASP.NET-MVC

Response.Cache.SetCacheability(HttpCacheability.NoCache);  // HTTP 1.1.
Response.Cache.AppendCacheExtension("no-store, must-revalidate");
Response.AppendHeader("Pragma", "no-cache"); // HTTP 1.0.
Response.AppendHeader("Expires", "0"); // Proxies.

Using ASP.NET Web API:

// `response` is an instance of System.Net.Http.HttpResponseMessage
response.Headers.CacheControl = new CacheControlHeaderValue
{
    NoCache = true,
    NoStore = true,
    MustRevalidate = true
};
response.Headers.Pragma.ParseAdd("no-cache");
// We can't use `response.Content.Headers.Expires` directly
// since it allows only `DateTimeOffset?` values.
response.Content?.Headers.TryAddWithoutValidation("Expires", 0.ToString()); 

Using ASP.NET:

Response.AppendHeader("Cache-Control", "no-cache, no-store, must-revalidate"); // HTTP 1.1.
Response.AppendHeader("Pragma", "no-cache"); // HTTP 1.0.
Response.AppendHeader("Expires", "0"); // Proxies.

Using ASP.NET Core v3

// using Microsoft.Net.Http.Headers
Response.Headers[HeaderNames.CacheControl] = "no-cache, no-store, must-revalidate";
Response.Headers[HeaderNames.Expires] = "0";
Response.Headers[HeaderNames.Pragma] = "no-cache";

Using ASP:

Response.addHeader "Cache-Control", "no-cache, no-store, must-revalidate" ' HTTP 1.1.
Response.addHeader "Pragma", "no-cache" ' HTTP 1.0.
Response.addHeader "Expires", "0" ' Proxies.

Using Ruby on Rails, or Python/Flask:

headers["Cache-Control"] = "no-cache, no-store, must-revalidate" # HTTP 1.1.
headers["Pragma"] = "no-cache" # HTTP 1.0.
headers["Expires"] = "0" # Proxies.

Using Python/Django:

response["Cache-Control"] = "no-cache, no-store, must-revalidate" # HTTP 1.1.
response["Pragma"] = "no-cache" # HTTP 1.0.
response["Expires"] = "0" # Proxies.

Using Python/Pyramid:

request.response.headerlist.extend(
    (
        ('Cache-Control', 'no-cache, no-store, must-revalidate'),
        ('Pragma', 'no-cache'),
        ('Expires', '0')
    )
)

Using Go:

responseWriter.Header().Set("Cache-Control", "no-cache, no-store, must-revalidate") // HTTP 1.1.
responseWriter.Header().Set("Pragma", "no-cache") // HTTP 1.0.
responseWriter.Header().Set("Expires", "0") // Proxies.

Using Apache .htaccess file:

<IfModule mod_headers.c>
    Header set Cache-Control "no-cache, no-store, must-revalidate"
    Header set Pragma "no-cache"
    Header set Expires 0
</IfModule>

Using HTML4:

<meta http-equiv="Cache-Control" content="no-cache, no-store, must-revalidate">
<meta http-equiv="Pragma" content="no-cache">
<meta http-equiv="Expires" content="0">

HTML meta tags vs HTTP response headers

Important to know is that when an HTML page is served over an HTTP connection, and a header is present in both the HTTP response headers and the HTML <meta http-equiv> tags, then the one specified in the HTTP response header will get precedence over the HTML meta tag. The HTML meta tag will only be used when the page is viewed from a local disk file system via a file:// URL. See also W3 HTML spec chapter 5.2.2. Take care with this when you don't specify them programmatically because the webserver can namely include some default values.

Generally, you'd better just not specify the HTML meta tags to avoid confusion by starters and rely on hard HTTP response headers. Moreover, specifically those <meta http-equiv> tags are invalid in HTML5. Only the http-equiv values listed in HTML5 specification are allowed.

Verifying the actual HTTP response headers

To verify the one and other, you can see/debug them in HTTP traffic monitor of webbrowser's developer toolset. You can get there by pressing F12 in Chrome/Firefox23+/IE9+, and then opening the "Network" or "Net" tab panel, and then clicking the HTTP request of interest to uncover all detail about the HTTP request and response. The below screenshot is from Chrome:

Chrome developer toolset HTTP traffic monitor showing HTTP response headers on stackoverflow.com

Setting Secure and HTTPOnly Flag for Session Generated Cookie in Classic ASP Website Running on IIS 6.0

The ASP Session Cookie can not be modified by Classic ASP code, so for IIS 6 you would need to have ISAPI module rewrite the cookies.

Setting HTTPONLY for Classic Asp Session Cookie

http://msdn.microsoft.com/en-us/library/ms972826

Client side JavaScript workaround

http://ko-lwin.blogspot.com/2010/12/how-to-secure-classic-asp-session-id.html


ASP Classic Set Cookie HTTPOnly Secure with Code or Web.Config

ASP Classic Set Cookie HTTPOnly Secure with Code or Web.Config


As promised, here is my Web.Config code for this subject. 

I want to remind you that I have this code but I use ASP Classic to Set, Read, Update. 
This will help you understand what needs to be done and your ASP Classic Options that I do have running on live sites. 

<outboundRules>
<rule name="Add Strict-Transport-Security when HTTPS" enabled="true">
<match serverVariable="RESPONSE_Strict_Transport_Security" pattern=".*" />
<conditions>
<add input="{HTTPS}" pattern="on" ignoreCase="true" />
</conditions>
<action type="Rewrite" value="max-age=31536000" />
</rule>

<rule name="Add Secure">
<match serverVariable="RESPONSE_Set_Cookie" pattern=".*" />
<conditions>
<add input="{R:0}" pattern="; Secure" negate="true" />
</conditions>
<action type="Rewrite" value="{R:0}; Secure" />
</rule>

<rule name="Add HttpOnly">
<match serverVariable="RESPONSE_Set_Cookie" pattern=".*" />
<conditions>
<add input="{R:0}" pattern="; HttpOnly" negate="true" />
</conditions>
<action type="Rewrite" value="{R:0}; HttpOnly" />
</rule>

</outboundRules>

Now that you have my Outbound ReWrite rules let's jump to the actual code that will be doing the work. 

First, you'll need to test, so you'll need to read these HTTPOnly Header cookies., You would think you could just write some simple, 

Response.Cookies("DATA")("KEY") 

Which made it really easy reading the cookie because all you needed to do is Request.Cookie("KEY") and that would return your data. 

Life was good, easy, response. = write, request. = read. All was good in the ASP Classic world. 

Until this HTTP_COOKIE thing which is a Server Variable. Sounds good so far, could even be something of a security upgrade due to the fact it's really a server variable and connected to your browses ability to read server header information. 

Request.ServerVariables("HTTP_COOKIE")

You can Google what a HTTP_Cookie looks like. I would like for you to start your reading here. 

https://www.owasp.org/index.php/HttpOnly

That will give you everything you need to know to this point. 

I'll be using two of my sites to show the different HTTPOnly methods, Non Secured and Secured. I'll also start the process of how you set and read these cookies. 

http://tools.seobook.com/server-header-checker/?page=single&url=www.truckandtools.com&useragent=1&typeProtocol=11

SERVER RESPONSE: HTTP/1.1 200 OK
Cache-Control:private
Content-Length:51094
Content-Type:text/html; Charset=UTF-8
Expires:Fri, 18 Nov 2016 16:44:50 GMT
Server:Microsoft-IIS/7.5
Set-Cookie:ASPSESSIONIDCSDTRABR=BLAHBLAHBLAHBLAHBLAH; path=/; HttpOnly
Date:Fri, 18 Nov 2016 16:14:51 GMT

The above was added using this code. 

Response.AddHeader "Set-Cookie", strCookieName & "=" & strCookieKey & "=" & strCookieValue & "; expires=" & strGMTDateRFC22 & "; domain="& strCookieDomain &"; path=/; HTTPOnly"

 

Next example is a HTTPOnly Secure header. 

http://tools.seobook.com/server-header-checker/?page=single&url=www.mnworks.net&useragent=1&typeProtocol=11

SERVER RESPONSE: HTTP/1.1 200 OK
Cache-Control:private
Content-Length:47530
Content-Type:text/html; Charset=UTF-8
Expires:Fri, 18 Nov 2016 16:53:40 GMT
Server:Microsoft-IIS/7.5
Set-Cookie:ASPSESSIONIDQCARTBAT=BLAHBLAHBLAHBLAHBLAH; path=/; Secure; HttpOnly
Date:Fri, 18 Nov 2016 16:23:41 GMT

The above had a few extra items but I removed them to keep focus on the HTTP Cookies. 

You see the only difference is Secure: was added. 

The code looks like this: 

Response.AddHeader "Set-Cookie", strCookieName & "=" & strCookieKey & "=" & strCookieValue & "; expires=" & strGMTDateRFC22 & "; domain="& strCookieDomain &"; path=/; secure; HTTPOnly"

Now that you know how to set the cookies let's read them so we can actually use them. 
First of all, I don't set cookies that are not used to benefit the visitor and member. So nothing is added until the visitor actually logs in. To show examples I'm setting a page that will add cookies without you having to login. 

If you are using Google Chrome you could use "EditThisCookie" extension.
Look for the HTTPOnly cookies that are checked and the ones with the domain that matches my domain. I use Google products so you'll see other cookies not associated with my httponly cookies. 

So let's set the cookie without using the web.config method so we stay true to form with ASP Classic. 

The URL is to my Truck and Tools site which is a membership site running Classic ASP. 
For Secured tests I'll be using my Job Board site which also runs the same code. 

When a visitor first reaches your webiste: 

Using Chrome's Cookie reader you're going to see truckat which is my identifier for my Truck and Tools httponly cookie. 
The ls= is Location State and most likely it's showing US. 
I don't use this for anything other than testing if the cookie can be added. You can change your state and it's updated on the session but not the cookie until you login. 

Now, the issue I have had was reading the damn things. HTTP Cookies are a continious string which uses the key name of the cookie to split the array of data you want to store. 
This makes using a simple loop impossible. But if you have found a loop that works and don't mind sharing drop me a line. For now, here's what I use. 

Going back to the code test page you'll find a full HTTP header cookie in line two of the example page. 


I setup a page on my other site to show you the acutal functional code and with the Chrome Cookie Editor you should be able to see it live.

I'll paste the code here that makes the page work but you should check the page to see the break down of each part of the code.

Cookie-Set: HTTPOnly Live Demo and Example of my work. 

HTTPOnly Set-Cookie ASP Classic

Here I'll share with you my method of using HTTP cookies. This site uses HTTPOnly cookies so let's add a TESTCOOKIE to the mix for you to see how I do things.

Keys and Values I will be working with.
TESTCookie=Key1=Value1Key2=Value2Key3=Value3Key4=Value4ASP=CodeDev=Murray

ASP CLASSIC CODE:
Response.AddHeader "Set-Cookie", "TESTCookie=Key1=Value1Key2=Value2Key3=Value3Key4=Value4ASP=CodeDev=Murray; expires=Sat, 19 Nov 2016 19:41:11 GMT; domain=truckandtools.com; path=/; HTTPOnly"


HTTPOnly: Request your HTTP Cookie

We use the Request.ServerVariables("HTTP_COOKIE") in this example.

It looks like this: ASPSESSIONIDCSDTRABR=FIKHLKGDIPEKKGBPMHKFJAHN; truckat=ls=US; TESTCookie=Key1=Key1 is King of the Hill KeyKey2=Value2Key3=Value3Key4=Value4ASP=CodeDev=Murray; _ga=GA1.3.1978168732.1464795103;


HTTPOnly Set-Cookie Cookie Name

We now need to extract our cookie from the full HTTP Cookie: It will look like this:
TESTCookie=Key1=Value1Key2=Value2Key3=Value3Key4=Value4ASP=CodeDev=Murray
The ASP CLassic code to do this part.
strCookie0 = Request.ServerVariables("HTTP_COOKIE")
If InStr(strCookie0,"TESTCookie=") Then
chttpLen = Len("TESTCookie=")
j = InStrRev(strCookie0, "TESTCookie=")
if j > 0 Then
strCookieTEMP = Mid(strCookie0, j+chttpLen)
end if
j = InStr(strCookieTEMP, ";")
if j > 0 Then
strCookieTEMP = Left(strCookieTEMP, j-1)
End If
strCookie0 = strCookieTEMP
End If

HTTPOnly Set-Cookie Your Key and Values

Key1=Value1Key2=Value2Key3=Value3Key4=Value4ASP=CodeDev=Murray

Now we need to extract the Keys and Values from our HTTP Cookie. I like to use a simple function call. Really can't loop because of how the cookie is saved. We really need to check what's in the string.

Next up is how creative you want to be.

Known Value Length:
If InStr(strCookie0,"key1=") Then
j = InStrRev(strCookie0, "key1=")
if j > 0 Then
strCookieTEMP = Mid(strCookie0, j+5)
end if
j = 9
if j > 0 Then
strCookieTEMP = Left(strCookieTEMP, j-0)
End If
strCookie1 = strCookieTEMP
End If

Code with a known delimiter / tag. If InStr(strCookie0,"key1=") Then
j = InStrRev(strCookie0, "key1=")
if j > 0 Then
strCookieTEMP = Mid(strCookie0, j+5)
end if
j = InStr(strCookieTEMP, "key2=")
if j > 0 Then
strCookieTEMP = Left(strCookieTEMP, j-5)
End If
strCookie4 = strCookieTEMP
End If

You should be able to see the pattern of extracting your values from the HTTP Cookie.


HTTPOnly Set-Cookie Update Values and even add Keys

Let's add new values and update current ones as if we are really doing some work here.

Let's get all of our Values from our current keys. I wont cheat and just type them in. I will run the code as in the example below.

Let's put it all together now.

If InStr(strCookie0,"TESTCookie=") Then
chttpLen = Len("TESTCookie=")
j = InStrRev(strCookie0, "TESTCookie=")
if j > 0 Then
strCookieTEMP = Mid(strCookie0, j+chttpLen)
end if
j = InStr(strCookieTEMP, ";")
if j > 0 Then
strCookieTEMP = Left(strCookieTEMP, j-1)
End If
strCookie0 = strCookieTEMP
End If

Returns
Key1=Value1Key2=Value2Key3=Value3Key4=Value4ASP=CodeDev=Murray

Next we extract our keys and values. I'll show you the first key then update it and delete it.

strCookie0 = Request.ServerVariables("HTTP_COOKIE")
If InStr(strCookie0,"TESTCookie=") Then
chttpLen = Len("TESTCookie=")
j = InStrRev(strCookie0, "TESTCookie=")
if j > 0 Then
strCookieTEMP = Mid(strCookie0, j+chttpLen)
end if
j = InStr(strCookieTEMP, ";")
if j > 0 Then
strCookieTEMP = Left(strCookieTEMP, j-1)
End If
strCookie0 = strCookieTEMP
End If
If InStr(strCookie0,"Key1=") Then
j = InStrRev(strCookie0, "Key1=")
if j > 0 Then
strCookieTEMP = Mid(strCookie0, j+5)
end if
j = InStr(strCookieTEMP, "Key2=")
if j > 0 Then
strCookieTEMP = Left(strCookieTEMP, j-1)
End If
strCookie4 = strCookieTEMP
End If

Returns
Value1


HTTPOnly Set-Cookie Update Value Key1

Yes, key1 is different from Key1, please keep up.
Now, let's change the value of Key1. to Key1 is King of the Hill Key.

What would happen if we just updated Key1 without touching the other keys.

strGMTDateRFC22 = CookieServerUTC("d",Now(),6,"GMT")
Response.AddHeader "Set-Cookie", "TESTCookie=Key1=Key1 is King of the Hill KeyKey2=Value2Key3=Value3Key4=Value4ASP=CodeDev=Murray; expires="&strGMTDateRFC22&"; domain="&strHostByName&"; path=/; HTTPOnly"
strCookie0 = Request.ServerVariables("HTTP_COOKIE")
If InStr(strCookie0,"TESTCookie=") Then
chttpLen = Len("TESTCookie=")
j = InStrRev(strCookie0, "TESTCookie=")
if j > 0 Then
strCookieTEMP = Mid(strCookie0, j+chttpLen)
end if
j = InStr(strCookieTEMP, ";")
if j > 0 Then
strCookieTEMP = Left(strCookieTEMP, j-1)
End If
strCookie0 = strCookieTEMP
End If
If InStr(strCookie0,"Key1=") Then
j = InStrRev(strCookie0, "Key1=")
if j > 0 Then
strCookieTEMP = Mid(strCookie0, j+5)
end if
j = InStr(strCookieTEMP, "Key2=")
if j > 0 Then
strCookieTEMP = Left(strCookieTEMP, j-1)
End If
strCookie4 = strCookieTEMP
End If

Returns
Key1 is King of the Hill Key

Do we have any values and keys after our update otther than the key we updated? Let's look....

strCookie0 = Request.ServerVariables("HTTP_COOKIE")
If InStr(strCookie0,"TESTCookie=") Then
chttpLen = Len("TESTCookie=")
j = InStrRev(strCookie0, "TESTCookie=")
if j > 0 Then
strCookieTEMP = Mid(strCookie0, j+chttpLen)
end if
j = InStr(strCookieTEMP, ";")
if j > 0 Then
strCookieTEMP = Left(strCookieTEMP, j-1)
End If
strCookie0 = strCookieTEMP
End If

Returns
Key1=Key1 is King of the Hill KeyKey2=Value2Key3=Value3Key4=Value4ASP=CodeDev=Murray>

Can we add more keys with values? Yes,


So there you have it, your HTTPOnly Cookies all setup and ready to be used in your membership and eCommerce site. If you need help, ask me.