HTTP headers for Clickers
You will need MooSock or some Sock/raw sockets extension for this article. I'll assume you're using MooSock for now.

This article will briefly cover the basics of the HTTP specification, so Clickers can use MooSock to download files and so on.

HTTP servers
As you probably know, most pages on the internet are hosted by servers, such as the clickteam.com server. To request data from a server you have to send a HTTP request on port 80. To download a file with MooSock, first you need to connect to a server like so:

- Connect to www.clickteam.com on port 80

Do not include the http:// part - only web browsers use that. The www. bit is optional really, as you can see by visiting http://clickteam.com in your browser (you get the same page).

Hint: If you have a string such as http://www.myserver.com/folder/file.ext, you can parse out the server by using String Parser 2. Add a delimiter of /, and listGetAt$( String Parser, 2 ) will get the second token which is the server, returning www.myserver.com.

Sending an HTTP request
Just a bit of terminology: you send an HTTP request to a server to ask for a file, then the server sends an HTTP response with information and the data you asked for.
Let's start with a nice easy GET request. A very simple request looks like this:

GET /mypage.html HTTP/1.1
Host: www.myserver.com



It's hard to show, but you need two line breaks at the end of that to mark the end of the request.
Let's examine it.

GET /mypage.html HTTP/1.1
This basically means Send me mypage.html in the root directory, the same as www.myserver.com/mypage.html.
It also specifies that it's using the HTTP protocol version 1.1. You can use 1.0, but almost all servers today are HTTP 1.1, and HTTP 1.1 offers a lot more cool stuff like the ability to retrieve only part of a file (which means resumable downloads).

If you GET / HTTP/1.1, it's the same as going to www.myserver.com in your browser - you will be sent the default page for that server.

Note: For our purposes, all files are relative to the server you are currently connected to. You can't say something like GET www.a-different-server.com/file.html.

Host: www.myserver.com
After the GET line follows a series of HTTP headers, which provide additional information about the request. These are a name, followed by a colon, then a value, in this case Host: www.myserver.com.

The Host header is required in HTTP 1.1. You must send it, otherwise the server will say Bad request in the response and not give you any of your precious data.
The Host field is used to say which server holds the file you want.
As far as a clicker is concerned, just paste the same string you connect to in the Host field and things will work.

Receiving data back
MooSock has a Send Text Line feature which makes it easy to send HTTP requests. It automatically adds a line break at the end of the string you enter. So to send the request:
- Send text line GET /mypage.html HTTP/1.1
- Send text line Host: www.myserver.com
- Send text line
Don't forget to send a text line of at the end, as that adds a second line break to end the request.
This is the same as:
- Send text GET /mypage.html HTTP/1.1 + Newline$ + Host: www.myserver.com + Newline$ + Newline$
It looks neater and makes more sense to use the Send Text Line method though.

Receive your data with:
+ On Received
: Rich edit: Set text to Recv$( MooSock, 1024 )
Recv$ is the Receive Text expression.
Remember Rich Edit's Set Text action ADDS text, not sets the entire content of the rich edit object. So MooSock will download 1024 bytes (1kb) of data at a time, each time adding the text to the Rich Edit object.

Hint: It's handy to debug like this. If something isn't working, such as you sent an invalid request, you'll get an error message back which you can see in the Rich Edit object.

Note the server also sends back a status line (such as 200 OK) and some other headers, followed by two linebreaks, then the data. For Clickteam.com I get back:


HTTP/1.1 200 OK
Date: Mon, 03 Jan 2005 21:57:44 GMT
Server: Apache/2.0.51 (Gentoo/Linux) mod_ssl/2.0.51 OpenSSL/0.9.7d PHP/4.3.8
X-Powered-By: PHP/4.3.8
Transfer-Encoding: chunked
Content-Type: text/html; charset=ISO-8859-1

<html>
<head>
<title>Clickteam -- Creative Tools</title>
...etc


This tells us the version, the status code (200, which means OK), the date, the server software and version (apache), a non-standard header - X-Powered-By - saying it uses PHP, chunked transfer encoding (see the comments about chunks), and the MIME type and character set of the content. Data follows seperated by the usual double linebreak. To get the page alone, you'll need to use String Parser 2 to parse out the data part. If it's a binary file like an EXE file, you're probably better off using the Binary object to slice off the headers whilst maintaining all those nulls and things.

Server responses aren't usually useful to a Clicker other than to inspect the status code (you'll probably know the dreaded code 404 means Not Found).
More about status codes: http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

Other request headers you can use
The full HTTP specification is available here: http://www.w3.org/Protocols/rfc2616/rfc2616.html
The page which fully defines each header is here: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html (this includes response headers)
Anyways, in brief, here are some interesting or handy headers:

From: This used to be required in HTTP 1.0, but not any more: the internet used to be a trusted network of universities and nice people. From was used to send your email address in the request. As you can imagine in today's internet that's NOT useful. It's not required in HTTP 1.1, but it was mentioned since it was required in HTTP 1.0.

User-Agent: This also used to be required in HTTP 1.0, but is optional for HTTP 1.1. This header is simply the program you're using to make the request. Modern browsers have complicated strings incooperating operating system, engine, compatibilities and things, but you can just use User-Agent: MyBrilliantApp/1.0 or whatever. Most web stats scripts show user agent strings plus the number of requests they make so this might be useful for tracking statistics and usage.

Range: Ask for just part of a file. Range: bytes=100-199 will retrieve bytes 100 to 199 inclusive. If you successfully download the first 10mb of a big file, you can request the same file again but add this header:
Range: bytes=10485760-
The server will skip the first 10mb of the file and send the rest of the file (if you dont specify a second number it sends until the end). Thus you can make download resumers with this header. (One of the cool features of HTTP 1.1)
Another piece of terminology, it's called a 'Partial Get' if you make a request with a Range header.

Referer: [sic] If you clicked a link on a different page, the referer field (NOT referrer, referer is a spelling mistake which somehow was preserved in the specification) should hold that page you came from. If you directly enter an address though, obviously you have no referer. Useful for stats tracking and stuff, plus you could script your server to log the referer in a database on a 404 error, so you can automatically find all your broken links!

Cache: To reduce bandwidth usage and speed things up, if you request a page, it gets saved, and subsequent requests return the saved version without accessing the net. This can be annoying for clickers, send the header Cache-control: no-cache to skip caching and always access the internet to get the latest file.

Posting data
If you use PHP or ASP scripts, you can post data with MooSock to provide data to a script automatically. To do this, just change GET to POST and after the two line breaks have your data, in the same format it appears in query strings, e.g.:
name=Bob&age=20&email=bob@bob.com
Unfortunately I haven't tested this, you may also need to send Content-Length or Content-Type headers. Ask google.

The end
Hopefully you find this useful
A very simple example that downloads the default Clickteam.com page: http://www.gullen.pwp.blueyonder.co.uk/files/httpmoosock.cca
Notice this just retrieves a page which is designed to redirect you to a different page based on your language. MooSock is very low level and just sends data around - browsers are the programs that actually implement things like going to a different page when told to redirect.