• Howdy! Welcome to our community of more than 130.000 members devoted to web hosting. This is a great place to get special offers from web hosts and post your own requests or ads. To start posting sign up here. Cheers! /Peo, FreeWebSpace.net

$HTTP_REFERER and visits by Favorites/Home links

Cheap Bastard

New Member
Okay, i've got a simple counter script going. Just got the referral stats working, but i'm getting a very small amount of data since most visits are by links in the Favorites or by browsers that have it set as the homepage (that and i have low traffic, but i'm talking percentage-wise).

So how... what... or rather, where could i find whether or not a visit just has an "empty" $HTTP_REFERER or whether it was a visit by favorites or a visit by homepage?

Thanks
 

jm4n

New Member
It's difficult to trust the HTTP_REFERER, because this is set by the user agent (the browser). Some agents strip this information; some send a referer even when you click the back button; others are just plain broken. Even worse, there are proxies that strip this header out of the request all together...

I don't think browsers send anything specific for bookmarks/favorites, but I'm not sure. In any case, I wouldn't take the HTTP_REFERER to be anything definitive, in any situation, since it's basically user input. Never trust user input.
 

Cheap Bastard

New Member
you know of something more reliable?

I read some other posts, but $HTTP_X_FORWARDED_FOR or whatever it was again doesn't work.

I'm not looking for something perfect... just looking for the most precise (in most cases anyway) variable...
 

jm4n

New Member
I think once you look at the whole picture you'll understand why there really isn't anything reliable for this information.

First, the only information the server has is this:

- Remote IP address
- Web Browser and Version *
- What page they're looking for
- Where the user came from *

The two * entries are sent by the browser, and can't always be trusted. I can spoof those to quite easily using 'wget', or any Perl script using LWP::UserAgent, etc. And of course the UserAgent can be hacked in the Windows registry for IE (mine has the string "37331 h4x0r" appended to it ;).

Back to the referer string though. The browser sends you the URL of the last page it was on. Now some browsers will only send this information if there was an actual link on the page.

Some versions of IE, though, will send an HTTP_REFERER when you hit the 'back' button. This had me confused a while back until I dug through the logs -- I thought Microsoft was linking to me, but it was just a user hitting the back button.

Some browsers also send a referer of the last page you were on if you click one of your bookmarks. For example, let's say my "home page" in IE was Yahoo.com. I open a fresh browser window, and click my link to your website. You will see "yahoo.com" as a referer, even though it really had nothing to do with how I got to your site.

The worst part is, with IE, this behavior seems to change from version to version and platform to platform... I don't know about Netscape, but I've seen it send "[Bookmark]" as the referer if the user clicked a bookmark...

---
Now, since the only referer information the web server has is that one little string, there really is no other way to determine how the user got to your site.

Even the Remote IP address can be spoofed easily under NT or any *nix. Plus, it could be that of a proxy, cache, anonymiser, etc. BTW, that X_FORWARDED_FOR has nothing to do with a referer, it's something sent by some proxies.

End result (and I believe I said this in the X_FORWARDED_FOR thread as well): never trust user input. If it came from outside your own control, it is user input, and it is not to be trusted :)

Sorry if this long rant doesn't help... mind if I ask what exactly you're trying to accomplish btw? I'm curious... if you're just looking for a count of people who get to your site via a bookmark, there's not much you can do about that...
 

Cheap Bastard

New Member
look, i know it's somewhat unreliable... And yes, you mess up those variables. Fact is, 90% of the web uses IE. About 80% uses 5.0,5.01 or 5.5. And i bet about 90% doesn't mess up any of those variables. Heck, most of them can't uninstall software...

My point is, however advanced you may think web users are, you're wrong. There's a small percentage that is; almost all visitors here are. There's a whole new world out there :)

btw, i'm trying to write my own tracker... i noticed how stats4all (on my previous site) had bookmarks as the #1 cause of traffic, i wanna figure out how to do it too. Right now, about 20-30% of my traffic has a $HTTP_REFERER... So i wanna know about how many of the other 70-80% is an empty $HTTP_REFERER and how many of those are bookmark visits.
 

jm4n

New Member
By my previous post, I only meant that you couldn't use the HTTP_REFERER for anything important (like, for example, restricting image views to referers within your domain). That's why I asked toward the end what the intended purpose was. Yes, the HTTP_REFERER is fine for statistical tracking, and most stats packages use this.

But again, even though you're just doing this for statistical tracking, I don't think there's a standard on what browsers send for bookmarked links. I will however perform a quick test and see what IE does in this situation...

From what I can tell, it sends a blank referer (IE 5.5, Windows 2000 SP2). I just bookmarked my site, closed and re-opened a new browser window, and clicked the bookmark (or "Favorite")... Mozilla 0.93 does the same thing (blank referer). I don't have Netscape handy to test...

So I can't tell you how this site you reference is tracking this information. It might be worthwhile to email the author of whatever tracking software they're using, or the webmaster of the site in question, as I can't see anything in my testing that could identify a bookmark vs just typing in the URL. It's possible that they are just assuming a blank referer to be a bookmark, but there are many other situations that can cause an empty referer, including (but not limited to):

- Typing the URL manually
- Clicking a link in an Email (or any other program)
- Clicking a shortcut on one's desktop (similar to bookmark I guess)
- Drag and drop a link from any page to any browser window, including the current one (I just tested this)

My guess is that that's what they're doing, but again, if they know a more accurate method, I would definately ask them.
 

jm4n

New Member
That would be a link on someone's home page on their local drive. A lot of people create an HTML page on their hard drive, and put links on it to sites they frequent. This way, their home page loads quickly, and they have a collection of sites to choose from there, rather than having to dig through bookmarks. I used to do this, and had a Google search box, documentation links, and other stuff right there.
 
Top