newzbot: changelog

18 Aug 2005

• readnews.cgi was 500'ing because the CGI wasn't executable !@. fixed.

• webserver hard-drive crashed; reimaged machine. Down for ~20h.

• added experimental RSS feed of recently added servers
• fuzting w/ a stylesheet

11 Apr 2005

• messing with google ads

• ad banner & page header cleanup

17 Oct 2004

• scans were not running! re-enabled.

12..17 May 2004

• behavior of feedback page changed slightly; old message are at the bottom & new messages are at the top
• results of votes are not displayed until critical mass (currently 5 votes)
• created cute browser icon

26 Apr 2004

• changed look of the feedback page
• feedback can be attached to servers (presently only visible in the search page)
• feedback page can remember your nickname
• last N searches can be remembered
• created preferences page to control search & nickname memory

12 Apr 2004

• moved mysql to new machine; 30m downtime

2 Apr 2004

• group names from search.php weren't URL encoded.

8 Feb 2004

• Moved db files to a new disk; 30 minute outage.

2 Dec 2003

• Improved 'Post' verbage in serverlist & search and improved (?) FAQ entry for posting.
• Disabled display of 'OS' and 'Location' information in serverlist and search.

• Disabled collection of Location: information -- the data source (netgeo) is no longer being maintained.

• Disabled collection of operating system information.

15 Apr 2003

• Whoops. The scanner broke and started flagging servers as invalid. I fixed it & a new scan is running, but it will take a few days for the search engine to be re-populated with newsgroups.

12 Apr 2003

• display average article size w/ group search results.

12 Jan 2003

• PHPized remaining static pages
• PHPized exclude & suggest CGI's
• added active list link
• added cute (?) newzbot logos (I doodled them back in November & my girlfriend drew them :)
• wrote simple privacy statement

6 Jan 2003

• DSL is back, after WAY too long.

• DSL down for move; site is down

6 Oct 2002

• show server name when showing server rating

4 Sep 2002

• optimized search.php slightly; only do a full LIKE query if there are few dots in the search string

3 Sep 2002

• made some cosmetic & functional changes to the server rating. also starting to nuke off-topic comments from the feedback page -- mostly winmx stuff.

22 Aug 2002

• Added links to a DMCA Safe Harbor page. Apparently this is now necessary. chillingeffects.org is a good resource for this sort of thing.

4 Aug 2002

• added another "highly experimental" feature, server rating. Once sufficient votes have been collected, I'll allow you to sort the lists according to rating.

3 Aug 2002

more changes to the search engine:
• only show servers that have more than zero articles in the requested group. optionally display all servers w/ appropriate hints.
• added HIGH_PRIORITY to all SELECTs.. don't know if it will do any good. I need to move to a transactional table type (innodb or postgres..) to better handle DELETEs

23 June 2002

• Upgraded to PHP 4.2 at the urging of n/shh//bi :-)

2 June 2002

• search.php now allows rechecking / verification of servers. If you think the server has gone offline or private, you can have newzbot recheck it and, if it's truly offline, it will be removed.

4 May 2002

• Playing with fulltext indexing for for faster partial group searches ...

29 Apr 2002

• Removed average article count from search.php output.. the "Days" and "Last post" information should be much more informative.

• nntp-scan bot now uses netgeo to record the geographic location of the server.

• Had some free time this weekend so I enabled the search bot to pull down per-group article information via XOVER & record it in a table. Modified the search engine to display this information in the form of "days of news" (days of news archived per-group per-server) and "last post" (how long ago this server had a posting to the searched-for group.) This will make the search bot much slower, but will hopefully provide useful information.
• Updated the faq accordingly
• Added <title> tags to & fixed cosmetic stuff in search.php
• removed references to old search.html and nph-dbif.cgi page in various pages
• nph-dbif.cgi now redirects (HTTP 302) to search.php
• Added Connections: information to status.php

4 Mar 2002

• put a largely untested search.php into production :) The PHP version is lighter than the perl CGI, and there are some slight user interface changes that hopefully make it more useful.

• a few serverlist.php changes:
  • use SELECT .. LIMIT to lighten the DB/network load
  • reduce font size if showing a long item
  • display certain items using htmlentities() (doh)
  • shuffled the checkboxes & form entries around to reduce visual footprint. (still needs work, though.)
• bounced MySQL instance to enable slow-query logging
• Picked up on some old work to record information about the articles stored on a server (range of article dates per group & days of news, article sizes, etc.) This will be eventually implemented so you can see which servers get fresh news & how long they retain it. It's time-consuming to grab this information, which makes it a pain to test, but I think it will be useful.

• use new 'readergroups' table (instead of GDBM file) to determine which groups the search engine should show as readable. This table is updated regularly, so the search engine and the news reader should never be too out of sync.
• Working on converting nph-dbif.cgi to PHP .. hopefully this will take care of some of the high load conditions that have been persisting for a while. (The other alternative would be using something like mod_perl .. but since I've already started down the PHP path, I think I'll use mod_perl for my next project instead :)
• 118,891 unique IPs and 1,457,682 hits for January 2002. Just think, if I added 30 images/icons/general phl00ph to my site, I'd have more than a million hits per day !@# I'd need an entire marketing department, though, to help me decide what background color I should use.

• Added GigaNews.com ad banner

• Database server was down December 14th & 15th for hardware maintenence (added drives & 100mbit NIC to kadath) & some rack reorganzation.

• serverlist.php now allows sorting by article retention

• Donate to the American Red Cross.

  I fear the reaction of my government, fear waking up to learn that we've just turned the whole of Afganistan into smoked glass. I fear the slow coagulation of this into a holy war; the "good Christian nation" v. the "evil heathen Muslim towel-heads". Now is not a good time to be different-looking in America.

  I hope clearer heads will prevail. I hope for targeted justice, not blanket vengance.

• Upgraded to MySQL 3.23.41

• The 4gb SCSI disk that held the mysql databases started logging errors and some tables (none of them newzbot-related) started getting corrupted. Last night and into the AM, I added a pair of mirrored 9gb drives and moved the mysql databases to them.
• Finally added freeskylarov.org and anti-dmca.org banners. Writing code is not a crime!

• 78,194 unique IPs and 861,800 hits for July 2001.

• Adobe dropped their ridiculous complaint, but it doesn't matter; the guy is still in jail.

• Added Boycott Adobe banner

• "Sorted by longevity" in the index page and "Sorted by age" (same thing) in the page trailers didn't actually do what they were supposed to do. Fixed now.
  (I didn't start logging "date discovered" until the the balmy Bay Area summer of 1997..)

• 97,016 unique visitors and 1,100,833 hits for June 2001. (I moved from the ~200k static server listing pages to 40-servers-per-page dynamic server listings, so the hit count went up but the bytes served went down. Unfortunately, if the database is down, the server lists are unavailable, but I rarely need to shut MySQL down.)

• ISP had another unscheduled outage from ~3am to noon PDT. !@#

• ISP had a router problem; unscheduled DSL outage from 3:30pm to 7:30pm PDT

• added servers.xml for programs that want to back-end into newzbot.com.
• fixed (?) some ugly <table> bugs in serverlist.php
• fixed some goofed <body> tags.

• upgraded web server from 32mb to 128mb. Thanks to Jeff Williams for the donated RAM.

• added highly experimental server statistics page
• customlist.php -> serverlist.php redirector (d0h)
• no more static pages of server listings
• php-ized index page & rearranged a bit
• serverlist.php displays lock icon () for news servers that support NNTP over SSL (SNEWS)

• 94,082 unique visitors and 887,575 hits for May 2001.

• upgraded tillamook (WWW server) to Apache 1.3.14, PHP 4, and Linux kernel 2.2.19 (the reboot cost me my 215-day uptime... not bad for an old P100)
• added "plain HTML" output to customlist.php

• upgraded DB server to MySQL 3.23.37.

• 86,694 unique visitors & 783,908 hits for April 2001. Seeing as how things peaked with 112,626 unique visitors & 923,341 hits back in November 2000, I can only assume that when it's cold out, people flock to newzbot, but when spring comes, they all go outside. I should take a lesson from that..

• workaround for ORDER BY bottleneck in fuzzy search. mysql doesn't seem to like ORDER BY on large tables..

• hmm, getting 'table full' errors from nph-dbif.cgi; now doing SET SQL_BIG_TABLES = 1 for potential trouble queries. Later versions of mysql allegedly do this automatically..
• added OS output to customlist.php and nph-dbif.cgi. How netcrafty..

• support group names with '+' for this guy.
• nntp-scan now logs OS fingerprinting info (nmap -O). Eventually this info will be visible in the customlist page and the search engine.

• added POST testing / latency information through alt.test, misc.test, and microsoft.test. Verification & latency information is available through nph-dbif.cgi.
• customlist.php - limit output to n rows & added Previous & Next buttons.

• customlist.php - got rid of extra SELECT so the table loads faster.

• nntp-scan engine now uses new newzbot.pm perl module
• added more rigorus testing of servers (iterate over GROUPs until we find one with articles in it, then iterate over ARTICLEs until we can pull down one successfully. This weeds out servers with no articles as well as servers that allow connections and LIST, but balk at ARTICLE.)
• now supports testing servers via a foreign proxy, so servers that are public to my ISP (ideally) don't get flagged.
• started logging changes to host information in a 'hosthist' table. "Eventually" the hostinfo page will display the collected information.

• added 128mb RAM to 'kadath' (the MySQL server) for a total of 192mb RAM. wh00t.

1 April 2001

• 89,749 unique visitors and 868,642 hits for March 2001.

30 March 2001

• fixed bug in customlist.php (ordering by speed returned error)

29 March 2001

• Enabled customlist.php
• nph-dbif.cgi and readnews.cgi now use the newzbot.pm module
• customlist.php and feedback.php now use the newzbot.php library
• added new ad banners (alibis.com), mainly to give newzbot::ad and newzbot.php something to rotate through :-)

28 March 2001

• search page (nph-dbif.cgi) now displays time since a server was last checked.
• feedback page defaults to showing the last 30 days of messages
• nph-dbif.cgi: added another GUBA ad banner sans the nekkid chick

• DSL connection is back up. My ISP intentionally shut off my connectivity due to portscanning. After explaining myself, connectivity was re-established. However, this does cause me to re-evaluate the scanning methodology to ensure that only news servers are scanned, and not end users.

• DSL connection is down. Urp!

• nph-dbif.cgi: added a GUBA ad banner
• nph-dbif.cgi: allow space in search queries & use it as an AND delimiter for fuzzy search

• changed internal IPs & split up DNS into int, ext, and dmz domains.
• updated db server hostname for nph-dbif.cgi & feedback.php

• nph-dbif.cgi: workaround for hosts that are still linked but invalid
• nph-dbif.cgi: added verbage to explain SSL icon

• added new database flag type -3 (bogus news server; ie newsfeeds.com)
• nph-dbif.cgi: removed "no hostid for ..." message
• nph-dbif.cgi: re-enabled readnews.cgi

• started work on newzbot.pm perl module. modularization r00lz.

• 164,158 unique visitors and 915,995 hits for February 2001.

Older stuff

14 Nov 2000: Addressed a problem that was causing an (unintentional) denial-of-service in the searcher cgi (nph-dbif.cgi). Used the very cool mod_throttle_access to accomplish this. Hopefully this kind of thing will be built-in to Apache 2.0!

20 Sep 2000: I revamped the way the nntp scanner works. It's now scalable to any Unix machine on my LAN, and a bit more fault tolerant, too. Before, the nntp-scan process ran on one machine ('kadath', also the database server.) If I had to reboot kadath, the scan would be interrupted and I'd have to start from scratch. Now, the nntp-scan process can run on multiple machines (my laptop, for instance, and a Sparc IPX that's otherwise sitting around idle), and if the scanning process dies or is aborted, the host(s) it was working on are put back in the scanning queue and will be tried again later. It's kinda cool.

I suppose the next step would be to have other people do scanning for me and then send valid servers to me, so all I do is load them in the database .. hmm..

27 April 2000: I spent a day or so creating a feedback page (for those of you who remember the old one, I think you'll like this one better.) It's about 300 lines of PHP and the threading is probably buggy as hell, but if you have something to say, feel free to post up.

11 April 2000, 00:30 PDT: It's ba-aack..

My stuff survived the move northward, and I have a faster DSL connection to boot. Whoo hoo! Quite an experience to move from a heavily saturated 33.6kb modem to a DSL connection!

I've made more changes to the back end -- putting the web server on a dedicated, faster box, for one -- so if you notice any dead links or other problems, please let me know. All http://www.jammed.com/~newzbot links should forward automatically to the new domain, www.newzbot.com. New scans should begin any day now . . .

7 Feb 2000: I redesigned the database schema to make it much more efficient. This means searches will be faster and shouldn't time out occasionally (SQL DELETEs can be a killer on a poorly designed database!)

Some things have disappeared, like the article count for the largest group. This will be coming back in a few weeks, along with some other things.

Also, I've noticed that many people have been complaining that the site is slow. Well, that's because my connection to the Internet is a 33.6 modem. Believe me, it sucks waay more for me than for you -- I telecommute off of this thing! But I'm relcoating in the end of March -- which means the site will be down for a few weeks -- but when it comes back up, it should be on a dramatically faster DSL line (the majority of which will be payed for by ad revenue! thanks for clicking on those banners, guys!). So just a little more patience, please.

I've also fixed the hardware problems (bad swap drive.. ooga). uptime should stay at 100% after I re-assemble my db server.

12 Dec 1999: Been having some painfully serious hardware problems lately, so the scans won't be (haven't been) updating regularly & the search database will be spontaneously unavailable. Sorry.

29 Nov 1999: Gobble gobble. Before I left for a thanksgiving vacation, I discovered & fixed a bug that essentially resulted in no new hosts getting scanned. It's fixed now.

29 Oct 1999: Urp. I'm having some disk problems & the group list database got corrupted. I'm rebuilding it now.. but searches will fail until the database is fully rebuilt.
It works again now. :-)

21 May 1999: the new search engine wasn't working with certain browsers. It turned out I was screwing up the HTTP headers. It should work now for everyone, but, as usual, if you have a problem, e-mail me.

I've also added news:// links to the groupname in the search engine, for your convenience.

15 May 1999: Well, it looks like the new search engine is online. Please let me know if you have any problems with it. I'll be adding more features to it as time goes by.

9 May 1999: I've been working with mySQL and the very useful DBI perl module lately. I'm in the process redoing most of back-end stuff to use SQL instead of flat files & GDBM. The results so far have been very promising. It will take me a while to redo the search engine to speak SQL, but I think it will be worth it. (Think of a query like "show me all servers in Australia that have more than 700 articles in the group 'alt.fan.beer.fosters' that have been in the database longer than three weeks and two days." That's the power of SQL. It's quite cool. Why didn't I do this sooner?)

Anyway, check this space for developments. The old engine will probably languish until I can get the new one up.. hopefully sometime in the next two weeks (I'm very busy at work.)

1 May 1999: OK, things are settled. A new scan should be up by sometime next week. Sorry for the delays / downtime ..

20 April 1999: I changed apartments again -- it will be a few days before I get myself settled. The lists may not be updated with their usual frequency.

20 Feb 1999: Grr. Everything was broked. It's fixed now :)

17 Feb 1999: Finally upgraded my two x86 machines from RedHat 4.2 to 5.1, as well as taking my Sparc 1 from Solaris 2.3 to 2.6. Now it's time to play with neat things like php3 and SQL.

Eventually, the site will be database-driven, which should make the search engine more useful (show the date the host was checked, if it allows posting, etc, etc, etc.) I'm also toying with the idea of letting people visit a page where they will be 'cookied' -- after which they will be able to view all the new servers since their last visit. Useful for those repeat visitors :-)

If I'm able to make enough money with my ad revenue, I will upgrade my network connection. This means a faster experience for you guys -- as well as potentially offering NNTP feeds -- so click on those banners!

12 Jan 1999: I had a bug! Can you believe it? It's true. I had a bug. Servers that were known to be bad were being published in the list. That ain't right. It's all better now.

3 Jan 1999: Search engine repaired. I gotta overhaul this thing sometime soon..

1 Jan 1999: Happy new year! The search engine is busted. I'm fixing it now. So much for "fully automated" ...

22 Dec 1998: added advertising to hopefully offset some of the costs of providing service. Click on those banners!

20 Dec 1998: resistance's motherboard failed. I moved the scanner to a different machine, "kadath".

5 Dec 1998: I duct-taped resistance back to working order -- it's now diskless & boots up via NFS. Updated lists are being published, but the search engine is busted. Grr. GDBM w/ NFS gets twitchy sometimes. I'm also going to remove the "feature-free newsreader" link. It's far too slow to be of any use. Those that rely on it can still use it if you've bookmarked it.

28 Nov 1998: Wow, dealing with fraud sure is annoying.

24 Nov 1998: OK, kids, expect delays. I've been pretty busy at work and haven't had a chance to fix the HD in resistance. No ETR at this point. Watch this space for news.

19 Nov 1998: Yikes. I've been busy, and haven't been able to address the recent problems with the scan. These are twofold:

one, I made a change in the way the scanner operates so it's a bit more efficient, but in doing so, I broke something. The breakage caused empty lists to get published. Oops. I just fixed it.

two, 'resistance' has a bad hard drive. It's caused all sorts of problems for the past month and I'm going to swap it out this weekend. (anyone want to donate a decent IDE drive? :-)

9 November 1998: OK, I know the scan hasn't run for a while. I'm breaking things, but they'll make things work better in the end. (I hope..)

1 November 1998: I just got back from vacation & brought resistance back online. A scan is running. Rejoice!

28 October 1998: The local power company, PG&E, shut off power Monday morning and is shutting the power off again Friday morning. Since resistance doesn't yet have a UPS, the scan can't run. Sorry. Wait another week or so :-) I should be getting an additional UPS soon, so power failures won't affect the scanner.

$Id: changelog.php,v 1.42 2005/08/19 04:45:15 jwa Exp $