Introduction to NewsGroups

(1)-- What are NewsGroups?

The internet originated as a tool for scientists to communicate ideas amongst eachother with 'newsgroups' being like the forums of today. Scientific texts fall under the academic freedom to discuss any subject without obstruction, so copyright doesn't really apply to texts in newsgroups because scientists should be able to examine everything. It didn't take long before the textmessages were grouped together according to subjectmatter, so that's where the name "NewsGroups" comes from.

The newsgroups are maintained by a network of servers connected together across the world -- mostly by ISPs and universities, I think, but plenty of commercial ones too. Anyone can post an e-mail message to a newsgroup on a newsserver, and that mail is then distributed automatically to all the other newsservers (like regular news spreads in real life). In fact, the messages are called "articles" like in a newspaper, each with its own 'header' headline that identifies the article. Note that the headers and articles are spread seperately, so a newsserver may list an article as "complete" but that may just mean that all the headers were found -- the actual articles that contain the stuff may not be fully on the server (yet). NewsServers are generally rated by two factors: completion(=how complete the articles are) and retention(=how long the headers+articles are kept available). And for commercial newsservers, the price and the downloadspeed you get, are important too, of course. Most ISPs have their own newsserver (probably with free access for their internetclients) and in some cases it will actually be pretty good.

Anything digital is actually just binary code, which means it is made up of only 1s and 0s (like 100011110100101111010111010 ... ). In binary code, there is no real difference between a readable text and a text that's just a bunch of characters put together which is exactly what a program or game or picture looks like as text = the readable source programming code of a game is unreadable because it has been 'compiled' to run faster (and to hide the sourcecode). Scientists didn't just want to show others readable text, but pictures and programs as well, so those unreadable texts, or "binaries" as they are called, also got posted to the newsservers (encoded in such a way that they only use readable letters but the text is still gibberish). Naturally, they also became grouped together, so look for groupnames that start with "alt.binaries" (ALT stands for "Anarchists, Lunatics and Terrorists") if you want more than just text.

(2)-- Accessing newsservers:

The newsservers are accessed via the nntp protocol (Network News Transport Protocol -- compare this with http protocol for websites), so you need a special newsreader. Some webbrowsers and e-mail programs can handle it, but you really need a specialized client for reading and automatically reconstructing the binaries from their encoded article-form on the newsserver (mostly used, is "yEncoded" or "yEnc") to their original form.

There's several free newsreaders around: a link to Slyck for a list of them. The best, in my opinion, is Newsleecher but the latest version is not free (and their supersearch feature is only available if you paid for access). GrabIt is also a great newsreader -- it's free and has a working search function. Scroll down to section (4) for details on how to configure a newsreader.

Note that some caution is adviced if you want to post stuff to a newsserver (with PowerPost for example). You're dealing with ISPs and educational institutions, so you'd better do it anonymously if you want to avoid getting a warning (or worse) for abuse from your ISP. Look for a 'remailer' service that will will strip the source from the mail's header and then send it onto the newsserver. This introduction guide is aimed at downloading only, though.

Typically, the stuff you find on newsgroups, comes in a whole set of files, with mainly these types:
• BunchOfStuff.nfo
  :: the .nfo file contains information about the posted stuff -- you can open it with Notepad, but be aware that .nfo files are usually associated with Windows information, so doubleclicking on them will likely give you an error until you rightclick on an .nfo file in Explorer and then use the 'Open With' menuitem to associate them with Notepad.
• BunchOfStuff.nzb :: a small file that is like a torrentfile for newsgroup articles. If you only download this, and then import it into your newsreader, the newsreader will automatically look for all the BunchOfStuff files that are associated with it. There are plenty of .nzb sites & forums on the web, that list nzb files exactly like other sites do with torrents.
• BunchOfStuff.rar with a whole series of
  BunchOfStuff.r00,
  BunchOfStuff.r01,
  BunchOfStuff.r02, ...
  :: you often find these in torrents too -- it's a (series of) WinRAR compressed volumes (like a series of .zip files) that contain the actual stuff that is being shared -- you can extract the stuff from it by selecting one of the rar files in WinRAR and then selecting "Extract to" from its menu.
• BunchOfStuff.par2 with a whole series of
  BunchOfStuff.vol00+1.par2,
  BunchOfStuff.vol01+2.par2,
  BunchOfStuff.vol03+4.par2,
  BunchOfStuff.vol07+7.par2, ...
  :: these files contain a number of repairblocks (the +4 part indicates how many). Check the following guide-section for a closer look at repairing files with these .par2 files. Note that you don't have to grab all of these immediately -- just get a few small ones in case there's a couple of damaged blocks and you can always download the rest if needed.
• BunchOfStuff.sfv
  :: the .sfv file (or sometimes an .md5 file) is similar to a .par2 file, but it can't repair anything -- it will just check whether the file has been damaged or not. The par2 check is better, so only check with par2 if available, but .sfv files are still used on FTP servers (which is where the stuff usually comes from) so you regularly find them on the newsgroups too. Always download them if they're included, because the .par2 check may look for them.
Note that the stuff doesn't have to be in a compressed rar: you will often find .mp3 and .avi, ... but remember that if they're not accompanied by either an .sfv (or .md5) or a .par2 file, then you have no way of checking whether they are complete or errorfree after you download them -- even if the newsserver lists them as complete.

(3)-- Repair incomplete files with QuickPar:

When newsservers exchange their new messages with eachother, the transfer can go wrong, or maybe one of the server's harddisks has an error -- this may not be a problem when the error affects one letter in a readable text, but programs only work if all the binary bits are in exactly the right order, and because they are a lot bigger (they stretch across many articles) than textdocuments, they are more likely to contain errors from being transferred across the world.

To counter those errors, people usually send a set of "par2" files with them, that carry repairblocks for the original files so if a part of the file got damaged, the repairblock will take its place. QuickPar is the tool to use for repairs. Once QuickPar is installed, you can doubleclick on any .par2 file posted with the stuff, and it will check how many blocks of the stuff is undamaged -- it will also scan how many undamaged repairblocks are available in the .par2 files. If you don't have enough repairblocks, then you're out of luck unless the original poster decides to post a new set of .par2 files. Sometimes, people 'crosspost' the same stuff to several newsgroups, so you may find a copy in another group that's a little less damaged.

An example of how QuickPar works:
suppose you have a BunchOfStuff.rar (with the stuff you want inside it), and a .par2 set of files that is associated with the post:
BunchOfStuff.par2
BunchOfStuff.vol00+1.par2,
BunchOfStuff.vol01+2.par2,
BunchOfStuff.vol03+4.par2,
BunchOfStuff.vol07+7.par2, ...
You want to check the .rar file to see if it is undamaged so to do that, doubleclick on any of the .par2 files and QuickPar will start checking the .rar and will tell you how many repairblocks you need if it's damaged. Suppose that 13 blocks in the .rar are damaged, then there's no problem because you have enough repairblocks in the .par2 files: the +2 part indicates how many repairblocks are in the .par2 file so in total, you have 1+2+4+7 = 14 repairblocks (if the .par2 files were damaged too, then there may be less available repairblocks inside them, but QuickPar will tell you that too).


You can see in the example picture above, that QuickPar is checking all the blocks in all the files, and then tells me that there's so much damage that I will have to go find 22 more blocks -- which sadly, I couldn't, so still waiting now for a repost of the stuff, or extra pars, or maybe I'll find a torrent that has the files too, so I can download some complete files from the torrent, and then the repairblocks will be enough to repair the rest.

(4)-- Configuring a newsreader.

For all newsreaders, this is the sequence of actions you must take in them, to be able to choose what stuff you want to download:
• Add and configure the newsserver that you're gonna use (given that you are allowed access to it, of course).
• Get the GroupList from the newsserver. Not all newsservers carry the same groups, but hopefully yours will have some big alt.binaries groups to get the good stuff.
• "Subscribe" to the groups you want to check. Sort the groups by size first: the bigger ones are more interesting ;)
• Update the Group(s), so the newsreader will get all the headers for that group from the server (some newsreaders can only update one group at a time). If the newsserver has a long retention period (=lots of stuff on it), then this will take too long -- around 2 million is the maximum I get, but it'll depend on the speed you're getting and how much downloadvolume you're willing to spend on it. The first time, you'll have to do a full update, but afterwards you can do incremental updates so you only download the new headers. But you'd better switch to downloading via .nzb files if the groups are too big because .nzb files only contain the headers associated with the posted stuff -- very similar to .torrent files.
  Note that because the headers and articles are spread seperately, there may be groups on the server that you can download articles from, but not the headers. Some ISPs censor the headers of a group, but still have all the articles, in which case you can still download from a censored group if you find an .nzb file with the missing headers (or get the headers from a different newsserver that doesn't censor it).
• Activate the group once you have the headers, so it shows them and then you can pick the ones you want to download (sort by subjectname first for easy picking). The headers you choose will be put into the downloadqueue, and it will start downloading the articles (you can pause & stop this). Best way is to sort the downloadqueue by date first and get the oldest articles first because the older an article gets, the more likely it is to be damaged. And another reason: very new headers may mean that the associated article isn't fully on the newsserver yet (headers get spread more rapidly as they are just the title for an article). Remember that the article (which contains the data) may not always be completely on the newsserver even though the header lists it as complete -- commercial newsservers are likely to have a better "completion" rate than ISP-based servers, but a backup newsserver ("fill-server") may be handy to minimize the need for QuickPar repair work.
• It's best to do a par2 check immediately after downloading the stuff, because you may need extra repair-blocks and it's best to get those as quickly as possible before they get damaged too. Usually, people post the .nfo & .rar stuff first, and then a .par2 set follows so the pars are likely to be less damaged as they're not as old. Note that a QuickPar check is very processor- and memory-intensive, and your newsreader may also use quite a bit of memory, so make sure your system can handle it (QuickParring a reasonably damaged 4GB DVD takes about 7 minutes on my dualcore AMD X2-3800+ with 1GB ram).

^^^Picture Above :: On the "Usenet Mgr" tabpage, click on the "Add New Server" button (top-left) and you'll get the Server Setup panel on which you can configure your newsserver.

-- On the "Login Details" tabpage of that panel, fill in the following:
• Server Address: You need to know the server's web-address (either the domainname or the IP number) -- most ISPs will have something like news.ISPname.com but you'll need to check first.
• Port: 119 is generally used, so leave that if you didn't get specific details for it.
• Server nickname: free to choose, just to identify it.
• Number of connections: most newsservers have a limit on the number of connections you can use, and some will also limit the speed that each connection can achieve, so check whether your newsserver host has instructions for this -- you can always edit the server's properties later on if you chose too many or too few.
• Priority: You can set a priority for each server, because you may wish to use some servers as backups so NewsLeecher will try to download from a priority1 server first, and if the article is incomplete on that one, it will switch to a priority2 server if no other priority1 servers are listed. Note that there's a bug in NL's priority system, though (NewsLeecher will keep trying to download indefinitely).
• Server requires LogIn: if the server requires you to have a username and password, then fill it in here.
  
  -- On the "Server options" tabpage of that panel, fill in the following:
• Use for Group Updates: use the server to get the headers for the groups. Some newshosts may have seperate servers for articles and headers.
• Use for server searches: NewsLeecher's own SuperSearch function doesn't work unless you paid for access to it, but your own newsserver-host may have a similar searchfunction ("XPAT" command) so this option will still let you search the newsgroups on the newsserver.
• Use for article downloads: use the server to get the actual articles of data. Some newshosts may have seperate servers for articles and headers.
• Exclude header download from "downloaded bytes" stats: not important -- just for keeping track.
• Send "Group" command to server before downloading: enable it as it may speed things up on some servers, but you can ignore this (unless the newsserver host has a specific setting for it).

Click OK to complete the server setup, and you'll notice that NewsLeecher has added the server on the left, and the bottom-right section will show the number of connections that server added, with speed & status info for each (and a general speed-graph on the bottomleft).


^^^Picture Above :: Still on the "Usenet Mgr" tabpage, click on the "All groups" button (top-middle) and then click the "Fetch Group List" button and Newsleecher will get the list of groups that are on the server (you may have to press the blinking "Connect" iconbutton in the topmenu first). After downloading the list, doubleclick on the 'Popularity' columntitle to sort the groups -- the biggest alt.binaries groups should be the most interesting (you can filter them by name to make it easier). Select & rightclick on the groups you want to "Subscribe" to, so they will be added to a shortlist of favorite groups.


^^^Picture Above :: Select the "Active Group" tabpage, and you'll see your subscribed groups on the left, but you need to update them first so select & rightclick the ones you want to update. The headers for the groups will then start to download, and you can see them if you checkmark a group on the left or rightclick and "activate" it.

The headers are listed on the right, in different colors: red for incompletes, white for complete headers, and green for newly downloaded complete headers (colors may differ in other newsreaders). The "subject" column shows you the name of the article -- most give an accurate description of what is in it, but note that there's plenty of poorly named posts too, and you will often find that you can only identify what's in the posted rar files, by taking a code/ID-number from the subjectline, and entering that code in a forum (not always English speaking) or webpage associated with the user that posted it. A couple of sites that you will see often: FTD, FTN (needs registering). If there's an .nfo file included, then rightclick it and select "Open/Read" -- they usually contain all the info you need to know what's in the posted stuff (check section (2) of this guide for more details about .nfo files). Doubleclick a header, or select & rightclick+"Leech" to move them to the downloadqueue and the articles associated with the headers will start to download (it's best to sort the downloadqueue by age -- download the oldest articles first).




^^^Picture Above :: The GrabIt installation wizard will have asked you the server details already, but just to make sure, rightclick the newsserver item listed the left column and select "Server Properties" (If you need to add a server first, then rightclick the "My GrabIt" item).

-- On the "Server Properties" panel, fill in the following:
• HostName: (=server address) You need to know the server's web-address (either the domainname or the IP number) -- most ISPs will have something like news.ISPname.com but you'll need to check first.
• Port: 119 is generally used, so leave that if you didn't get specific details for it.
• Account Name & Password: if the server requires you to have a username and password, then fill it in here.
• Server timeout after: keep at default 3 minutes, unless you see lots of timeouts in the connection's status on the bottom of the "Batch" tabpage (=downloadqueue) -- increase it then, as the server is probably too busy to respond quick enough.
• Retries on error: keep it at the default 3.
• Maximum allowed connections: most newsservers have a limit on the number of connections you can use, and some will also limit the speed that each connection can achieve, so check whether your newsserver host has instructions for this -- you can always edit the server's properties later on if you chose too many or too few.

^^^Picture Above :: Select the newsserver on the left, and rightclick it for "Refresh GroupList" and GrabIt will get the list of groups that are on the server and to see them, click on the "All Groups" tabpage (top-middle). Doubleclick on the 'Article Count' columntitle to sort the groups -- the biggest alt.binaries groups should be the most interesting (you can filter them by name to make it easier). Select & rightclick on the groups you want to "Subscribe" to, so they will be added to a shortlist of favorite groups that appears in the left column as you select the groups.

You can now select a group on the left, and do an update (full update to get all the headers, incremental to get only the new headers that you don't have yet).


^^^Picture Above :: GrabIt will automatically switch to the "Articles" tab to show the headers that are in the selected group (the headers represent the articles, of course). The colors have pretty much the same function as in NewsLeecher: red for incompletes, and white or yellow for completes. You can use the vertically placed iconbuttons to filter the headerlist. Doubleclick on the 'Subject' columntitle to make it easier to select the headers/articles that you want to download.

The "subject" column shows you the name of the article -- most give an accurate description of what is in it, but note that there's plenty of poorly named posts too, and you will often find that you can only identify what's in the posted rar files, by taking a code/ID-number from the subjectline, and entering that code in a forum (not always English speaking) or webpage associated with the user that posted it. A couple of sites that you will see often: FTD, FTN (needs registering). If there's an .nfo file included, then rightclick it and select "Read Article" -- they usually contain all the info you need to know what's in the posted stuff (check section (2) of this guide for more details about .nfo files). Doubleclick a header/article, or select & rightclick+"Grab selected articles" to move them to the downloadqueue (=on the "Batch" tabpage) and the articles associated with the headers will start to download (it's best to sort the batch/downloadqueue by age -- download the oldest articles first).


That's it! -- Happy downloading!!

ADL_242

Thanks
I have just got ultraleech and now this neat Intro to help me figure this usenet thingy out

thanks man

i used to use them at UNIX system, but that was along time ago

i think i should go back to my old love

I have considered taking the plunge into learning about newsgroups for some time. Now I am starting to do it. Thanks for your lesson.

It looks pretty complicated , but I think I can learn it.

Got it all set up and working

Can anyone recomend any good groups for movies?

If you use .nzb files (like .torrent files) then you don't have to worry about the groups anymore. Here's some searchsites -- they allow you to create an .nzb file of the stuff you want. Newsleecher will automatically import the articles mentioned in the .nzb, if you doubleclick on the .nzb in Windows Explorer.

http://newzleech.com/
http://www.ngindex.com/
http://www.newzbin.com/
http://www.nzbindex.nl/

The par2 files are paused by default, I believe (check it in your options), so you don't needlessly download them all. It's best to par-check the files immediately after downloading, to know how many (undamaged) par2 blocks you need to get.

In steps:
-- search for the stuff you want. Make sure that you have all the parts, then click the "Get nzb" button to save the article-list to .nzb file.
-- Browse to the .nzb file in Windows Explorer and doubleclick it (or run it straight from your browser), and NewsLeecher should add the articles to its queue and if you're already connected, it'll start downloading them.
-- Once the download is complete (the extra .par2 blocks will still be paused), doubleclick on the primary .par2 file that did get downloaded (or any par2 that belongs to the same stuff), and QuickPar will check them all. If you need extra blocks, then download the amount you need, and re-run QuickPar and let it repair the files.



EDIT :: there are also nzb forums like torrentforums, by the way. And if you want to look up an 'FTD id', then use: http://nfo.ftd.nu/

have you mentioned to ppl that to get a good retention you also need to pay, basically you need money to get an account and to get the best speeds

I think so but dunno how clear I made it Some ISPs now outsource their newsservice to commercial newsproviders, so it's worth checking whether this is the case.

btw, Welcome to the Darkside, DylanDog!

Thx man, am glad to be here.

Oh and its best to make things clear, ppl wont get stuff from fast usenet servers for free, they must pay, I am a user but dont give money because I know ppl so they gave me access, I must say its like a topsite except its 0hr, topsites are 0sec,lol, so basically you get your stuff pretty fast even faster then on level 10 trackers, theres no need for uploading and the servers depending on your speed can get easy 6mb speed, simply awesome, its totally worth the money.

DD

I use Xnews made by Lucid. It's FREE and it has no limitations that I've found. I've used it for over 3 years and 4 different ISP's.

Comcrap (comcast) was the only one (shortly after buying out TCI in Wa.) that limited Usenet to 2GB per month. All others since have offered unlimited/uncapped bandwith through their news servers.

There are free usenet software sources that are fantastic. The problem is whether or not your ISP gives you that freedom.

Jic you're curious, I tried giganews and other "pay" news services when my limits were in place. If your provider gives you Usenet, use it, and don't pay.

I tend to use it for other than what I find here on the Darkside, but i'm familiar with the system.

winrar, HJsplit, and Quickpar downloads are all free and will benefit you immensely.

BTW if you run into Yenc files, you can get Yenc to decript, but a fantastic feature of Xnews is that Yenc decoding is embedded and it auto deciphers those files.

Check it out.

If I can add or help anyone further with Usenet/Newsgroups or readers, let me know.

I use Newzbin and giganews :) and I like it.... though its a bit expensive

ok i just found another good guide that might also help anyone who likes to download stuff from newsgroups

i hope you dont mind ADL_242

Slyck's Guide to The Newsgroups

I use Forte Agent and NewsGuy, but as Mr. Cheese said, its a bit expensive

what about vm newsgroups any one got free links for that
they swicth the old ones off so no longer get stuff with grabit

Altopia.com
$9/month for 5 connections
unlimited transfer


Best $9 I spend every month.

My speed usually stays right around 800KiB/Sec

that's a 700MB d/l in around 15 min.. Compared to an aXXo torrent of the same size taking me around 2 hours.

And the best part is the SSL connection.

thanks for the awesome guide.grabbed grabit and after a quick google i found my isps newserver addy. then a quick trip to scene releases to find a recent nzb and within an hour i have a complete movie now i know 175kbps isnt amazing but my torrents struggle to see 20kbps at this time of day and its free so im very happy

Very Nice ADL_242 !!! I have been using newsgroups for years and it's a great place to find what you want . Your guide is very informative and well written for those who are looking into usenet . I am one of the lucky ones who has a ISP that provides free news server . I have Giganews and it provides me with over 107,000 groups so can get just about anything I want . Again, this is a very good guide for those who want to learn usenet .

Good job on the guide ADL_242