Total indexing...

[Scraper]

As my handle suggests, I use auto generation software to make really large websites (1000 webpages +).

I'm posting this thread because I wonder if someone could suggest to me some software that would work to index ALL the thousand pages rather than just a couple of hundred of them or something.

Does such a software exist?

[pamz]

I see that many "gurus" are touting http://www.robthesearchengines.com
Have any one bought and tried it?
Is is just like the rest of the DG sites that will disappear in a few months or is it different?

[businessguy]

I have never heard of the tool before.

But I have to say that I really hate the new push to name squeeze pages. You cannot even read what someone is trying to sell you for good money without giving up your name and email.

Arnold

[CJAndT]

Are you referring to "indexing" all your pages as in have all of your pages indexed in search engines?

If so, NO software exists to get all your pages indexed.
What gets indexed is completely up to each individual search engine.
Unless you own Google, MSN or Yahoo, you have no control over what they actually index.

And, with "scraper" sites especially, get used to a site doing good for a month or two, then dropping off to virtual oblivion. Very few scraper sites have any longevity - if you must do scraper sites, the key is to keep putting up new sites, and NEVER use any "default" templates.

[Scraper]

Chris,

I'm relatively new to the Adsense thing (started in June) and I've only been doing "scraper" sites.

I know exactly what you mean by them having no longevity. From June - mid August, I had one site do $1800 in Adsense revenue and now - it's GONE!

And another thing - have you tried any of the sitemap generators to get your sites indexed faster? I just bought Secret Spider Generator and it made a sitemap with all 2000+ pages of my Directory Generator website.

I've yet to see how fast the site will start seeing some Adsense revenue. You ever tried it this way?

[CJAndT]

As far as sitemeaps, I'm using one called SoftPlus GSiteCrawler (don't have a URL off hand, but you can do a search for it). It is a free Windows app that will crawl your entire site - just give it the root URL (http://www.somesite.com/) but make sure you put in the trailing "/" or it may follow off site links, and you wouldn't want that.

I have used it for some smaller sites (100 or so pages) and it takes about 5 minutes for that. I currently have it spidering a 600,000+ page site, and it has been going for about 4 days now, with alot more to go...

Anyways, providing the sitemap just lets Google know what is there, it still is no guarantee that the site will be fully spidered (or spidered at all...), but there is no harm that can come from it.

[kbaxter]

1. If you scrape, stay away from using Google Sitemap.

2. If you want to get 'fully indexed', it's all about the inbound... and quality of the inbound links.

3.Blog and Ping still works... but be careful on how often you do it. 2x per dy is ENOUGH.

4. Create a linking system between your sites (assuming you have a unique IP... and Class C IP for each site).

5. Post to blogs using xml and rss to blog.

Keith

[biggles]

I wonder why you say - if you scrape stay away from sitemap.

I have 50 or 60 real content pages, and a DG site behind them. I put a sitemap up about 2 months ago. Googlebot visits several times a day and takes 20 to 50 pages each time. Google doesn't often feed me any visitors for the scrapper pages, but doesn't seem to choke on them either.

What is your experience that led to that statement?

[mgrcentral]

I would have to agree - using the google sitemap facility would be like emailing google and saying "Do you like my generated site of thousands of pages? Please take a look...."