Pages not being indexed?

[mynewchoice]

As I develop the content on my site and try to work on increasing traffic, I have been checking Google to see if my pages are being indexed by simply performing a search such as:

site:mynewchoice.com

This results in one result for my main page. Does that mean that none of my other pages are being indexed? Is it only indexing the main page of my blog? I am receiving regular visits from Googlebot according to my logs.

I have a robots meta tag set to "INDEX,FOLLOW" and I also have a robots.txt file at my root that only has disallow set on one directory. Did I do something wrong? Shouldn't I be seeing the article pages being indexed as well?

Maybe my permalinks are the reason for the lack of indexing. The articles all are set to a path like this: "/<year>/<month>/<day>/<articlename>/". Did I allow too many directories in my permalinks? Should I change that now?

-mnc

[ricklomas]

My daughter woke in the midle of the night and I can't get back to sleep, so I have made you an XML Google Site Map (http://www.ricklomas.com/wordpress/?p=30).
If anybody else requires this service I will require $10 (by PayPal)
I have found that submitting sitemaps to Google does seem to improve the amount of indexed pages and so increase traffic slightly.
After you have downloaded your XML Google Sitemap from my web site (http://www.ricklomas.com/wordpress/?p=30) your next stop should be here! :) (https://www.google.com/webmasters/sitemaps/siteoverview?hl=en)

[mynewchoice]

Thanks. On my blog, I already am using the Google Sitemaps plugin and have it updated whenever I add a new post or page. I did notice that my sitemap.xml does not seem to include the "/feed" links or the "/trackback" links. Maybe that is a limitation of the Wordpress plugin that I am using.

One thing I had apparently not done was actually submit my sitemap to Google, which I am doing right now.

Thanks for the pointers!

-mnc

[ricklomas]

I never even thought to see if anyone had done a sitemap plugin!:eek:
BTW I loved Bryan's theme, so I am giving it a go on my Rick Lomas Blog (http://www.ricklomas.com/wordpress/)too.
I really wish those wide skyscrapers were not so pig ugly!:)

[mynewchoice]

BTW I loved Bryan's theme, so I am giving it a go on my Rick Lomas Blog (http://www.ricklomas.com/wordpress/)too.
I really wish those wide skyscrapers were not so pig ugly!:)

Yeah, I have been pretty happy with Bryan's theme to this point although I went with the narrow layout. I like the wider layout but it is just a bit too wide for 800x600 which is what I decided as my lowest resolution to support.

-mnc

[Elizabeth]

Let me try asking this here instead. Will the sitemap update itself when a person adds new content? Or do you have to create something new every time you add an article or new page? Or is it somewhere in between?

:confused: :confused: :confused:

[mynewchoice]

Let me try asking this here instead. Will the sitemap update itself when a person adds new content? Or do you have to create something new every time you add an article or new page? Or is it somewhere in between?


Elizabeth,

It probably depends on your implementation. For instance, I have a Wordpress blog and am using the Google Sitemap plugin. The plugin automatically updates the sitemap every time I update or add content to the site.

How did you create the sitemap the first time? Did you do it through an automated process?

-mnc

[Elizabeth]

Elizabeth,

How did you create the sitemap the first time? Did you do it through an automated process?

-mnc

I haven't done it yet. Am a bit confused over the whole thing. :confused: I have both a conventional site and a type pad blog. You just gave me an idea to check type pad for tips over there - thanks. It seems like the site map should be able to detect new content, but I don't understand at what level or how it works...

[mynewchoice]

I haven't done it yet. Am a bit confused over the whole thing. :confused: I have both a conventional site and a type pad blog. You just gave me an idea to check type pad for tips over there - thanks. It seems like the site map should be able to detect new content, but I don't understand at what level or how it works...

I would imagine that type pad has some sort of plugin that can generate the Google sitemap. Basically, it is just an XML file that contains information about the links/content on your site. When adding new content to the site, the plugin typically will automatically update the XML file with the appropriate information.

On your conventional site, I am not sure but there may be other utilities that will help automate the process. Maybe someone with experience using the sitemap on a more convential site will have some feedback for you.

-mnc

[Elizabeth]

Follow up for anyone reading this thread: It turns out that google site maps only work on type pad accounts at the highest paying level (Pro). In my case, it may make more sense to shift everything over to movable type than to upgrade. I would have started with movable type in the first place, except my web hosting company at the time couldn't accommodate it. I've since switched hosting companies. ;)

[Sbabb]

mnc: Google will eventually get around to indexing your whole site. Sometimes it takes awhile, though.

Have you considered cutting back on the number of posts displayed on the main page? If I scroll down to read everything there, the AdSense is long gone when I'm done reading and I'm looking for something to click. I have a WP blog managed site and I only display the latest post on the front page. Anywhere you scroll, you can see AdSense.


Scott

[mynewchoice]

mnc: Google will eventually get around to indexing your whole site. Sometimes it takes awhile, though.

Have you considered cutting back on the number of posts displayed on the main page? If I scroll down to read everything there, the AdSense is long gone when I'm done reading and I'm looking for something to click. I have a WP blog managed site and I only display the latest post on the front page. Anywhere you scroll, you can see AdSense.
Scott

Scott,

Thank you for the feedback. I have considered cutting back on the number of posts displayed but I didn't want to go too far as I would like to have a handful of posts to entice visitors to continue reading.

Currently I am displaying 8 posts on the main page. What do you think would be a more reasonable number? 5? I just didn't want to have too few posts on the main page.

Another thing I was thinking about was adding two additional ad units to the main page, as I only have the one right now in the sidebar on the main page. I could add an additional skyscraper to that center section and/or add a square or leaderboard down towards the bottom of the page.

I'll have to review the site and see where additional ad units will fit best. Thanks for the idea.

-mnc

[Elizabeth]

Mnc, another option -- maybe you've tested this already and didn't like it -- would be to display several article starts on the main page for enticement, then have people click through to finish reading whatever they're interested in on another page. The target page displays the full complement of AdSense.

Also, and this is one thing type pad does do well, it's displayed as a "permalink" meaning that the full article is individually archived (with all its AdSense) for eventual inclusion in the search engines' databases.

See example: http://www.healthylivingdiy.com

Elizabeth

[mynewchoice]

Mnc, another option -- maybe you've tested this already and didn't like it -- would be to display several article starts on the main page for enticement, then have people click through to finish reading whatever they're interested in on another page. The target page displays the full complement of AdSense.

Also, and this is one thing type pad does do well, it's displayed as a "permalink" meaning that the full article is individually archived (with all its AdSense) for eventual inclusion in the search engines' databases.

See example: http://www.healthylivingdiy.com

The example URL doesn't seem to be working, as I am receiving an error message that the forwarding service has an issue or the site is not online.

There are some posts where I use the snippet idea and have a link to continue reading the post. However, some of the posts are shorter and I am not sure would really warrant having the continuation link. But it is another possibility to consider.

FWIW, Wordpress has a similar permalink capability where every post receives a unique URL that can be used for addressing specific posts. Thanks for the feedback!

-mnc

[Elizabeth]

The example URL doesn't seem to be working, as I am receiving an error message that the forwarding service has an issue or the site is not online.


Hmmm. Works when I click on it, so I don't know what the problem might be. Thanks for the tip, though.

BTW I occasionally have a shorter post and in that case, simply don't continue through to the second page. Most are long enough to continue.

Elizabeth

[mynewchoice]

Hmmm. Works when I click on it, so I don't know what the problem might be. Thanks for the tip, though.

For reference, here is a screenshot of what I see when I visit the URL that you had specified. My assumption is that this is not what you are seeing as I do not see any posts at all.

-mnc

[Elizabeth]

You're right, that is not at all what I see. Wonder where the problem is? Wonder why I see the blog and you don't? Wonder how I'll know when / if it's fixed, since it looks fine when I call it up? Looks like another mess! :eek:

Thanks, I think.

[mynewchoice]

You're right, that is not at all what I see. Wonder where the problem is? Wonder why I see the blog and you don't? Wonder how I'll know when / if it's fixed, since it looks fine when I call it up? Looks like another mess! :eek:

Elizabeth,

Just wanted to let you know that your site is now displaying properly for me.

-mnc

[Elizabeth]

Elizabeth,

Just wanted to let you know that your site is now displaying properly for me.

-mnc

Thanks! All I've done so far is file a support ticket with my registrar. Perhaps they did something on their end -- or else it's another mystery. I am quite happy to hear it's back, though. Thanks for checking. :D

Have you noticed any improvement since you posted your site map with Google? Or is it too soon to tell?

Elizabeth

[mynewchoice]

Have you noticed any improvement since you posted your site map with Google? Or is it too soon to tell?

No improvement with my actual post permalinks being indexed. Thus far Google is still only showing my main page as being indexed. I'm trying to do some reading on Wordpress as I am getting a bunch of 302 redirects that might be from trying to crawl my permalinks.

The only reason I am questioning that is there are many other popular blogs that use the same permalink structure that I am using and are all indexed. It might just be a matter of time since my site is still new.

-mnc

[Elizabeth]

It might just be a matter of time since my site is still new.

-mnc

Thanks for the update. I hope you're right on the time factor. I read an interesting article today that says if "G" is ranking us on history, then it only stands to reason that they want to make sure we have some longevity & fortitude before they start showing us. :)

Enjoy the wait, I guess, and as Joel says - when they do let you "out," hopefully you'll rise to the top!

Elizabeth

[mynewchoice]

Enjoy the wait, I guess, and as Joel says - when they do let you "out," hopefully you'll rise to the top!

In some of my research, I noticed that I have some really good results on certain keyword phrases where my page is being returned in the top 10 results and another in the top 30.

But so far it still looks like only my index page is actually being shown on the search engines. Just now I added a sitemap plugin (the old-fashioned HTML style sitemap of links) and added that to my site. Maybe that will help the search engines crawl all of the pages. Worth a shot I guess.

-mnc

[Elizabeth]

In some of my research, I noticed that I have some really good results on certain keyword phrases where my page is being returned in the top 10 results and another in the top 30.

But so far it still looks like only my index page is actually being shown on the search engines. Just now I added a sitemap plugin (the old-fashioned HTML style sitemap of links) and added that to my site. Maybe that will help the search engines crawl all of the pages. Worth a shot I guess.

-mnc

That's cool! I've had the old-fashioned hand-generated site map on my http://www.WordCures.com site from the get-go and I would have said it wasn't helping much... Except that I just did a "site:xxx.xxx" check on Google and they seem to have just learned that I exist. 82 pages indexed. Yea! (No traffic yet, and they still show no links, but at least something's happening.) I posted the site in May.

[mynewchoice]

That's cool! I've had the old-fashioned hand-generated site map on my http://www.WordCures.com site from the get-go and I would have said it wasn't helping much... Except that I just did a "site:xxx.xxx" check on Google and they seem to have just learned that I exist. 82 pages indexed. Yea! (No traffic yet, and they still show no links, but at least something's happening.) I posted the site in May.

Congrats! Every little bit helps and the more pages that are indexed on the search engines, the more chances people have of finding your site. Hearing this confirms my thoughts that I just need to be patient.

My site has been around for a few years but it is only recently that I actually developed an actual site. My plan is to just keep plugging away with original content, building backlinks wherever possible and wait for more to be indexed.

-mnc

[Elizabeth]

My plan is to just keep plugging away with original content, building backlinks wherever possible and wait for more to be indexed.

-mnc

Sounds very familiar... :D

[wm8c]

There are a number of free site map tools available on the internet, just Google the term "xml sitemap generator tools" and you will have many to choose from and they are all mostly free. I actually bought one from CoffeCup software but it's not necessary.

You have to re-run these programs when you add new content but then you can just replace your existing sitemap with the new one and off you go. If you are not already a memeber of Google Sitemaps, register. I always re-submit my sitemap when I add a couple of new pages.

Another trick is to download the Google Toolbar and once installed, type in your URL in the Google search on the toolbar. If your page is not indexed, it will get it into the cue. I've had pages get indexed as quick as 48 hours and as long as 14 days. It helps to have an older site, as that seems to help speed up the process but it can be done. Good luck...

[mynewchoice]

Todd,

Thanks for the feedback. I have a Wordpress plugin that generates my XML sitemap but I have not been resubmitting that to Google Sitemaps. That is an interesting idea.

I do have the Google Toolbar installed as I wanted to review the PR feature included in the toolbar. My main page is indexed in Google, it is all of my article links that are not yet showing up.

Thanks for the tips.

-mnc

[Elizabeth]

Update: I just checked my stats and found "2" referrals from google to my main site and "1" to my blog. "8" hits from search engine traffic combined on the main site, and one new subscriber from search engine results. This is after 3-1/2 months of the big zero.

Woo-hoo!:D

So based on this one small case study, since I apparently was indexed just last week, it must have helped. :rolleyes:

[BTW I think the indexing breakthrough was just a matter of waiting it out.]

WAIT! I think I know what I did! I put a google search box on my hand-created site map page shortly before the site got indexed. Could it have been a coincidence??? Example: http://www.wordcures.com/SiteManagement/SiteMap.html

[cheekyspider]

Index visibility - http://www.wordcures.com/

search engine pages indexed
MSN 108
Google 24
Teoma 0
Yahoo 61

Index visibility - http://www.healthylivingdiy.com/


search engine pages indexed
MSN 0
Google 1
Teoma 0
Yahoo 30

[mynewchoice]

It does seem to take Google a little longer to index more pages on one site than the other SE. As cheekyspider indicated with the stats for Elizabeth's two sites, I have seen similar results for my site:

MSN - 61 results
Yahoo - 15 pages / 53 inlinks
Google - 1 result

-mnc (still being patient - seeing nice traffic boost from blog carnival)

[Elizabeth]

... which confuses me further, since when I do a "site:www.mysite.com" search for my 2 url's (on the google tool bar) I get more indexed pages than cheekyspider shows (82 and 15, the latter of which was higher a few days ago). Cheekyspider's results are closer to what I was getting a couple of weeks ago, before this alleged big breakthrough.

I don't understand the fluctuation. And I don't understand why cheekyspider and I would get two different results doing the same search. Any ideas?

E

Nonetheless, there seems to be a slight improvement in results.

[mynewchoice]

Elizabeth, here is another alternative that you will find returns different results as well. Try "site:mysite.com" as opposed to "site:www.mysite.com".

Also, when I do that search (with www) for your site, I also see 82 results. However, if you actually click on page 8, the results change to display only 48 results with the rest likely falling under the "omitted results" link displayed by Google.

All this seems to say to me is that we can try our best to understand the search engines and their indexing/ranking but it will still remain somewhat a mystery.

-mnc