Technical help regarding google bot

[DropWizard.com]

Google bot has been pounding a couple of my ca sites this month. 29k visits on one and 12k on the other blowing through about 1.2 gigs of bandwith for sites that were set for 250mb. I've had to raise the bandwidth a couple of times to keep them up. I'm not sure what solution can be done to this problem other than banning their IP but that could have detrimental effects on rankings.

Is there a solution to this? Or??????

[Tia Wood]

Do you mind posting the URL?

[DropWizard.com]

A link is a link

http://www.inkrefill.ca

http://www.digitalprojectorbulb.com both are using associate-o-matic amazon stores.

[south]

If you have the site verified with google, you can set a custom crawl rate for it in google's webmaster tools.

Find it under site configuration / settings.

https://www.google.com/webmasters/tools/

[Tia Wood]

In addition to south, I think they are getting stuck on the pagination numbers in which you can add nofollow links to if associate-o-matic allows customizing like that. But I definitely try south's suggestion first.

[south]

And I would personally be careful about the adsense on ink refills. Not sure if that fits in their tos or not. I suppose it depends on how they count ads. But with no content, could get you banned, or the site blocked. Again, not sure, just my initial thoughts.

[DropWizard.com]

Thanks south I hadn't thought of the google webmaster options.

The software allows the input of the adsense code and I've seen other sites using adsense also. But they do have a forum so I'll follow that thought up. Not that I much care we're removing adsense from all our sites as they get revised. I'm tired of getting thrown pennies on the dollar while they take the majority of the rev.

Tia

thanks for the suggestion. I've pm'd you some info.

[Tia Wood]

Thanks. PMed you back.

[POLiSH]

I would not ban the bot but rather adjust the crawl rate using the google.com/webmaster tools for the domain in question.




As stated above ^^^

[whitebark]

Many spam bot operators fake the user agent and claim it as google bot. Check the ips - if not google's - ban them.

[Tia Wood]

Using google.com/webmaster tools or adding a nofollow might be ignored by other bots. There may be another solution as his pagination is set to 400 pages max, 10 products per page and under multiple categories, that adds up fast. There is a feature in associate-o-matic that I directed him to that would lower the pages it returns. That is where they are getting stuck. Resolve that and there is no need to ban, adjust or limit anything unless he didn't want it crawled at all.

[POLiSH]

Using google.com/webmaster tools or adding a nofollow might be ignored by other bots. There may be another solution as his pagination is set to 400 pages max, 10 products per page and under multiple categories, that adds up fast. There is a feature in associate-o-matic that I directed him to that would lower the pages it returns. That is where they are getting stuck. Resolve that and there is no need to ban, adjust or limit anything unless he didn't want it crawled at all.

Good points, for some reason I was thinking google was the only one in question. Going back to his first post. We are not talking the avg site, he's looking at almost 30,000 visitors per month, so this is a high traffic site with lot's of images. Let's look at the size of the page.

The Ink Refill site.

44699 kb x 30,000 = 1,340970,000. and that's just for one page.

If you think that 250meg will hold 30k in visitors per month on a ecommerce store, then I am missing something.

[DropWizard.com]

Good points, for some reason I was thinking google was the only one in question. Going back to his first post. We are not talking the avg site, he's looking at almost 30,000 visitors per month, so this is a high traffic site with lot's of images. Let's look at the size of the page.

The Ink Refill site.

44699 kb x 30,000 = 1,340970,000. and that's just for one page.

If you think that 250meg will hold 30k in visitors per month on a ecommerce store, then I am missing something.

Perhaps the fact that it gets around 20 visitors/day. The balance of the bandwidth is being eaten up by googlebot. I've reduced the 400 to 100 and the 100 to 50 as you suggested Tia. We'll see what affect this has. I went to their support forum and frankly there wasn't much help although there are others with the same problems.

I've also noticed the growth in disk space used and wonder if it's a result of googlebot hammering all those pages that are set for caching. Perhaps a clearing???

[DropWizard.com]

Ok a sample of recent visitors: As you can see it appears googlebot is trying to index amazon via our site All coming from 66.249.71.82
A lot of this stuff doesn't even belong on our site: (digital bulb btw)


/index.php?c=LCDProjectors&n=160356011&i=B0001LTT4Q &x=HP_48X_CDR_100_SPINDLE
/index.php?c=DigitalProjectors&n=160355011&i=B00004 Z6N6&x=Belkin_36_Piece_Demagnatized_Computer_Tool_ Kit_with_Case_Black
/index.php?c=LCDProjectors&n=160356011&i=B00011R2VS &x=Sony_DRU_530A_8x_Internal_Dual_Format_DVD+_RW_D rive
/index.php?c=DigitalProjectors&n=160355011&i=B00004 Z6XS&x=Belkin_F8E062_55_Piece_Computer_Tool_Kit_wi th_Black_Case_Demagnetized_Tools
/index.php?c=LCDProjectors&n=160356011&i=B0002340SI &x=KHypermedia_25_x_DVD+R_47_GB_4x_spindle_storage _media
/index.php?c=DigitalProjectors&n=160355011&i=B0002W 3A6W&x=Paladin_Tools_1576_Data_Link_ID_And_Cable_C heck
/index.php?c=LCDProjectors&n=160356011&i=B000233WGO &x=Khypermedia_Spindle_of_25_Blank_4x_DVD_R_Dis cs
/index.php?c=Home&n=281056&i=B000RZPK1M&x=Cyberpowe r_CP825LCD_UPS_825VA_450W_LCD_Display_8_Outlet_RJ1 1_Compact_Design_EMI_RFI
/index.php?c=VideoProjector&n=160355011&i=B000ON2TQ G&x=Panasonic_Prestige_NN_SD997S_22cuft_1250_Watt_ Sensor_Microwave_Oven_Stainless_Steel
/index.php?c=LCDProjectors&n=160356011&i=B0043862N4 &x=Mini_Bluetooth_Wireless_Keyboard_for_iPad_iPhon e_40_OS_Android_Window_Mobile_Symbian_Smartphone
/index.php?c=VideoProjector&n=160355011&i=027373279 X&x=The_New_Business_Road_Test_What_entrepreneurs_ and_executives_should_do_before_writing_a_business _plan_3rd_Edition
/index.php?c=VideoProjector&n=160355011&i=B000I98ZM 8&x=2_x_RCA_Female_1_x_35mm_Mono_Female_Adapter
/index.php?c=DigitalProjectors&n=160355011&i=B000NB CEM8&x=Sylvania_SA135_10_Amp_Table_Top_Timer_with_ Motion_Sensor_and_Alarm
/index.php?c=DigitalProjectors&n=160355011&i=B00005 3U3X&x=Star_Wars_The_Force_Unleashed_Ultimate_Sith _Edition
/index.php?c=Home&n=281056&i=B000FQ2D5E&x=Metal_Gea r_Solid_4_Guns_of_the_Patriots
/index.php?c=VideoProjector&n=160355011&i=B004DJSVI W&x=X_Mini_II_Capsule_Speaker_Pink
/index.php?c=DigitalProjectors&n=160355011&i=B00009 N6VJ&x=Monster_SV100_F_25_WHT_Coaxial_Video_Cable_ 25_feet
/index.php?c=DigitalProjectors&n=160355011&i=B000O8 AHLA&x=GLS_Audio_Locking_Series_Generation_4_Gold_ Connector_Banana_Plugs_Banana_Clips_20_Pack_10_Red s_10_Blacks
/index.php?c=DigitalProjectors&n=160355011&i=B00004 Z61J&x=Belkin_18_Piece_Computer_Tool_Kit_with_Dema gnetized_Tools_Black_Case
/index.php?c=LCDProjectors&n=160356011&i=B004Z75V08 &x=Contour_1500_Contour_Plus_Camera
/index.php?c=DigitalProjectors&n=160355011&i=B00328 HR5S&x=Sony_DSC_W330_141MP_Digital_Camera_with_4x_ Wide_Angle_Zoom_with_Digital_Steady_Shot_Image_Sta bilization_and_30_inch_LCD_Blue
/index.php?c=3D&n=160356011&i=B0021AEPTY&x=Elgato_1 0020196_Turbo264_HD_Encoder_Accelerator
/index.php?c=node&n=3349991&i=B000QDIUEO&x=JVCC_ART _11_Artist_Board_Console_Tape_3_4_in_x_60_yds_Whit e
/index.php?c=node&n=3349991&i=B00009XVIS&x=Photofle x_FV_Connector_7
/index.php?c=LCDProjectors&n=160356011&i=B004GHMS44 &x=Seidio_Ultimate_Screen_Guard_for_Samsung_Focus_ 2_Pack_Crystal_Clear
/index.php?c=LCDProjectors&n=160356011&i=B000E8BGCE &x=Ethernet_Cable_CAT5e_25_ft_White
/index.php?c=LCDProjectors&n=160356011&i=B0046HNWO4 &x=3M_CP40_Shootn_Share_Camcorder_with_Projector_W hite
/index.php?c=LCDProjectors&n=160356011&i=B0029NRM2U &x=Magic_Flight_Launch_Box
/index.php?c=VideoProjector&n=160355011&i=B004LU1CA 2&x=Satechi_BT_Wireless_Bluetooth_Portable_Speaker _System_for_iPhone_Android_Smart_Phones_iPad_Table ts_Macbook_Notebooks
/index.php?c=LCDProjectors&n=160356011&i=B0002EH4YO &x=Mini_Multi_Battery_Tester
/index.php?c=DigitalProjectors&n=160355011&i=B00386 VMWI&x=NCAA_Football_11
/index.php?c=LCDProjectors&n=160356011&i=B003O6N64Y &x=Ear_Force_PX5_Programmable_Wireless_71_Dolby_Di gital_Surround_Sound_Headset_with_Bluetooth
/index.php?c=3D&n=160356011&i=B000HJPK2C&x=Nikon_70 _300mm_f_45_56G_ED_IF_AF_S_VR_Nikkor_Zoom_Lens_for _Nikon_Digital_SLR_Cameras
/index.php?c=VideoProjector&n=160355011&i=B000YZAZM O&x=Evenflo_ABC_SmartSteps_ExerSaucer
/index.php?c=LCDProjectors&n=160356011&i=0321749847 &x=Apple_Pro_Video_Training_Aperture_3_Apple_Pro_T raining
/index.php?c=3D&n=160356011&i=B003UATR3I&x=Elgato_E yeTV_Hybrid_10021020
/index.php?c=Home&n=281056&i=B003EQ4C92&x=Royal_Opu lance_Satin_Queen_Sheet_Set_Giraffe_Brown_Ivory
/index.php?c=LCDProjectors&n=160356011&i=B004LDE3PU &x=Justin_Bieber_Limited_Edition_3_d_Glasses
/index.php?c=VideoProjector&n=160355011&i=B004O77NL E&x=Samsung_UN40D6500_40_Inch_1080p_120HZ_3D_LED_T V_Black
/index.php?c=LCDProjectors&n=160356011&i=B00009W3DU &x=RCA_DLNB2BOX_LNB_Dual

[Tia Wood]

I have no clue how that's happening. I followed a few links and it sends me back to the first page as if they weren't found. Google isn't going to crawl Amazon via your site and it isn't going to modify and follow unSEO friendly links like that. I highly suggest contacting associate-o-matic to find out what could be going on unless POLiSH has an idea of what could be happening.

[POLiSH]

From what I have read regarding associate-o-matic and what you are viewing DropWizard, this seems to be a known problem for people using associate-o-matic using Amazon Affiliate program and all of the sudden getting blown away with bandwidth usage. If you have already tuned down the crawl rate on google, give it around 72 hours and see if you see a decrease. If you don't I would be looking for a throttle down option via your amazon account.

[DropWizard.com]

Yeah some of the links resolve to pages some don't. And the links are unfriendly that was one of the points I made about having to rewrite the htaccess rules but that's beyond my tech ability.

Well on to associate to see what else. Their solution also seems to be turning down the crawl rate.

Conclusion. Software is not ready for the big time

Thanks everyone for the help. It's greatly appreciated.